1
|
Brewer BC, Bantis LE. Cutoff estimation and construction of their confidence intervals for continuous biomarkers under ternary umbrella and tree stochastic ordering settings. Stat Med 2024; 43:606-623. [PMID: 38038216 PMCID: PMC10880868 DOI: 10.1002/sim.9974] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Revised: 10/30/2023] [Accepted: 11/17/2023] [Indexed: 12/02/2023]
Abstract
Tuberculosis (TB) studies often involve four different states under consideration, namely, "healthy," "latent infection," "pulmonary active disease," and "extra-pulmonary active disease." While highly accurate clinical diagnosis tests do exist, they are expensive and generally not accessible in regions where they are most needed; thus, there is an interest in assessing the accuracy of new and easily obtainable biomarkers. For some such biomarkers, the typical stochastic ordering assumption might not be justified for all disease classes under study, and usual ROC methodologies that involve ROC surfaces and hypersurfaces are inadequate. Different types of orderings may be appropriate depending on the setting, and these may involve a number of ambiguously ordered groups that stochastically exhibit larger (or lower) marker scores than the remaining groups. Recently, there has been scientific interest on ROC methods that can accommodate these so-called "tree" or "umbrella" orderings. However, there is limited work discussing the estimation of cutoffs in such settings. In this article, we discuss the estimation and inference around optimized cutoffs when accounting for such configurations. We explore different cutoff alternatives and provide parametric, flexible parametric, and non-parametric kernel-based approaches for estimation and inference. We evaluate our approaches using simulations and illustrate them through a real data set that involves TB patients.
Collapse
Affiliation(s)
- Benjamin C Brewer
- Department of Biostatistics & Data Science, The University of Kansas Medical Center, Kansas City, Kansas, USA
| | - Leonidas E Bantis
- Department of Biostatistics & Data Science, The University of Kansas Medical Center, Kansas City, Kansas, USA
| |
Collapse
|
2
|
Samawi H, Alsharman M, Keko M, Kersey J. Post-test diagnostic accuracy measures under tree ordering of disease classes. Stat Med 2023; 42:5135-5159. [PMID: 37720999 DOI: 10.1002/sim.9905] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 08/15/2023] [Accepted: 09/01/2023] [Indexed: 09/19/2023]
Abstract
The medical field commonly employs post-test measures such as predictive values and likelihood ratios to assess diagnostic accuracy. Predictive values, including positive and negative values (PPV and NPV), indicate the probability that individuals have a target health condition based on test results. On the other hand, likelihood ratios, including positive and negative ratios (LR+ and LR- respectively), compare the probability of a particular test result between the diseased and non-diseased groups. While predictive values are useful in evaluating diagnostic test accuracy in populations with varying disease prevalence, likelihood ratios provide a direct link between pre-test and post-test probabilities in specific patients. In this study, we introduce and analyze a new approach called generalized predictive values and likelihood ratios, using a tree ordering of disease classes. We evaluate the effectiveness of these methods through simulation studies and illustrate their use with real data on lung cancer.
Collapse
Affiliation(s)
- Hani Samawi
- Department of Biostatistics, Epidemiology and Environmental Health Sciences, Jiann-Ping Hsu College of Public Health, Georgia Southern University, Statesboro, Georgia, USA
| | - Marwan Alsharman
- Department of Biostatistics, Epidemiology and Environmental Health Sciences, Jiann-Ping Hsu College of Public Health, Georgia Southern University, Statesboro, Georgia, USA
| | - Mario Keko
- Department of Biostatistics, Epidemiology and Environmental Health Sciences, Jiann-Ping Hsu College of Public Health, Georgia Southern University, Statesboro, Georgia, USA
| | - Jing Kersey
- Department of Biostatistics, Epidemiology and Environmental Health Sciences, Jiann-Ping Hsu College of Public Health, Georgia Southern University, Statesboro, Georgia, USA
| |
Collapse
|
3
|
MacAskill MR, Pitcher TL, Melzer TR, Myall DJ, Horne KL, Shoorangiz R, Almuqbel MM, Livingston L, Grenfell S, Pascoe MJ, Marshall ET, Marsh S, Perry SE, Meissner WG, Theys C, Le Heron CJ, Keenan RJ, Dalrymple-Alford JC, Anderson TJ. The New Zealand Parkinson’s progression programme. J R Soc N Z 2022. [DOI: 10.1080/03036758.2022.2111448] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2022]
Affiliation(s)
- Michael R. MacAskill
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
| | - Toni L. Pitcher
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
| | - Tracy R. Melzer
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
- School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand
| | - Daniel J. Myall
- New Zealand Brain Research Institute, Christchurch, New Zealand
| | | | - Reza Shoorangiz
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
- Department of Electrical and Computer Engineering, University of Canterbury, Christchurch, New Zealand
| | - Mustafa M. Almuqbel
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Pacific Radiology, Christchurch, New Zealand
| | - Leslie Livingston
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
| | - Sophie Grenfell
- New Zealand Brain Research Institute, Christchurch, New Zealand
| | - Maddie J. Pascoe
- New Zealand Brain Research Institute, Christchurch, New Zealand
- School of Physical Education, Sport and Exercise Sciences, University of Otago, Dunedin, New Zealand
| | - Ethan T. Marshall
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
| | - Steven Marsh
- Department of Medical Physics, University of Canterbury, Christchurch, New Zealand
| | - Sarah E. Perry
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
| | - Wassilios G. Meissner
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
- Institute of Neurodegenerative Diseases (IMN), University of Bordeaux, Bordeaux, France
| | - Catherine Theys
- New Zealand Brain Research Institute, Christchurch, New Zealand
- School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand
| | - Campbell J. Le Heron
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
- School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand
- Department of Neurology, Canterbury District Health Board, Christchurch, New Zealand
| | - Ross J. Keenan
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Pacific Radiology, Christchurch, New Zealand
| | - John C. Dalrymple-Alford
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
- School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand
| | - Tim J. Anderson
- New Zealand Brain Research Institute, Christchurch, New Zealand
- Department of Medicine, University of Otago, Christchurch, New Zealand
- Department of Neurology, Canterbury District Health Board, Christchurch, New Zealand
| |
Collapse
|
4
|
To DK, Adimari G, Chiogna M, Risso D. Receiver operating characteristic estimation and threshold selection criteria in three-class classification problems for clustered data. Stat Methods Med Res 2022; 31:1325-1341. [PMID: 35360997 DOI: 10.1177/09622802221089029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Statistical evaluation of diagnostic tests, and, more generally, of biomarkers, is a constantly developing field, in which complexity of the assessment increases with the complexity of the design under which data are collected. One particularly prevalent type of data is clustered data, where individual units are naturally nested into clusters. In these cases, Bias can arise from omission, in the evaluation process, of cluster-level effects and/or individual covariates. Focusing on the three-class case and for continuous-valued diagnostic tests, we investigate how to exploit the clustered structure of data within a linear-mixed model approach, both when the assumption of normality holds and when it does not. We provide a method for the estimation of covariate-specific receiver operating characteristic surfaces and discuss methods for the choice of optimal thresholds, proposing three possible estimators. A proof of consistency and asymptotic normality of the proposed threshold estimators is given. All considered methods are evaluated by extensive simulation experiments. As an application, we study the use of the Lysosomal Associated Membrane Protein Family Member 5 gene expression as a biomarker to distinguish among three types of glutamatergic neurons.
Collapse
Affiliation(s)
- Duc-Khanh To
- Department of Statistical Sciences, 9308University of Padova, Italy
| | | | - Monica Chiogna
- Department of Statistical Sciences "Paolo Fortunati", 9296University of Bologna, Italy
| | - Davide Risso
- Department of Statistical Sciences, 9308University of Padova, Italy
| |
Collapse
|
5
|
Feng Y, Tian L. Flexible diagnostic measures and new cut-point selection methods under multiple ordered classes. Pharm Stat 2021; 21:220-240. [PMID: 34449107 DOI: 10.1002/pst.2166] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Revised: 07/21/2021] [Accepted: 08/01/2021] [Indexed: 11/08/2022]
Abstract
Medical diagnosis is essentially a classification problem and usually it is done with multiple ordered classes. For example, cancer diagnosis might be "non-malignant," "early stage," or "late stage." Therefore, appropriate measures are needed to assess the accuracy of diagnostic markers under multiple ordered classes. However, all existing measures fail to differentiate among some distinctly different biomarkers. This paper presents a multi-step procedure for evaluating biomarker accuracy under multiple ordered classes. This procedure leads to two new flexible overall measures as well as three new cut-point selection methods with great computational ease. The performance of proposed measures and cut-point selection methods are numerically explored via a simulation study. In the end, an ovarian cancer dataset from the Prostate, Lung, Colorectal, and Ovarian cancer study is analyzed. The proposed accuracy measures were estimated for markers CA125 and HE4, and cut-points were estimated for the risk of ovarian malignancy algorithm score.
Collapse
Affiliation(s)
- Yingdong Feng
- Department of Biostatistics, University at Buffalo, Buffalo, New York, USA
| | - Lili Tian
- Department of Biostatistics, University at Buffalo, Buffalo, New York, USA
| |
Collapse
|
6
|
Gao Y, Tian L. Confidence interval estimation for sensitivity and difference between two sensitivities at a given specificity under tree ordering. Stat Med 2021; 40:3695-3723. [PMID: 33906262 DOI: 10.1002/sim.8993] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Revised: 03/24/2021] [Accepted: 04/01/2021] [Indexed: 11/07/2022]
Abstract
This article considers a setting in diagnostic studies (or biomarker study) which involves a healthy class and a diseased class and the latter consists of several subclasses. The problem of interest is to evaluate the accuracy of a biomarker (or a diagnostic test) measured on a continuous scale correctly identifying healthy subjects from diseased subjects without requiring specification of an ordering in terms of marker values for subclasses relative to each other within the diseased class. Such setting is quite common in practice and it falls in the framework of tree ordering or umbrella ordering. This article explores several parametric and nonparametric approaches for estimating confidence intervals of sensitivity of single biomarker and difference between sensitivities of two correlated biomarkers under tree ordering at a given specificity. The performances of all the methods are evaluated and compared by a comprehensive simulation study. A published microarray data set is analyzed using the proposed methods.
Collapse
Affiliation(s)
- Yi Gao
- Department of Biostatistics, University at Buffalo, Buffalo, New York, USA
| | - Lili Tian
- Department of Biostatistics, University at Buffalo, Buffalo, New York, USA
| |
Collapse
|
7
|
Feng Y, Tian L. Issues and solutions in biomarker evaluation when subclasses are involved under binary classification. Stat Methods Med Res 2020; 30:87-98. [PMID: 32726186 DOI: 10.1177/0962280220938077] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
In practice, it is common to evaluate biomarkers in binary classification settings (e.g. non-cancer vs. cancer) where one or both main classes involve multiple subclasses. For example, non-cancer class might consist of healthy subjects and benign cases, while cancer class might consist of subjects at early and late stages. The standard practice is pooling within each main class, i.e. all non-cancer subclasses are pooled together to create a control group, and all cancer subclasses are pooled together to create a case group. Based on the pooled data, the area under ROC curve (AUC) and other characteristics are estimated under binary classification for the purpose of biomarker evaluation. Despite the popularity of this pooling strategy in practice, its validity and implication in biomarker evaluation have never been carefully inspected. This paper aims to demonstrate that pooling strategy can be seriously misleading in biomarker evaluation. Furthermore, we present a new diagnostic framework as well as new accuracy measures appropriate for biomaker evaluation under such settings. In the end, an ovarian cancer data set is analyzed.
Collapse
Affiliation(s)
- Yingdong Feng
- Department of Biostatistics, University at Buffalo, Buffalo, NY, USA
| | - Lili Tian
- Department of Biostatistics, University at Buffalo, Buffalo, NY, USA
| |
Collapse
|
8
|
Feng Y, Tian L. Measuring diagnostic accuracy for biomarkers under tree-ordering. Stat Methods Med Res 2018; 28:1328-1346. [DOI: 10.1177/0962280218755810] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
Abstract
In the field of diagnostic studies for tree or umbrella ordering, under which the marker measurement for one class is lower or higher than those for the rest unordered classes, there exist a few diagnostic measures such as the naive AUC ( NAUC), the umbrella volume ( UV), and the recently proposed TAUC, i.e. area under a ROC curve for tree or umbrella ordering (TROC). However, an important characteristic about tree or umbrella ordering has been neglected. This paper mainly focuses on promoting the use of the integrated false negative rate under tree ordering ( ITFNR) as an additional diagnostic measure besides TAUC, and proposing the idea of using ( TAUC, ITFNR) instead of TAUC to evaluate the diagnostic accuracy of a biomarker under tree or umbrella ordering. Parametric and non-parametric approaches for constructing joint confidence region of ( TAUC, ITFNR) are proposed. Simulation studies under a variety of settings are carried out to assess and compare the performance of these methods. In the end, a published microarray data set is analyzed.
Collapse
Affiliation(s)
- Yingdong Feng
- Department of Biostatistics, University at Buffalo, Buffalo, NY, USA
| | - Lili Tian
- Department of Biostatistics, University at Buffalo, Buffalo, NY, USA
| |
Collapse
|
9
|
Coolen-Maturi T. Three-group ROC predictive analysis for ordinal outcomes. COMMUN STAT-THEOR M 2017. [DOI: 10.1080/03610926.2016.1212074] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Affiliation(s)
- Tahani Coolen-Maturi
- Department of Economics and Finance, Durham University Business School, Durham University, Durham, UK
| |
Collapse
|
10
|
Li J, Fine JP, Pencina MJ. Multi-category diagnostic accuracy based on logistic regression. ACTA ACUST UNITED AC 2017. [DOI: 10.1080/24754269.2017.1319105] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Affiliation(s)
- Jialiang Li
- Department of Statistics and Applied Probability, Duke-NUS Graduate Medical School, Singapore Eye Research Institute, National University of Singapore, Singapore
| | - Jason P. Fine
- Department of Biostatistics, University of North Carolina, Chapel Hill, NC, USA
| | | |
Collapse
|
11
|
Kiatsupaibul S, J. Hayter A, Liu W. Rank constrained distribution and moment computations. Comput Stat Data Anal 2017. [DOI: 10.1016/j.csda.2016.08.009] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
|
12
|
Zhang Y, Alonzo TA. Inverse probability weighting estimation of the volume under the ROC surface in the presence of verification bias. Biom J 2016; 58:1338-1356. [PMID: 27338713 DOI: 10.1002/bimj.201500225] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2015] [Revised: 02/28/2016] [Accepted: 03/10/2016] [Indexed: 11/08/2022]
Abstract
In diagnostic medicine, the volume under the receiver operating characteristic (ROC) surface (VUS) is a commonly used index to quantify the ability of a continuous diagnostic test to discriminate between three disease states. In practice, verification of the true disease status may be performed only for a subset of subjects under study since the verification procedure is invasive, risky, or expensive. The selection for disease examination might depend on the results of the diagnostic test and other clinical characteristics of the patients, which in turn can cause bias in estimates of the VUS. This bias is referred to as verification bias. Existing verification bias correction in three-way ROC analysis focuses on ordinal tests. We propose verification bias-correction methods to construct ROC surface and estimate the VUS for a continuous diagnostic test, based on inverse probability weighting. By applying U-statistics theory, we develop asymptotic properties for the estimator. A Jackknife estimator of variance is also derived. Extensive simulation studies are performed to evaluate the performance of the new estimators in terms of bias correction and variance. The proposed methods are used to assess the ability of a biomarker to accurately identify stages of Alzheimer's disease.
Collapse
Affiliation(s)
- Ying Zhang
- Department of Biostatistics, University of Southern California, Keck School of Medicine, Los Angeles, California 90033, USA.
| | - Todd A Alonzo
- Department of Biostatistics, University of Southern California, Keck School of Medicine, Los Angeles, California 90033, USA
| | | |
Collapse
|
13
|
|
14
|
Wang D, Attwood K, Tian L. Receiver operating characteristic analysis under tree orderings of disease classes. Stat Med 2015; 35:1907-26. [DOI: 10.1002/sim.6843] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2015] [Revised: 11/15/2015] [Accepted: 11/19/2015] [Indexed: 11/11/2022]
Affiliation(s)
- Dan Wang
- Department of Biostatistics & Bioinformatics; Roswell Park Cancer Institute; Elm and Carlton Streets Buffalo 14263 NY U.S.A
- Department of Biostatistics; SUNY University at Buffalo; 3435 Main St. Buffalo 14214 NY U.S.A
| | - Kristopher Attwood
- Department of Biostatistics & Bioinformatics; Roswell Park Cancer Institute; Elm and Carlton Streets Buffalo 14263 NY U.S.A
| | - Lili Tian
- Department of Biostatistics & Bioinformatics; Roswell Park Cancer Institute; Elm and Carlton Streets Buffalo 14263 NY U.S.A
- Department of Biostatistics; SUNY University at Buffalo; 3435 Main St. Buffalo 14214 NY U.S.A
| |
Collapse
|
15
|
Coolen-Maturi T, Elkhafifi FF, Coolen FP. Three-group ROC analysis: A nonparametric predictive approach. Comput Stat Data Anal 2014. [DOI: 10.1016/j.csda.2014.04.005] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
16
|
Schubert CM, Guennel T. Comparing Performance of Multiclass Classification Systems with ROC Manifolds: When Volume and Correct Classification Fails. COMMUN STAT-SIMUL C 2014. [DOI: 10.1080/03610918.2013.794284] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
17
|
Van Hoorde K, Vergouwe Y, Timmerman D, Van Huffel S, Steyerberg EW, Van Calster B. Assessing calibration of multinomial risk prediction models. Stat Med 2014; 33:2585-96. [PMID: 24549725 DOI: 10.1002/sim.6114] [Citation(s) in RCA: 37] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2013] [Revised: 12/16/2013] [Accepted: 01/27/2014] [Indexed: 11/07/2022]
Abstract
Calibration, that is, whether observed outcomes agree with predicted risks, is important when evaluating risk prediction models. For dichotomous outcomes, several tools exist to assess different aspects of model calibration, such as calibration-in-the-large, logistic recalibration, and (non-)parametric calibration plots. We aim to extend these tools to prediction models for polytomous outcomes. We focus on models developed using multinomial logistic regression (MLR): outcome Y with k categories is predicted using k - 1 equations comparing each category i (i = 2, … ,k) with reference category 1 using a set of predictors, resulting in k - 1 linear predictors. We propose a multinomial logistic recalibration framework that involves an MLR fit where Y is predicted using the k - 1 linear predictors from the prediction model. A non-parametric alternative may use vector splines for the effects of the linear predictors. The parametric and non-parametric frameworks can be used to generate multinomial calibration plots. Further, the parametric framework can be used for the estimation and statistical testing of calibration intercepts and slopes. Two illustrative case studies are presented, one on the diagnosis of malignancy of ovarian tumors and one on residual mass diagnosis in testicular cancer patients treated with cisplatin-based chemotherapy. The risk prediction models were developed on data from 2037 and 544 patients and externally validated on 1107 and 550 patients, respectively. We conclude that calibration tools can be extended to polytomous outcomes. The polytomous calibration plots are particularly informative through the visual summary of the calibration performance.
Collapse
Affiliation(s)
- Kirsten Van Hoorde
- KU Leuven Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics, Leuven, Belgium; KU Leuven iMinds Future Health Department, Leuven, Belgium
| | | | | | | | | | | |
Collapse
|
18
|
Wan S, Zhang B. Semiparametric ROC surface estimation for continuous diagnostic tests via polytomous logistic regression procedures. J STAT COMPUT SIM 2013. [DOI: 10.1080/00949655.2012.684096] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/28/2022]
|
19
|
Assessing the discriminative ability of risk models for more than two outcome categories. Eur J Epidemiol 2012; 27:761-70. [PMID: 23054032 DOI: 10.1007/s10654-012-9733-3] [Citation(s) in RCA: 62] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2012] [Accepted: 09/14/2012] [Indexed: 12/21/2022]
Abstract
The discriminative ability of risk models for dichotomous outcomes is often evaluated with the concordance index (c-index). However, many medical prediction problems are polytomous, meaning that more than two outcome categories need to be predicted. Unfortunately such problems are often dichotomized in prediction research. We present a perspective on the evaluation of discriminative ability of polytomous risk models, which may instigate researchers to consider polytomous prediction models more often. First, we suggest a "discrimination plot" as a tool to visualize the model's discriminative ability. Second, we discuss the use of one overall polytomous c-index versus a set of dichotomous measures to summarize the performance of the model. Third, we address several aspects to consider when constructing a polytomous c-index. These involve the assessment of concordance in pairs versus sets of patients, weighting by outcome prevalence, the value related to models with random performance, the reduction to the dichotomous c-index for dichotomous problems, and interpretation. We illustrate these issues on case studies dealing with ovarian cancer (four outcome categories) and testicular cancer (three categories). We recommend the use of a discrimination plot together with an overall c-index such as the Polytomous Discrimination Index. If the overall c-index suggests that the model has relevant discriminative ability, pairwise c-indexes for each pair of outcome categories are informative. For pairwise c-indexes we recommend the 'conditional-risk' method which is consistent with the analytical approach of the multinomial logistic regression used to develop polytomous risk models.
Collapse
|
20
|
Shiu SY, Gatsonis C. On ROC analysis with nonbinary reference standard. Biom J 2012; 54:457-80. [PMID: 22641278 DOI: 10.1002/bimj.201100206] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2011] [Revised: 02/21/2012] [Accepted: 03/23/2012] [Indexed: 11/10/2022]
Abstract
Statistical methods for the evaluation of the accuracy of diagnostic tests usually assume a binary true disease status. However, this assumption may not be realistic in practical settings in which "disease" is defined by dichotomizing continuous or ordinal categorical measures using a pre-specified threshold value. In this paper, we focus on the analysis of studies in which both the diagnostic test and the reference standard are reported as continuous measures. We propose a semiparametric model for estimating the sensitivity, specificity, and the ROC curve as functions of reference standard thresholds. Under suitable order restrictions on the mean of the test result variable, fitting is done via two alternative approaches: isotonic regression and monotone smoothing splines. The model provides the basis to assess the effect of varying reference standard threshold on the performance of a diagnostic test. An example to evaluate the ability of the maximal SUV-lean (standardized uptake value normalized to lean body mass) in predicting axillary node involvement in women diagnosed with breast cancer is presented.
Collapse
Affiliation(s)
- Shang-Ying Shiu
- Department of Statistics, National Taipei University, 151 University Road, New Taipei City, 237, Taiwan.
| | | |
Collapse
|
21
|
Shan G, Hutson AD, Wilding GE. Two-stage k-sample designs for the ordered alternative problem. Pharm Stat 2012; 11:287-94. [PMID: 22408050 DOI: 10.1002/pst.1499] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2011] [Revised: 01/12/2012] [Accepted: 01/15/2012] [Indexed: 11/08/2022]
Abstract
In preclinical studies and clinical dose-ranging trials, the Jonckheere-Terpstra test is widely used in the assessment of dose-response relationships. Hewett and Spurrier (1979) presented a two-stage analog of the test in the context of large sample sizes. In this paper, we propose an exact test based on Simon's minimax and optimal design criteria originally used in one-arm phase II designs based on binary endpoints. The convergence rate of the joint distribution of the first and second stage test statistics to the limiting distribution is studied, and design parameters are provided for a variety of assumed alternatives. The behavior of the test is also examined in the presence of ties, and the proposed designs are illustrated through application in the planning of a hypercholesterolemia clinical trial. The minimax and optimal two-stage procedures are shown to be preferable as compared with the one-stage procedure because of the associated reduction in expected sample size for given error constraints.
Collapse
Affiliation(s)
- Guogen Shan
- Department of Biostatistics, University at Buffalo, Buffalo, NY, USA
| | | | | |
Collapse
|
22
|
Dong T, Tian L, Hutson A, Xiong C. Parametric and non-parametric confidence intervals of the probability of identifying early disease stage given sensitivity to full disease and specificity with three ordinal diagnostic groups. Stat Med 2011; 30:3532-45. [PMID: 22139763 DOI: 10.1002/sim.4401] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2011] [Accepted: 08/12/2011] [Indexed: 12/14/2022]
Abstract
In practice, there exist many disease processes with three ordinal disease classes, that is, the non-diseased stage, the early disease stage, and the fully diseased stage. Because early disease stage is likely the best time window for treatment interventions, it is important to have diagnostic tests that have good diagnostic ability to discriminate the early disease stage from the other two stages. In this paper, we present both parametric and non-parametric approaches for confidence interval estimation of probability of detecting early disease stage given the true classification rates for non-diseased group and diseased group, namely, the specificity and the sensitivity to full disease. We analyze a data set on the clinical diagnosis of early-stage Alzheimer's disease from the neuropsychological database at the Washington University Alzheimer's Disease Research Center using the proposed approaches.
Collapse
Affiliation(s)
- Tuochuan Dong
- Department of Biostatistics, University at Buffalo, Buffalo, NY 14214-3000, USA
| | | | | | | |
Collapse
|
23
|
Nakas CT, Alonzo TA, Yiannoutsos CT. Accuracy and cut-off point selection in three-class classification problems using a generalization of the Youden index. Stat Med 2011; 29:2946-55. [PMID: 20809485 DOI: 10.1002/sim.4044] [Citation(s) in RCA: 68] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/03/2023]
Abstract
We study properties of the index J(3), defined as the accuracy, or the maximum correct classification, for a given three-class classification problem. Specifically, using J(3) one can assess the discrimination between the three distributions and obtain an optimal pair of cut-off points c(1)<c(2) in the sense that the sum of the correct classification proportions will be maximized. It also serves as the generalization of the Youden index in three-class problems. Parametric and non-parametric approaches for estimation and testing are considered and methods are applied to data from an MRS study on human immunodeficiency virus (HIV) patients.
Collapse
Affiliation(s)
- Christos T Nakas
- Laboratory of Biometry, University of Thessaly, Fytokou Str, N. Ionia, 384 46 Magnesia, Greece.
| | | | | |
Collapse
|
24
|
Wang Z, Zhou XH, Wang M. Evaluation of diagnostic accuracy in detecting ordered symptom statuses without a gold standard. Biostatistics 2011; 12:567-81. [PMID: 21209155 DOI: 10.1093/biostatistics/kxq075] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022] Open
Abstract
Our research is motivated by 2 methodological problems in assessing diagnostic accuracy of traditional Chinese medicine (TCM) doctors in detecting a particular symptom whose true status has an ordinal scale and is unknown-imperfect gold standard bias and ordinal scale symptom status. In this paper, we proposed a nonparametric maximum likelihood method for estimating and comparing the accuracy of different doctors in detecting a particular symptom without a gold standard when the true symptom status had an ordered multiple class. In addition, we extended the concept of the area under the receiver operating characteristic curve to a hyper-dimensional overall accuracy for diagnostic accuracy and alternative graphs for displaying a visual result. The simulation studies showed that the proposed method had good performance in terms of bias and mean squared error. Finally, we applied our method to our motivating example on assessing the diagnostic abilities of 5 TCM doctors in detecting symptoms related to Chills disease.
Collapse
Affiliation(s)
- Zheyu Wang
- Department of Biostatistics, University of Washington, Box 357232, Seattle, Washington 98195, USA.
| | | | | |
Collapse
|
25
|
Tian L, Xiong C, Lai CY, Vexler A. Exact confidence interval estimation for the difference in diagnostic accuracy with three ordinal diagnostic groups. J Stat Plan Inference 2010; 141:549-558. [PMID: 23538945 DOI: 10.1016/j.jspi.2010.07.004] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
In the cases with three ordinal diagnostic groups, the important measures of diagnostic accuracy are the volume under surface (VUS) and the partial volume under surface (PVUS) which are the extended forms of the area under curve (AUC) and the partial area under curve (PAUC). This article addresses confidence interval estimation of the difference in paired VUS s and the difference in paired PVUS s. To focus especially on studies with small to moderate sample sizes, we propose an approach based on the concepts of generalized inference. A Monte Carlo study demonstrates that the proposed approach generally can provide confidence intervals with reasonable coverage probabilities even at small sample sizes. The proposed approach is compared to a parametric bootstrap approach and a large sample approach through simulation. Finally, the proposed approach is illustrated via an application to a data set of blood test results of anemia patients.
Collapse
Affiliation(s)
- Lili Tian
- Department of Biostatistics, University at Buffalo, 249 Farber Hall, 3435 Main St. Bldg. 26 Buffalo, NY 14214-3000, USA
| | | | | | | |
Collapse
|
26
|
Alonzo TA, Nakas CT, Yiannoutsos CT, Bucher S. A comparison of tests for restricted orderings in the three-class case. Stat Med 2009; 28:1144-58. [DOI: 10.1002/sim.3536] [Citation(s) in RCA: 14] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
27
|
Anglim PP, Galler JS, Koss MN, Hagen JA, Turla S, Campan M, Weisenberger DJ, Laird PW, Siegmund KD, Laird-Offringa IA. Identification of a panel of sensitive and specific DNA methylation markers for squamous cell lung cancer. Mol Cancer 2008; 7:62. [PMID: 18616821 PMCID: PMC2483990 DOI: 10.1186/1476-4598-7-62] [Citation(s) in RCA: 93] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2008] [Accepted: 07/10/2008] [Indexed: 02/06/2023] Open
Abstract
Background Lung cancer is the leading cause of cancer death in men and women in the United States and Western Europe. Over 160,000 Americans die of this disease every year. The five-year survival rate is 15% – significantly lower than that of other major cancers. Early detection is a key factor in increasing lung cancer patient survival. DNA hypermethylation is recognized as an important mechanism for tumor suppressor gene inactivation in cancer and could yield powerful biomarkers for early detection of lung cancer. Here we focused on developing DNA methylation markers for squamous cell carcinoma of the lung. Using the sensitive, high-throughput DNA methylation analysis technique MethyLight, we examined the methylation profile of 42 loci in a collection of 45 squamous cell lung cancer samples and adjacent non-tumor lung tissues from the same patients. Results We identified 22 loci showing significantly higher DNA methylation levels in tumor tissue than adjacent non-tumor lung. Of these, eight showed highly significant hypermethylation in tumor tissue (p < 0.0001): GDNF, MTHFR, OPCML, TNFRSF25, TCF21, PAX8, PTPRN2 and PITX2. Used in combination on our specimen collection, this eight-locus panel showed 95.6% sensitivity and specificity. Conclusion We have identified 22 DNA methylation markers for squamous cell lung cancer, several of which have not previously been reported to be methylated in any type of human cancer. The top eight markers show great promise as a sensitive and specific DNA methylation marker panel for squamous cell lung cancer.
Collapse
Affiliation(s)
- Paul P Anglim
- Department of Surgery, Norris Cancer Center, Keck School of Medicine, University of Southern California, Los Angeles, CA, 90089-9176, USA.
| | | | | | | | | | | | | | | | | | | |
Collapse
|
28
|
Alonzo TA, Nakas CT. Comparison of ROC Umbrella Volumes with an Application to the Assessment of Lung Cancer Diagnostic Markers. Biom J 2007; 49:654-64. [PMID: 17726716 DOI: 10.1002/bimj.200610363] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Receiver operating characteristic (ROC) analysis is widely used to assess the ability of diagnostic markers to correctly classify into one of two disease classes. ROC surfaces and umbrella surfaces generalize the utility of ROC analysis when there are three disease classes. Identification of lung cancer diagnostic markers is an active area of research since prognosis for those diagnosed with lung cancer is so poor and there is not an accurate method for early detection of lung cancer. A study conducted for the assessment of DNA methylation markers motivated the comparison of ROC umbrella surfaces which is developed in this article using U-statistics and bootstrap methodology.
Collapse
Affiliation(s)
- Todd A Alonzo
- Division of Biostatistics, University of Southern California Keck School of Medicine, 440 E. Huntington Dr, 4th floor, Arcadia, CA 91006, USA.
| | | |
Collapse
|