1
|
Wu M, Sun X, Liu A, Peng C, Li Z. Significance tests for covariates in the diagnostic accuracy index of a biomarker against a continuous gold standard. Stat Med 2023; 42:4015-4027. [PMID: 37455675 DOI: 10.1002/sim.9845] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Revised: 05/19/2023] [Accepted: 07/01/2023] [Indexed: 07/18/2023]
Abstract
Receiver operating characteristic (ROC) curve is a popular tool to describe and compare the diagnostic accuracy of biomarkers when a binary-scale gold standard is available. However, there are many examples of diagnostic tests whose gold standards are continuous. Hence, Several extensions of receiver operating characteristic (ROC) curve are proposed to evaluate the diagnostic potential of biomarkers when the gold standard is continuous-scale. Moreover, in evaluating these biomarkers, it is often necessary to consider the effects of covariates on the diagnostic accuracy of the biomarker of interest. Covariates may include subject characteristics, expertise of the test operator, test procedures or aspects of specimen handling. Applying the covariate adjustment to the case that the gold standard is continuous is challenging and has not been addressed in the literature. To fill the gap, we propose two general testing frameworks to account for the covariates effect on diagnostic accuracy. Simulation studies are conducted to compare the proposed tests. Data from a study that assessed three types of imaging modalities with the purpose of detecting neoplastic colon polyps and cancers are used to illustrate the proposed methods.
Collapse
Affiliation(s)
- Mixia Wu
- College of Statistics and Data Science, Beijing University of Technology, Beijing, China
| | - Xian Sun
- Biostatistics and Data Management, Regeneron Pharmaceuticals, Bernards, New Jersey, USA
| | - Aiyi Liu
- Biostatistics and Bioinformatics Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, Maryland, USA
| | - Chenchen Peng
- College of Statistics and Data Science, Beijing University of Technology, Beijing, China
| | - Zhaohai Li
- Department of Statistics, Columbian College of Arts & Sciences, the George Washington University, Washington, USA
| |
Collapse
|
2
|
Waldera-Lupa DM, Poschmann G, Kirchgaessler N, Etemad-Parishanzadeh O, Baberg F, Brocksieper M, Seidel S, Kowalski T, Brunn A, Haghikia A, Gold R, Stefanski A, Deckert M, Schlegel U, Stühler K. A Multiplex Assay for the Stratification of Patients with Primary Central Nervous System Lymphoma Using Targeted Mass Spectrometry. Cancers (Basel) 2020; 12:cancers12071732. [PMID: 32610669 PMCID: PMC7407338 DOI: 10.3390/cancers12071732] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2020] [Revised: 06/25/2020] [Accepted: 06/26/2020] [Indexed: 12/12/2022] Open
Abstract
Primary central nervous system lymphomas (PCNSL) account for approximately 2% to 3% of all primary brain tumors. Until now, neuropathological tumor tissue analysis, most frequently gained by stereotactic biopsy, is still the diagnostic gold standard. Here, we rigorously analyzed two independent patient cohorts comprising the clinical entities PCNSL (n = 47), secondary central nervous system lymphomas (SCNSL; n = 13), multiple sclerosis (MS, n = 23), glioma (n = 10), other tumors (n = 17) and tumor-free controls (n = 21) by proteomic approaches. In total, we identified more than 1220 proteins in the cerebrospinal fluid (CSF) and validated eight candidate biomarkers by a peptide-centric approach in an independent patient cohort (n = 63). Thus, we obtained excellent diagnostic accuracy for the stratification between PCNSL, MS and glioma patients as well as tumor-free controls for three peptides originating from the three proteins VSIG4, GPNMB4 and APOC2. The combination of all three biomarker candidates resulted in diagnostic accuracy with an area under the curve (AUC) of 0.901 (PCNSL vs. MS), AUC of 0.953 (PCNSL vs. glioma) and AUC 0.850 (PCNSL vs. tumor-free control). In summary, the determination of VSIG4, GPNMB4 and APOC2 in CSF as novel biomarkers for supporting the diagnosis of PCNSL is suggested.
Collapse
Affiliation(s)
- Daniel M. Waldera-Lupa
- Institute of Molecular Medicine, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (D.M.W.-L.); (G.P.); (N.K.); (O.E.-P.); (F.B.); (M.B.)
| | - Gereon Poschmann
- Institute of Molecular Medicine, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (D.M.W.-L.); (G.P.); (N.K.); (O.E.-P.); (F.B.); (M.B.)
| | - Nina Kirchgaessler
- Institute of Molecular Medicine, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (D.M.W.-L.); (G.P.); (N.K.); (O.E.-P.); (F.B.); (M.B.)
| | - Omid Etemad-Parishanzadeh
- Institute of Molecular Medicine, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (D.M.W.-L.); (G.P.); (N.K.); (O.E.-P.); (F.B.); (M.B.)
| | - Falk Baberg
- Institute of Molecular Medicine, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (D.M.W.-L.); (G.P.); (N.K.); (O.E.-P.); (F.B.); (M.B.)
| | - Mareike Brocksieper
- Institute of Molecular Medicine, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (D.M.W.-L.); (G.P.); (N.K.); (O.E.-P.); (F.B.); (M.B.)
| | - Sabine Seidel
- Department of Neurology, Knappschaftskrankenhaus, Ruhr-University Bochum, 44789 Bochum, Germany; (S.S.); (T.K.); (U.S.)
| | - Thomas Kowalski
- Department of Neurology, Knappschaftskrankenhaus, Ruhr-University Bochum, 44789 Bochum, Germany; (S.S.); (T.K.); (U.S.)
| | - Anna Brunn
- Institute of Neuropathology, Faculty of Medicine and University Hospital Cologne, University of Cologne, 50937 Cologne, Germany; (A.B.); (M.D.)
| | - Aiden Haghikia
- Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum, 44789 Bochum, Germany; (A.H.); (R.G.)
| | - Ralf Gold
- Department of Neurology, St. Josef-Hospital, Ruhr-University Bochum, 44789 Bochum, Germany; (A.H.); (R.G.)
| | - Anja Stefanski
- Molecular Proteomics Laboratory, Biologisch-Medizinisches Forschungszentrum, Heinrich-Heine-University Düsseldorf, 40225 Düsseldorf, Germany;
| | - Martina Deckert
- Institute of Neuropathology, Faculty of Medicine and University Hospital Cologne, University of Cologne, 50937 Cologne, Germany; (A.B.); (M.D.)
| | - Uwe Schlegel
- Department of Neurology, Knappschaftskrankenhaus, Ruhr-University Bochum, 44789 Bochum, Germany; (S.S.); (T.K.); (U.S.)
| | - Kai Stühler
- Institute of Molecular Medicine, Medical Faculty, Heinrich-Heine-University, 40225 Düsseldorf, Germany; (D.M.W.-L.); (G.P.); (N.K.); (O.E.-P.); (F.B.); (M.B.)
- Molecular Proteomics Laboratory, Biologisch-Medizinisches Forschungszentrum, Heinrich-Heine-University Düsseldorf, 40225 Düsseldorf, Germany;
- Correspondence: ; Tel.: +49-211-811-3036
| |
Collapse
|
3
|
A machine learning approach for the prediction of pulmonary hypertension. PLoS One 2019; 14:e0224453. [PMID: 31652290 PMCID: PMC6814224 DOI: 10.1371/journal.pone.0224453] [Citation(s) in RCA: 31] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2019] [Accepted: 10/14/2019] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Machine learning (ML) is a powerful tool for identifying and structuring several informative variables for predictive tasks. Here, we investigated how ML algorithms may assist in echocardiographic pulmonary hypertension (PH) prediction, where current guidelines recommend integrating several echocardiographic parameters. METHODS In our database of 90 patients with invasively determined pulmonary artery pressure (PAP) with corresponding echocardiographic estimations of PAP obtained within 24 hours, we trained and applied five ML algorithms (random forest of classification trees, random forest of regression trees, lasso penalized logistic regression, boosted classification trees, support vector machines) using a 10 times 3-fold cross-validation (CV) scheme. RESULTS ML algorithms achieved high prediction accuracies: support vector machines (AUC 0.83; 95% CI 0.73-0.93), boosted classification trees (AUC 0.80; 95% CI 0.68-0.92), lasso penalized logistic regression (AUC 0.78; 95% CI 0.67-0.89), random forest of classification trees (AUC 0.85; 95% CI 0.75-0.95), random forest of regression trees (AUC 0.87; 95% CI 0.78-0.96). In contrast to the best of several conventional formulae (by Aduen et al.), this ML algorithm is based on several echocardiographic signs and feature selection, with estimated right atrial pressure (RAP) being of minor importance. CONCLUSIONS Using ML, we were able to predict pulmonary hypertension based on a broader set of echocardiographic data with little reliance on estimated RAP compared to an existing formula with non-inferior performance. With the conceptual advantages of a broader and unbiased selection and weighting of data our ML approach is suited for high level assistance in PH prediction.
Collapse
|
4
|
Ke BS, Chiang AJ, Chang YCI. Influence Analysis for the Area Under the Receiver Operating Characteristic Curve. J Biopharm Stat 2017; 28:722-734. [PMID: 28920760 DOI: 10.1080/10543406.2017.1377728] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Abstract
Classification measures play essential roles in the assessment and construction of classifiers. Hence, determining how to prevent these measures from being affected by individual observations has become an important problem. In this paper, we propose several indexes based on the influence function and the concept of local influence to identify influential observations that affect the estimate of the area under the receiver operating characteristic curve (AUC), an important and commonly used measure. Cumulative lift charts are also used to equipoise the disagreements among the proposed indexes. Both the AUC indexes and the graphical tools only rely on the classification scores, and both are applicable to classifiers that can produce real-valued classification scores. A real data set is used for illustration.
Collapse
Affiliation(s)
- Bo-Shiang Ke
- a Institute of Statistics, National Chiao Tung University , Hsinchu , Taiwan
| | - An Jen Chiang
- b Department of Obstetrics and Gynecology , Kaohsiung Veterans General Hospital , Kaohsiung , Taiwan
| | | |
Collapse
|
5
|
Shen W, Ning J, Yuan Y, Lok AS, Feng Z. Model-free scoring system for risk prediction with application to hepatocellular carcinoma study. Biometrics 2017; 74:239-248. [PMID: 28742219 DOI: 10.1111/biom.12750] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2015] [Revised: 06/01/2017] [Accepted: 06/01/2017] [Indexed: 12/19/2022]
Abstract
There is an increasing need to construct a risk-prediction scoring system for survival data and identify important risk factors (e.g., biomarkers) for patient screening and treatment recommendation. However, most existing methodologies either rely on strong model assumptions (e.g., proportional hazards) or only handle binary outcomes. In this article, we propose a flexible method that simultaneously selects important risk factors and identifies the optimal linear combination of risk factors by maximizing a pseudo-likelihood function based on the time-dependent area under the receiver operating characteristic curve. Our method is particularly useful for risk evaluation and recommendation of optimal subsequent treatments. We show that the proposed method has desirable theoretical properties, including asymptotic normality and the oracle property after variable selection. Numerical performance is evaluated on several simulation data sets and an application to hepatocellular carcinoma data.
Collapse
Affiliation(s)
- Weining Shen
- Department of Statistics, University of California, Irvine, California, U.S.A
| | - Jing Ning
- Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, Texas, U.S.A
| | - Ying Yuan
- Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, Texas, U.S.A
| | - Anna S Lok
- Division of Gastroenterology, University of Michigan Health System, Michigan, U.S.A
| | - Ziding Feng
- Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, Texas, U.S.A
| |
Collapse
|
8
|
Kang L, Liu A, Tian L. Linear combination methods to improve diagnostic/prognostic accuracy on future observations. Stat Methods Med Res 2013; 25:1359-80. [PMID: 23592714 DOI: 10.1177/0962280213481053] [Citation(s) in RCA: 35] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Multiple diagnostic tests or biomarkers can be combined to improve diagnostic accuracy. The problem of finding the optimal linear combinations of biomarkers to maximise the area under the receiver operating characteristic curve has been extensively addressed in the literature. The purpose of this article is threefold: (1) to provide an extensive review of the existing methods for biomarker combination; (2) to propose a new combination method, namely, the nonparametric stepwise approach; (3) to use leave-one-pair-out cross-validation method, instead of re-substitution method, which is overoptimistic and hence might lead to wrong conclusion, to empirically evaluate and compare the performance of different linear combination methods in yielding the largest area under receiver operating characteristic curve. A data set of Duchenne muscular dystrophy was analysed to illustrate the applications of the discussed combination methods.
Collapse
Affiliation(s)
- Le Kang
- Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, MD, USA
| | - Aiyi Liu
- Biostatistics and Bioinformatics Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development, Bethesda, MD, USA
| | - Lili Tian
- Department of Biostatistics, State University of New York at Buffalo, Buffalo, NY, USA
| |
Collapse
|