1
|
Allotey PA, Harel O. Modeling geostatistical incomplete spatially correlated survival data with applications to COVID-19 mortality in Ghana. SPATIAL STATISTICS 2023; 54:100730. [PMID: 36844103 PMCID: PMC9940474 DOI: 10.1016/j.spasta.2023.100730] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Revised: 01/01/2023] [Accepted: 02/14/2023] [Indexed: 06/18/2023]
Abstract
Survival models which incorporate frailties are common in time-to-event data collected over distinct spatial regions. While incomplete data are unavoidable and a common complication in statistical analysis of spatial survival research, most researchers still ignore the missing data problem. In this paper, we propose a geostatistical modeling approach for incomplete spatially correlated survival data. We achieve this by exploring missingness in outcome, covariates, and spatial locations. In the process, we analyze incomplete spatially-referenced survival data using a Weibull model for the baseline hazard function and correlated log-Gaussian frailties to model spatial correlation. We illustrate the proposed method with simulated data and an application to geo-referenced COVID-19 data from Ghana. There are several disagreements between parameter estimates and credible intervals widths obtained using our proposed approach and complete case analysis. Based on these findings, we argue that our approach provides more reliable parameter estimates and has higher predictive accuracy.
Collapse
Affiliation(s)
- Prince Addo Allotey
- Department of Statistics, University of Connecticut, 215 Glenbrook Rd Unit 4120, Storrs, 06269-4120, CT, USA
| | - Ofer Harel
- Department of Statistics, University of Connecticut, 215 Glenbrook Rd Unit 4120, Storrs, 06269-4120, CT, USA
| |
Collapse
|
2
|
Weinstein EJ, Ritchey ME, Lo Re V. Core concepts in pharmacoepidemiology: Validation of health outcomes of interest within real-world healthcare databases. Pharmacoepidemiol Drug Saf 2023; 32:1-8. [PMID: 36057777 PMCID: PMC9772105 DOI: 10.1002/pds.5537] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2022] [Revised: 08/09/2022] [Accepted: 08/19/2022] [Indexed: 02/06/2023]
Abstract
Real-world healthcare data, including administrative and electronic medical record databases, provide a rich source of data for the conduct of pharmacoepidemiologic studies but carry the potential for misclassification of health outcomes of interest (HOIs). Validation studies are important ways to quantify the degree of error associated with case-identifying algorithms for HOIs and are crucial for interpreting study findings within real-world data. This review provides a rationale, framework, and step-by-step approach to validating case-identifying algorithms for HOIs within healthcare databases. Key steps in validating a case-identifying algorithm within a healthcare database include: (1) selecting the appropriate health outcome; (2) determining the reference standard against which to validate the algorithm; (3) developing the algorithm using diagnosis codes, diagnostic tests or their results, procedures, drug therapies, patient-reported symptoms or diagnoses, or some combinations of these parameters; (4) selection of patients and sample sizes for validation; (5) collecting data to confirm the HOI; (6) confirming the HOI; and (7) assessing the algorithm's performance. Additional strategies for algorithm refinement and methods to correct for bias due to misclassification of outcomes are discussed. The review concludes by discussing factors affecting the transportability of case-identifying algorithms and the need for ongoing validation as data elements within healthcare databases, such as diagnosis codes, change over time or new variables, such as patient-generated health data, are included in these data sources.
Collapse
Affiliation(s)
- Erica J Weinstein
- Division of Infectious Diseases, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Pharmacoepidemiology Research and Training, Center for Clinical Epidemiology and Biostatistics, and Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Mary Elizabeth Ritchey
- Med Tech Epi, LLC, Philadelphia, PA, USA
- Center for Pharmacoepidemiology and Treatment Science, Rutgers University, New Brunswick, New Jersey, USA
| | - Vincent Lo Re
- Division of Infectious Diseases, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Pharmacoepidemiology Research and Training, Center for Clinical Epidemiology and Biostatistics, and Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| |
Collapse
|
3
|
Arifin WN, Yusof UK. Partial Verification Bias Correction Using Inverse Probability Bootstrap Sampling for Binary Diagnostic Tests. Diagnostics (Basel) 2022; 12:diagnostics12112839. [PMID: 36428900 PMCID: PMC9689704 DOI: 10.3390/diagnostics12112839] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 11/03/2022] [Accepted: 11/15/2022] [Indexed: 11/19/2022] Open
Abstract
In medical care, it is important to evaluate any new diagnostic test in the form of diagnostic accuracy studies. These new tests are compared to gold standard tests, where the performance of binary diagnostic tests is usually measured by sensitivity (Sn) and specificity (Sp). However, these accuracy measures are often biased owing to selective verification of the patients, known as partial verification bias (PVB). Inverse probability bootstrap (IPB) sampling is a general method to correct sampling bias in model-based analysis and produces debiased data for analysis. However, its utility in PVB correction has not been investigated before. The objective of this study was to investigate IPB in the context of PVB correction under the missing-at-random assumption for binary diagnostic tests. IPB was adapted for PVB correction, and tested and compared with existing methods using simulated and clinical data sets. The results indicated that IPB is accurate for Sn and Sp estimation as it showed low bias. However, IPB was less precise than existing methods as indicated by the higher standard error (SE). Despite this issue, it is recommended to use IPB when subsequent analysis with full data analytic methods is expected. Further studies must be conducted to reduce the SE.
Collapse
Affiliation(s)
- Wan Nor Arifin
- School of Computer Sciences, Universiti Sains Malaysia, Gelugor 11800, Pulau Pinang, Malaysia
- Biostatistics and Research Methodology Unit, School of Medical Sciences, Universiti Sains Malaysia, Kubang Kerian 16150, Kelantan, Malaysia
- Correspondence: (W.N.A.); (U.K.Y.)
| | - Umi Kalsom Yusof
- School of Computer Sciences, Universiti Sains Malaysia, Gelugor 11800, Pulau Pinang, Malaysia
- Correspondence: (W.N.A.); (U.K.Y.)
| |
Collapse
|
4
|
Day E, Eldred-Evans D, Prevost AT, Ahmed HU, Fiorentino F. Adjusting for verification bias in diagnostic accuracy measures when comparing multiple screening tests - an application to the IP1-PROSTAGRAM study. BMC Med Res Methodol 2022; 22:70. [PMID: 35300611 PMCID: PMC8932251 DOI: 10.1186/s12874-021-01481-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2021] [Accepted: 11/18/2021] [Indexed: 11/29/2022] Open
Abstract
Introduction Novel screening tests used to detect a target condition are compared against either a reference standard or other existing screening methods. However, as it is not always possible to apply the reference standard on the whole population under study, verification bias is introduced. Statistical methods exist to adjust estimates to account for this bias. We extend common methods to adjust for verification bias when multiple tests are compared to a reference standard using data from a prospective double blind screening study for prostate cancer. Methods Begg and Greenes method and multiple imputation are extended to include the results of multiple screening tests which determine condition verification status. These two methods are compared to the complete case analysis using the IP1-PROSTAGRAM study data. IP1-PROSTAGRAM used a paired-cohort double-blind design to evaluate the use of imaging as alternative tests to screen for prostate cancer, compared to a blood test called prostate specific antigen (PSA). Participants with positive imaging (index) and/or PSA (control) underwent a prostate biopsy (reference standard). Results When comparing complete case results to Begg and Greenes and methods of multiple imputation there is a statistically significant increase in the specificity estimates for all screening tests. Sensitivity estimates remained similar across the methods, with completely overlapping 95% confidence intervals. Negative predictive value (NPV) estimates were higher when adjusting for verification bias, compared to complete case analysis, even though the 95% confidence intervals overlap. Positive predictive value (PPV) estimates were similar across all methods. Conclusion Statistical methods are required to adjust for verification bias in accuracy estimates of screening tests. Expanding Begg and Greenes method to include multiple screening tests can be computationally intensive, hence multiple imputation is recommended, especially as it can be modified for low prevalence of the target condition. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-021-01481-w.
Collapse
Affiliation(s)
- Emily Day
- Imperial Clinical Trials Unit, Imperial College London, London, UK
| | - David Eldred-Evans
- Imperial Prostate, Division of Surgery, Department of Surgery and Cancer, Faculty of Medicine, Imperial College London, London, UK
| | - A Toby Prevost
- Nightingale-Saunders Unit, King's Clinical Trials Unit, King's College London, London, UK
| | - Hashim U Ahmed
- Imperial Prostate, Division of Surgery, Department of Surgery and Cancer, Faculty of Medicine, Imperial College London, London, UK.,Imperial Urology, Imperial College Healthcare NHS Trust, London, UK
| | - Francesca Fiorentino
- Imperial Clinical Trials Unit, Imperial College London, London, UK. .,Nightingale-Saunders Unit, King's Clinical Trials Unit, King's College London, London, UK. .,Division of Surgery, Imperial College London, St Mary's Hospital, Praed Street, London, W2 1NY, UK.
| |
Collapse
|
5
|
Arifin WN, Yusof UK. Correcting for partial verification bias in diagnostic accuracy studies: A tutorial using R. Stat Med 2022; 41:1709-1727. [PMID: 35043447 DOI: 10.1002/sim.9311] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2021] [Revised: 12/19/2021] [Accepted: 12/21/2021] [Indexed: 11/06/2022]
Abstract
Diagnostic tests play a crucial role in medical care. Thus any new diagnostic tests must undergo a thorough evaluation. New diagnostic tests are evaluated in comparison with the respective gold standard tests. The performance of binary diagnostic tests is quantified by accuracy measures, with sensitivity and specificity being the most important measures. In any diagnostic accuracy study, the estimates of these measures are often biased owing to selective verification of the patients, which is referred to as partial verification bias. Several methods for correcting partial verification bias are available depending on the scale of the index test, target outcome, and missing data mechanism. However, these are not easily accessible to the researchers due to the complexity of the methods. This article aims to provide a brief overview of the methods available to correct for partial verification bias involving a binary diagnostic test and provide a practical tutorial on how to implement the methods using the statistical programming language R.
Collapse
Affiliation(s)
- Wan Nor Arifin
- School of Computer Sciences, Universiti Sains Malaysia, Pulau Pinang, Malaysia.,Biostatistics and Research Methodology Unit, School of Medical Sciences, Universiti Sains Malaysia, Kelantan, Malaysia
| | - Umi Kalsom Yusof
- School of Computer Sciences, Universiti Sains Malaysia, Pulau Pinang, Malaysia
| |
Collapse
|
6
|
Lee CH, Wang H. Multiple imputation confidence intervals for the mean of the discrete distributions for incomplete data. Stat Med 2021; 41:1172-1190. [PMID: 34786744 DOI: 10.1002/sim.9254] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Revised: 10/10/2021] [Accepted: 10/19/2021] [Indexed: 11/08/2022]
Abstract
Confidence intervals for the mean of discrete exponential families are widely used in many applications. Since missing data are commonly encountered, the interval estimation for incomplete data is an important problem. The performances of the existing multiple imputation confidence intervals are unsatisfactory. We propose modified multiple imputation confidence intervals to improve the existing confidence intervals for the mean of the discrete exponential families with quadratic variance functions. A simulation study shows that the coverage probabilities of the modified confidence intervals are closer to the nominal level than the existing confidence intervals when the true mean is near the boundaries of the parameter space. These confidence intervals are also illustrated with real data examples.
Collapse
Affiliation(s)
- Chung-Han Lee
- Institute of Statistics, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
| | - Hsiuying Wang
- Institute of Statistics, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
| |
Collapse
|
7
|
Estimation of the Average Kappa Coefficient of a Binary Diagnostic Test in the Presence of Partial Verification. MATHEMATICS 2021. [DOI: 10.3390/math9141694] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
The average kappa coefficient of a binary diagnostic test is a measure of the beyond-chance average agreement between the binary diagnostic test and the gold standard, and it depends on the sensitivity and specificity of the diagnostic test and on disease prevalence. In this manuscript the estimation of the average kappa coefficient of a diagnostic test in the presence of verification bias is studied. Confidence intervals for the average kappa coefficient are studied applying the methods of maximum likelihood and multiple imputation by chained equations. Simulation experiments have been carried out to study the asymptotic behaviors of the proposed intervals, given some application rules. The results obtained in our simulation experiments have shown that the multiple imputation by chained equations method provides better results than the maximum likelihood method. A function has been written in R to estimate the average kappa coefficient by applying multiple imputation. The results have been applied to the diagnosis of liver disease.
Collapse
|
8
|
Umemneku Chikere CM, Wilson K, Graziadio S, Vale L, Allen AJ. Diagnostic test evaluation methodology: A systematic review of methods employed to evaluate diagnostic tests in the absence of gold standard - An update. PLoS One 2019; 14:e0223832. [PMID: 31603953 PMCID: PMC6788703 DOI: 10.1371/journal.pone.0223832] [Citation(s) in RCA: 101] [Impact Index Per Article: 20.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2019] [Accepted: 09/29/2019] [Indexed: 12/29/2022] Open
Abstract
OBJECTIVE To systematically review methods developed and employed to evaluate the diagnostic accuracy of medical test when there is a missing or no gold standard. STUDY DESIGN AND SETTINGS Articles that proposed or applied any methods to evaluate the diagnostic accuracy of medical test(s) in the absence of gold standard were reviewed. The protocol for this review was registered in PROSPERO (CRD42018089349). RESULTS Identified methods were classified into four main groups: methods employed when there is a missing gold standard; correction methods (which make adjustment for an imperfect reference standard with known diagnostic accuracy measures); methods employed to evaluate a medical test using multiple imperfect reference standards; and other methods, like agreement studies, and a mixed group of alternative study designs. Fifty-one statistical methods were identified from the review that were developed to evaluate medical test(s) when the true disease status of some participants is unverified with the gold standard. Seven correction methods were identified and four methods were identified to evaluate medical test(s) using multiple imperfect reference standards. Flow-diagrams were developed to guide the selection of appropriate methods. CONCLUSION Various methods have been proposed to evaluate medical test(s) in the absence of a gold standard for some or all participants in a diagnostic accuracy study. These methods depend on the availability of the gold standard, its' application to the participants in the study and the availability of alternative reference standard(s). The clinical application of some of these methods, especially methods developed when there is missing gold standard is however limited. This may be due to the complexity of these methods and/or a disconnection between the fields of expertise of those who develop (e.g. mathematicians) and those who employ the methods (e.g. clinical researchers). This review aims to help close this gap with our classification and guidance tools.
Collapse
Affiliation(s)
- Chinyereugo M. Umemneku Chikere
- Institute of Health & Society, Faculty of Medical Sciences Newcastle University, Newcastle upon Tyne, England, United Kingdom
| | - Kevin Wilson
- School of Mathematics, Statistics and Physics, Newcastle University, Newcastle upon Tyne, England, United Kingdom
| | - Sara Graziadio
- National Institute for Health Research, Newcastle In Vitro Diagnostics Co-operative, Newcastle upon Tyne Hospitals National Health Services Foundation Trust, Newcastle upon Tyne, England, United Kingdom
| | - Luke Vale
- Institute of Health & Society, Faculty of Medical Sciences Newcastle University, Newcastle upon Tyne, England, United Kingdom
| | - A. Joy Allen
- National Institute for Health Research, Newcastle In Vitro Diagnostics Co-operative, Newcastle University, Newcastle upon Tyne, England, United Kingdom
| |
Collapse
|
9
|
Consequences of Model Misspecification for Maximum Likelihood Estimation with Missing Data. ECONOMETRICS 2019. [DOI: 10.3390/econometrics7030037] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Researchers are often faced with the challenge of developing statistical models with incomplete data. Exacerbating this situation is the possibility that either the researcher’s complete-data model or the model of the missing-data mechanism is misspecified. In this article, we create a formal theoretical framework for developing statistical models and detecting model misspecification in the presence of incomplete data where maximum likelihood estimates are obtained by maximizing the observable-data likelihood function when the missing-data mechanism is assumed ignorable. First, we provide sufficient regularity conditions on the researcher’s complete-data model to characterize the asymptotic behavior of maximum likelihood estimates in the simultaneous presence of both missing data and model misspecification. These results are then used to derive robust hypothesis testing methods for possibly misspecified models in the presence of Missing at Random (MAR) or Missing Not at Random (MNAR) missing data. Second, we introduce a method for the detection of model misspecification in missing data problems using recently developed Generalized Information Matrix Tests (GIMT). Third, we identify regularity conditions for the Missing Information Principle (MIP) to hold in the presence of model misspecification so as to provide useful computational covariance matrix estimation formulas. Fourth, we provide regularity conditions that ensure the observable-data expected negative log-likelihood function is convex in the presence of partially observable data when the amount of missingness is sufficiently small and the complete-data likelihood is convex. Fifth, we show that when the researcher has correctly specified a complete-data model with a convex negative likelihood function and an ignorable missing-data mechanism, then its strict local minimizer is the true parameter value for the complete-data model when the amount of missingness is sufficiently small. Our results thus provide new robust estimation, inference, and specification analysis methods for developing statistical models with incomplete data.
Collapse
|
10
|
Cho H, Matthews GJ, Harel O. Confidence Intervals for the Area Under the Receiver Operating Characteristic Curve in the Presence of Ignorable Missing Data. Int Stat Rev 2019; 87:152-177. [PMID: 31007356 PMCID: PMC6472951 DOI: 10.1111/insr.12277] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2017] [Accepted: 05/31/2018] [Indexed: 11/30/2022]
Abstract
Receiver operating characteristic curves are widely used as a measure of accuracy of diagnostic tests and can be summarised using the area under the receiver operating characteristic curve (AUC). Often, it is useful to construct a confidence interval for the AUC; however, because there are a number of different proposed methods to measure variance of the AUC, there are thus many different resulting methods for constructing these intervals. In this article, we compare different methods of constructing Wald-type confidence interval in the presence of missing data where the missingness mechanism is ignorable. We find that constructing confidence intervals using multiple imputation based on logistic regression gives the most robust coverage probability and the choice of confidence interval method is less important. However, when missingness rate is less severe (e.g. less than 70%), we recommend using Newcombe's Wald method for constructing confidence intervals along with multiple imputation using predictive mean matching.
Collapse
Affiliation(s)
- Hunyong Cho
- Department of Biostatistics, University of North Carolina at Chapel Hill, 135 Dauer Drive, Chapel Hill, NC 27599, USA
| | - Gregory J Matthews
- Department of Mathematics and Statistics, Loyola University Chicago, 1032 W. Sheridan Road, Chicago, IL 60660, USA
| | - Ofer Harel
- Department of Statistics, University of Connecticut, 215 Glenbrook Road U-4120, Storrs, CT 06269, USA
| |
Collapse
|
11
|
Lott A, Reiter JP. Wilson Confidence Intervals for Binomial Proportions With Multiple Imputation for Missing Data. AM STAT 2018. [DOI: 10.1080/00031305.2018.1473796] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/16/2022]
Affiliation(s)
- Anne Lott
- Department of Statistical Science, Duke University, Durham, NC
| | | |
Collapse
|
12
|
Montero-Alonso MA, Roldán-Nofuentes JA. Approximate confidence intervals for the likelihood ratios of a binary diagnostic test in the presence of partial disease verification. J Biopharm Stat 2018; 29:56-81. [PMID: 29584541 DOI: 10.1080/10543406.2018.1452025] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
Abstract
The classic parameters used to assess the accuracy of a binary diagnostic test (BDT) are sensitivity and specificity. Other parameters used to describe the performance of a BDT are likelihood ratios (LRs). The LRs depend on the sensitivity and the specificity of the diagnostic test, and they reflect how much greater the probability of a positive or negative diagnostic test result for individuals with the disease than that for the individuals without the disease. In this study, several confidence intervals are studied for the LRs of a BDT in the presence of missing data. Two confidence intervals were studied through the method of maximum likelihood and seven confidence intervals were studied by applying the multiple imputation by chained equations method. A program in R software has been written that allows us to solve the estimation problem posed. The results obtained have been applied to the two real examples.
Collapse
Affiliation(s)
- M A Montero-Alonso
- a Department of Statistics (Biostatistics), School of Medicine , University of Granada , Granada, Spain
| | - J A Roldán-Nofuentes
- a Department of Statistics (Biostatistics), School of Medicine , University of Granada , Granada, Spain
| |
Collapse
|
13
|
Shan G, Zhang H, Jiang T. Determining sample size for a binary diagnostic test in the presence of verification bias. J Biopharm Stat 2018; 28:1193-1202. [PMID: 29553878 DOI: 10.1080/10543406.2018.1452029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
Abstract
To compare a new binary diagnostic test with the gold standard, sensitivity and specificity are the two common measurements used to evaluate the new test. When not all the patients are verified by the gold standard due to time, budget, or cost considerations, several approaches have been proposed to compute sample size for such studies under the assumption of missing completely at random. However, the majority of them are based on asymptotic approaches that generally do not guarantee the type I and II error rates, and the remaining approaches use exact binomial distributions in sample size calculation but only the verified samples are used. In this article, for a study with verification bias, we propose computing exact sample sizes by using all the samples. The proposed approach is compared with the existing exact approach that compute sample size by using verified samples only, and the results show that the proposed approach requires fewer participants than the competitor.
Collapse
Affiliation(s)
- Guogen Shan
- a Epidemiology and Biostatistics Program, Department of Environmental and Occupational Health, School of Community Health Sciences , University of Nevada Las Vegas , Las Vegas , NV , USA
| | - Hua Zhang
- b School of Computer and Information Engineering , Zhejiang Gongshang University , Hangzhou , Zhejiang , China
| | - Tao Jiang
- c Department of Statistics , Zhejiang Gongshang University , Hangzhou , Zhejiang , China
| |
Collapse
|
14
|
The External Validity of Prediction Models for the Diagnosis of Obstructive Coronary Artery Disease in Patients With Stable Chest Pain: Insights From the PROMISE Trial. JACC Cardiovasc Imaging 2017. [PMID: 28624401 DOI: 10.1016/j.jcmg.2017.02.020] [Citation(s) in RCA: 40] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
Abstract
OBJECTIVES This study sought to externally validate prediction models for the presence of obstructive coronary artery disease (CAD). BACKGROUND A better assessment of the probability of CAD may improve the identification of patients who benefit from noninvasive testing. METHODS Stable chest pain patients from the PROMISE (Prospective Multicenter Imaging Study for Evaluation of Chest Pain) trial with computed tomography angiography (CTA) or invasive coronary angiography (ICA) were included. The authors assumed that patients with CTA showing 0% stenosis and a coronary artery calcium (CAC) score of 0 were free of obstructive CAD (≥50% stenosis) on ICA, and they multiply imputed missing ICA results based on clinical variables and CTA results. Predicted CAD probabilities were calculated using published coefficients for 3 models: basic model (age, sex, chest pain type), clinical model (basic model + diabetes, hypertension, dyslipidemia, and smoking), and clinical + CAC score model. The authors assessed discrimination and calibration, and compared published effects with observed predictor effects. RESULTS In 3,468 patients (1,805 women; mean 60 years of age; 779 [23%] with obstructive CAD on CTA), the models demonstrated moderate-good discrimination, with C-statistics of 0.69 (95% confidence interval [CI]: 0.67 to 0.72), 0.72 (95% CI: 0.69 to 0.74), and 0.86 (95% CI: 0.85 to 0.88) for the basic, clinical, and clinical + CAC score models, respectively. Calibration was satisfactory although typical chest pain and diabetes were less predictive and CAC score was more predictive than was suggested by the models. Among the 31% of patients for whom the clinical model predicted a low (≤10%) probability of CAD, actual prevalence was 7%; among the 48% for whom the clinical + CAC score model predicted a low probability the observed prevalence was 2%. In 2 sensitivity analyses excluding imputed data, similar results were obtained using CTA as the outcome, whereas in those who underwent ICA the models significantly underestimated CAD probability. CONCLUSIONS Existing clinical prediction models can identify patients with a low probability of obstructive CAD. Obstructive CAD on ICA was imputed for 61% of patients; hence, further validation is necessary.
Collapse
|
15
|
The Quantitative Science of Evaluating Imaging Evidence. JACC Cardiovasc Imaging 2017; 10:264-275. [DOI: 10.1016/j.jcmg.2016.12.010] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/20/2016] [Accepted: 12/21/2016] [Indexed: 01/08/2023]
|
16
|
Karch A, Koch A, Zapf A, Zerr I, Karch A. Partial verification bias and incorporation bias affected accuracy estimates of diagnostic studies for biomarkers that were part of an existing composite gold standard. J Clin Epidemiol 2016; 78:73-82. [PMID: 27107877 DOI: 10.1016/j.jclinepi.2016.03.022] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2015] [Revised: 03/08/2016] [Accepted: 03/30/2016] [Indexed: 11/29/2022]
Abstract
OBJECTIVE To investigate how choice of gold standard biases estimates of sensitivity and specificity in studies reassessing the diagnostic accuracy of biomarkers that are already part of a lifetime composite gold standard (CGS). STUDY DESIGN AND SETTING We performed a simulation study based on the real-life example of the biomarker "protein 14-3-3" used for diagnosing Creutzfeldt-Jakob disease. Three different types of gold standard were compared: perfect gold standard "autopsy" (available in a small fraction only; prone to partial verification bias), lifetime CGS (including the biomarker under investigation; prone to incorporation bias), and "best available" gold standard (autopsy if available, otherwise CGS). RESULTS Sensitivity was unbiased when comparing 14-3-3 with autopsy but overestimated when using CGS or "best available" gold standard. Specificity of 14-3-3 was underestimated in scenarios comparing 14-3-3 with autopsy (up to 24%). In contrast, overestimation (up to 20%) was observed for specificity compared with CGS; this could be reduced to 0-10% when using the "best available" gold standard. CONCLUSION Choice of gold standard affects considerably estimates of diagnostic accuracy. Using the "best available" gold standard (autopsy where available, otherwise CGS) leads to valid estimates of specificity, whereas sensitivity is estimated best when tested against autopsy alone.
Collapse
Affiliation(s)
- Annika Karch
- Institute for Biostatistics, Hannover Medical School, Carl-Neuberg-Str. 1, 30625 Hannover, Germany
| | - Armin Koch
- Institute for Biostatistics, Hannover Medical School, Carl-Neuberg-Str. 1, 30625 Hannover, Germany
| | - Antonia Zapf
- Department of Medical Statistics, University Göttingen, Humboldtallee 32, 37073 Göttingen, Germany
| | - Inga Zerr
- National Reference Centre for TSE, Department for Neurology, University Göttingen, Robert-Koch-Str. 40, 37075 Göttingen, Germany
| | - André Karch
- National Reference Centre for TSE, Department for Neurology, University Göttingen, Robert-Koch-Str. 40, 37075 Göttingen, Germany; Department of Epidemiology, Helmholtz Centre for Infection Research, Inhoffenstr. 7, 38124 Braunschweig, Germany; German Center for Infection Research, Hannover-Braunschweig Site, Feodor-Lynen-Str. 7, 30625 Hannover, Germany.
| |
Collapse
|
17
|
Ma X, Suri MFK, Chu H. A trivariate meta-analysis of diagnostic studies accounting for prevalence and non-evaluable subjects: re-evaluation of the meta-analysis of coronary CT angiography studies. BMC Med Res Methodol 2014; 14:128. [PMID: 25475705 PMCID: PMC4280699 DOI: 10.1186/1471-2288-14-128] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2014] [Accepted: 11/18/2014] [Indexed: 01/28/2023] Open
Abstract
BACKGROUND A recent paper proposed an intent-to-diagnose approach to handle non-evaluable index test results and discussed several alternative approaches, with an application to the meta-analysis of coronary CT angiography diagnostic accuracy studies. However, no simulation studies have been conducted to test the performance of the methods. METHODS We propose an extended trivariate generalized linear mixed model (TGLMM) to handle non-evaluable index test results. The performance of the intent-to-diagnose approach, the alternative approaches and the extended TGLMM approach is examined by extensive simulation studies. The meta-analysis of coronary CT angiography diagnostic accuracy studies is re-evaluated by the extended TGLMM. RESULTS Simulation studies showed that the intent-to-diagnose approach under-estimate sensitivity and specificity. Under the missing at random (MAR) assumption, the TGLMM gives nearly unbiased estimates of test accuracy indices and disease prevalence. After applying the TGLMM approach to re-evaluate the coronary CT angiography meta-analysis, overall median sensitivity is 0.98 (0.967, 0.993), specificity is 0.875 (0.827, 0.923) and disease prevalence is 0.478 (0.379, 0.577). CONCLUSIONS Under MAR assumption, the intent-to-diagnose approach under-estimate both sensitivity and specificity, while the extended TGLMM gives nearly unbiased estimates of sensitivity, specificity and prevalence. We recommend the extended TGLMM to handle non-evaluable index test subjects.
Collapse
Affiliation(s)
- Xiaoye Ma
- Division of Biostatistics, School of Public Health, University of Minnesota, A460 Mayo Building, MMC 303, 420 Delaware St, SE, 55455 Minneapolis, MN, USA.
| | | | | |
Collapse
|
18
|
Karakaya J, Karabulut E, Yucel RM. Sensitivity to imputation models and assumptions in receiver operating characteristic analysis with incomplete data. J STAT COMPUT SIM 2014; 85:3498-3511. [PMID: 26379316 DOI: 10.1080/00949655.2014.983111] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
Modern statistical methods using incomplete data have been increasingly applied in a wide variety of substantive problems. Similarly, receiver operating characteristic (ROC) analysis, a method used in evaluating diagnostic tests or biomarkers in medical research, has also been increasingly popular problem in both its development and application. While missing-data methods have been applied in ROC analysis, the impact of model mis-specification and/or assumptions (e.g. missing at random) underlying the missing data has not been thoroughly studied. In this work, we study the performance of multiple imputation (MI) inference in ROC analysis. Particularly, we investigate parametric and non-parametric techniques for MI inference under common missingness mechanisms. Depending on the coherency of the imputation model with the underlying data generation mechanism, our results show that MI generally leads to well-calibrated inferences under ignorable missingness mechanisms.
Collapse
Affiliation(s)
- Jale Karakaya
- Department of Biostatistics, Faculty of Medicine, Hacettepe University, Sihhiye, Ankara, Turkey
| | - Erdem Karabulut
- Department of Biostatistics, Faculty of Medicine, Hacettepe University, Sihhiye, Ankara, Turkey
| | - Recai M Yucel
- Department of Epidemiology and Biostatistics, School of Public Health, University at Albany, SUNY, One University Place, Rensselaer, NY, USA
| |
Collapse
|
19
|
Mutz R, Bornmann L, Daniel H. Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex‐ante and ex‐post peer evaluation data from the
A
ustrian science fund. J Assoc Inf Sci Technol 2014. [DOI: 10.1002/asi.23315] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Rüdiger Mutz
- Professorship for Social Psychology and Research on Higher Education ETH Zurich Muehlegasse 21 8001 Zurich Switzerland
| | - Lutz Bornmann
- Administrative Headquarters of the Max Planck Society Hofgartenstrasse 8 D‐80539 Munich Germany
| | - Hans‐Dieter Daniel
- Professorship for Social Psychology and Research on Higher Education ETH Zurich Muehlegasse 21 8001 Zurich Switzerland
- Evaluation Office University of Zurich Muehlegasse 21 8001 Zurich Switzerland
| |
Collapse
|
20
|
Ma X, Chen Y, Cole SR, Chu H. A hybrid Bayesian hierarchical model combining cohort and case-control studies for meta-analysis of diagnostic tests: Accounting for partial verification bias. Stat Methods Med Res 2014; 25:3015-3037. [PMID: 24862512 DOI: 10.1177/0962280214536703] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
To account for between-study heterogeneity in meta-analysis of diagnostic accuracy studies, bivariate random effects models have been recommended to jointly model the sensitivities and specificities. As study design and population vary, the definition of disease status or severity could differ across studies. Consequently, sensitivity and specificity may be correlated with disease prevalence. To account for this dependence, a trivariate random effects model had been proposed. However, the proposed approach can only include cohort studies with information estimating study-specific disease prevalence. In addition, some diagnostic accuracy studies only select a subset of samples to be verified by the reference test. It is known that ignoring unverified subjects may lead to partial verification bias in the estimation of prevalence, sensitivities, and specificities in a single study. However, the impact of this bias on a meta-analysis has not been investigated. In this paper, we propose a novel hybrid Bayesian hierarchical model combining cohort and case-control studies and correcting partial verification bias at the same time. We investigate the performance of the proposed methods through a set of simulation studies. Two case studies on assessing the diagnostic accuracy of gadolinium-enhanced magnetic resonance imaging in detecting lymph node metastases and of adrenal fluorine-18 fluorodeoxyglucose positron emission tomography in characterizing adrenal masses are presented.
Collapse
Affiliation(s)
- Xiaoye Ma
- Division of Biostatistics, School of Public Health, The University of Minnesota, Minneapolis, USA
| | - Yong Chen
- Division of Biostatistics, School of Public Health, The University of Texas, Houston, USA
| | - Stephen R Cole
- Department of Epidemiology, University of North Carolina, Chapel Hill, USA
| | - Haitao Chu
- Division of Biostatistics, School of Public Health, The University of Minnesota, Minneapolis, USA
| |
Collapse
|
21
|
Ünal İ, Burgut HR. Verification bias on sensitivity and specificity measurements in diagnostic medicine: a comparison of some approaches used for correction. J Appl Stat 2013. [DOI: 10.1080/02664763.2013.862217] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
22
|
Steele R, Hudson M, Lo E, Baron M. Clinical decision rule to predict the presence of interstitial lung disease in systemic sclerosis. Arthritis Care Res (Hoboken) 2012; 64:519-24. [PMID: 22213733 DOI: 10.1002/acr.21583] [Citation(s) in RCA: 106] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
OBJECTIVE To develop a clinical decision rule to predict the presence of interstitial lung disease (ILD) in systemic sclerosis (SSc; scleroderma) and to estimate the prevalence of SSc-ILD. METHODS Patient data were extracted from the Canadian Scleroderma Research Group registry. Three algorithms for the clinical decision rule were considered based on lung auscultation, chest radiography (CXR), and % predicted forced vital capacity (FVC). High-resolution computed tomography (HRCT) scans were used as the gold standard to determine the diagnostic properties of the 3 algorithms. Multiple imputation was used to impute HRCT data when missing, thereby avoiding bias due to differential referral for HRCT. RESULTS This study included 1,168 patients. Of the patients with HRCT scans, 65% had evidence of ILD, compared to 26% by physical examination and 22% by CXR. The FVC of those who did not have HRCT was 8.8% greater than those who did (95% confidence interval [95% CI] 6.0-11.6%). Algorithm A, which identified the presence of ILD based on crackles on lung auscultation and/or findings on CXR, had a likelihood ratio of 3.9, compared to 3.2 for Algorithm B (which included patients with FVC <70%) and 2.2 for Algorithm C (which included patients with FVC <80%). The prevalence of ILD in the cohort was estimated to be 52% (95% CI 46-59%). CONCLUSION We developed a simple clinical decision rule to predict SSc-ILD with good test characteristics. The prevalence of ILD in a large, unselected SSc cohort was estimated to be 52%.
Collapse
Affiliation(s)
- Russell Steele
- Jewish General Hospital and McGill University, Montreal, Quebec, Canada
| | | | | | | | | |
Collapse
|
23
|
de Groot JAH, Dendukuri N, Janssen KJM, Reitsma JB, Brophy J, Joseph L, Bossuyt PMM, Moons KGM. Adjusting for partial verification or workup bias in meta-analyses of diagnostic accuracy studies. Am J Epidemiol 2012; 175:847-53. [PMID: 22422923 DOI: 10.1093/aje/kwr383] [Citation(s) in RCA: 19] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022] Open
Abstract
A key requirement in the design of diagnostic accuracy studies is that all study participants receive both the test under evaluation and the reference standard test. For a variety of practical and ethical reasons, sometimes only a proportion of patients receive the reference standard, which can bias the accuracy estimates. Numerous methods have been described for correcting this partial verification bias or workup bias in individual studies. In this article, the authors describe a Bayesian method for obtaining adjusted results from a diagnostic meta-analysis when partial verification or workup bias is present in a subset of the primary studies. The method corrects for verification bias without having to exclude primary studies with verification bias, thus preserving the main advantages of a meta-analysis: increased precision and better generalizability. The results of this method are compared with the existing methods for dealing with verification bias in diagnostic meta-analyses. For illustration, the authors use empirical data from a systematic review of studies of the accuracy of the immunohistochemistry test for diagnosis of human epidermal growth factor receptor 2 status in breast cancer patients.
Collapse
Affiliation(s)
- Joris A H de Groot
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508GA Utrecht, the Netherlands.
| | | | | | | | | | | | | | | |
Collapse
|
24
|
Yan G, Greene T. Statistical analysis and design for estimating accuracy in clinical-center classification of cause-specific clinical events in clinical trials. Clin Trials 2012; 8:571-80. [PMID: 22013170 DOI: 10.1177/1740774511411320] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
BACKGROUND When cause-specific clinical events (for instance, hospitalization for cardiac disease) are used as the primary or key secondary outcomes, it is important to assess how accurately these events have been classified and estimate the error-corrected treatment effects to ensure validity. However, it may not be feasible to verify cause classification for every event in large clinical trials. PURPOSE We present statistical methods for the design and analysis of outcome-classification accuracy and for estimating error-corrected treatment effects. METHODS Using the Hemodialysis (HEMO) Study, in which primary causes were designated for all 7822 hospitalizations by the 15 participating clinical centers - but only two subsets were audited by the Outcome Committee - we applied existing methods to obtain unbiased estimates of the sensitivity and specificity of clinical-center classifications. The multiple imputation method was used to correct for the misclassification of events. We then examined how trial results were affected by three methods of event classification: unaudited, imputed, and adjudicated. RESULTS We applied a three-step procedure to extend the results for the two subsets of audited events to estimate the sensitivity and specificity for the complete set. Finite population sample size formulas were developed for designing the quality control sample. Based on the HEMO analysis, the estimate of the intervention effect using the unaudited outcome was biased; the bias was reduced using the outcome corrected by imputation. LIMITATIONS The methods are limited to situations in which there are clinical-center classifications for all clinical events but only partial availability of reference standard classifications, and the verification process does not depend on the true event cause or other unobserved information. CONCLUSIONS Designing a quality control study to estimate the accuracy of outcome classification is important. The multiple imputation method can be used to correct for errors in outcome classification and to estimate the error-corrected treatment effect. Trial results need to be reexamined using the error-corrected outcome.
Collapse
Affiliation(s)
- Guofen Yan
- Department of Public Health Sciences, University of Virginia, Charlottesville, VA 22908-0717, USA.
| | | |
Collapse
|
25
|
Richardson ML, Petscavage JM. Verification bias: an under-recognized source of error in assessing the efficacy of MRI of the meniscii. Acad Radiol 2011; 18:1376-81. [PMID: 21852159 DOI: 10.1016/j.acra.2011.06.014] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2011] [Revised: 06/14/2011] [Accepted: 06/15/2011] [Indexed: 11/26/2022]
Abstract
RATIONALE AND OBJECTIVES The sensitivity and specificity of magnetic resonance imaging (MRI) for diagnosis of meniscal tears has been studied extensively, with tears usually verified by surgery. However, surgically unverified cases are often not considered in these studies, leading to verification bias, which can falsely increase the sensitivity and decrease the specificity estimates. Our study suggests that such bias may be very common in the meniscal MRI literature, and illustrates techniques to detect and correct for such bias. MATERIALS AND METHODS PubMed was searched for articles estimating sensitivity and specificity of MRI for meniscal tears. These were assessed for verification bias, deemed potentially present if a study included any patients whose MRI findings were not surgically verified. Retrospective global sensitivity analysis (GSA) was performed when possible. RESULTS Thirty-nine of the 314 studies retrieved from PubMed specifically dealt with meniscal tears. All 39 included unverified patients, and hence, potential verification bias. Only seven articles included sufficient information to perform GSA. Of these, one showed definite verification bias, two showed no bias, and four others showed bias within certain ranges of disease prevalence. Only 9 of 39 acknowledged the possibility of verification bias. CONCLUSION Verification bias is underrecognized and potentially common in published estimates of the sensitivity and specificity of MRI for the diagnosis of meniscal tears. When possible, it should be avoided by proper study design. If unavoidable, it should be acknowledged. Investigators should tabulate unverified as well as verified data. Finally, verification bias should be estimated; if present, corrected estimates of sensitivity and specificity should be used. Our online web-based calculator makes this process relatively easy.
Collapse
|
26
|
Voils CI, Olsen MK, Williams JW, Impact Study Investigators. Identifying depressed older adults in primary care: a secondary analysis of a multisite randomized controlled trial. PRIMARY CARE COMPANION TO THE JOURNAL OF CLINICAL PSYCHIATRY 2011; 10:9-14. [PMID: 18311416 DOI: 10.4088/pcc.v10n0103] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/01/2007] [Accepted: 11/26/2007] [Indexed: 10/20/2022]
Abstract
OBJECTIVE To determine whether a subset of depressive symptoms could be identified to facilitate diagnosis of depression in older adults in primary care. METHOD Secondary analysis was conducted on 898 participants aged 60 years or older with major depressive disorder and/or dysthymic disorder (according to DSM-IV criteria) who participated in the Improving Mood-Promoting Access to Collaborative Treatment (IMPACT) study, a multisite, randomized trial of collaborative care for depression (recruitment from July 1999 to August 2001). Linear regression was used to identify a core subset of depressive symptoms associated with decreased social, physical, and mental functioning. The sensitivity and specificity, adjusting for selection bias, were evaluated for these symptoms. The sensitivity and specificity of a second subset of 4 depressive symptoms previously validated in a midlife sample was also evaluated. RESULTS Psychomotor changes, fatigue, and suicidal ideation were associated with decreased functioning and served as the core set of symptoms. Adjusting for selection bias, the sensitivity of these 3 symptoms was 0.012 and specificity 0.994. The sensitivity of the 4 symptoms previously validated in a midlife sample was 0.019 and specificity was 0.997. CONCLUSION We identified 3 depression symptoms that were highly specific for major depressive disorder in older adults. However, these symptoms and a previously identified subset were too insensitive for accurate diagnosis. Therefore, we recommend a full assessment of DSM-IV depression criteria for accurate diagnosis.
Collapse
Affiliation(s)
- Corrine I Voils
- Center for Health Services Research in Primary Care, Durham Veterans Affairs Medical Center, and the Department of Medicine, Division of General Internal Medicine, Duke University Medical Center, Durham, North Carolina, USA.
| | | | | | | |
Collapse
|
27
|
Petscavage JM, Richardson ML, Carr RB. Verification bias an underrecognized source of error in assessing the efficacy of medical imaging. Acad Radiol 2011; 18:343-6. [PMID: 21145764 DOI: 10.1016/j.acra.2010.10.005] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2010] [Revised: 10/14/2010] [Accepted: 10/14/2010] [Indexed: 11/24/2022]
Abstract
RATIONALE AND OBJECTIVES Diagnostic tests are validated by comparison against a "gold standard" reference test. When the reference test is invasive or expensive, it may not be applied to all patients. This can result in biased estimates of the sensitivity and specificity of the diagnostic test. This type of bias is called "verification bias," and is a common problem in imaging research. The purpose of our study is to estimate the prevalence of verification bias in the recent radiology literature. MATERIALS AND METHODS All issues of the American Journal of Roentgenology (AJR), Academic Radiology, Radiology, and European Journal of Radiology (EJR) between November 2006 and October 2009 were reviewed for original research articles mentioning sensitivity or specificity as endpoints. Articles were read to determine whether verification bias was present and searched for author recognition of verification bias in the design. RESULTS During 3 years, these journals published 2969 original research articles. A total of 776 articles used sensitivity or specificity as an outcome. Of these, 211 articles demonstrated potential verification bias. The fraction of articles with potential bias was respectively 36.4%, 23.4%, 29.5%, and 13.4% for AJR, Academic Radiology, Radiology, and EJR. The total fraction of papers with potential bias in which the authors acknowledged this bias was 17.1%. CONCLUSION Verification bias is a common and frequently unacknowledged source of error in efficacy studies of diagnostic imaging. Bias can often be eliminated by proper study design. When it cannot be eliminated, it should be estimated and acknowledged.
Collapse
|
28
|
de Groot JAH, Dendukuri N, Janssen KJM, Reitsma JB, Bossuyt PMM, Moons KGM. Adjusting for Differential-verification Bias in Diagnostic-accuracy Studies. Epidemiology 2011; 22:234-41. [DOI: 10.1097/ede.0b013e318207fc5c] [Citation(s) in RCA: 23] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
29
|
Correcting for Partial Verification Bias: A Comparison of Methods. Ann Epidemiol 2011; 21:139-48. [DOI: 10.1016/j.annepidem.2010.10.004] [Citation(s) in RCA: 29] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2010] [Revised: 09/10/2010] [Accepted: 10/06/2010] [Indexed: 11/18/2022]
|
30
|
Richardson ML, Petscavage JM. An interactive web-based tool for detecting verification (work-up) bias in studies of the efficacy of diagnostic imaging. Acad Radiol 2010; 17:1580-3. [PMID: 20926316 DOI: 10.1016/j.acra.2010.07.011] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2010] [Revised: 07/28/2010] [Accepted: 07/28/2010] [Indexed: 11/28/2022]
Abstract
RATIONALE AND OBJECTIVES Diagnostic tests are validated by comparison against a "gold standard" reference test. When the reference test is invasive or expensive, it may not be applied to all patients. This results in biased estimates of the sensitivity and specificity of the diagnostic test. This type of bias is called "verification bias" and is a common problem in imaging research. The purpose of this study was to create an interactive web-based tool to help detect the presence of prevalence of verification bias in data. MATERIALS AND METHODS The web-based tool was written using well-documented and freely available open-source software (HTML, Apache, Ruby, and R). Our calculator is currently hosted on a Mac OS X server, but should run with minor changes on most other platforms. RESULTS Our program applies user-supplied study data to generate a global sensitivity analysis plot containing noncorrected and corrected values of sensitivity and specificity and 95% confidence intervals for these estimates. Interpretation of the global sensitivity analysis (GSA) plot is relatively simple. Because the characteristic GSA "butterfly" plot contains all possible values of sensitivity and specificity, any indices lying outside of this butterfly are therefore incompatible with the data, confirming that they are biased. CONCLUSIONS Our online calculator makes it easy for investigators to use global sensitivity analysis to demonstrate the presence of significant verification bias and to estimate bias-corrected values of sensitivity and specificity.
Collapse
|
31
|
Roldán Nofuentes JA, Luna del Castillo JD, Marín Jimenez AE. Comparison of the accuracy of multiple binary tests in the presence of partial disease verification. J Stat Plan Inference 2010. [DOI: 10.1016/j.jspi.2010.02.026] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
|
32
|
Reitsma JB, Rutjes AWS, Khan KS, Coomarasamy A, Bossuyt PM. A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. J Clin Epidemiol 2009; 62:797-806. [PMID: 19447581 DOI: 10.1016/j.jclinepi.2009.02.005] [Citation(s) in RCA: 265] [Impact Index Per Article: 17.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2008] [Revised: 01/04/2009] [Accepted: 02/10/2009] [Indexed: 11/16/2022]
Abstract
OBJECTIVE In diagnostic accuracy studies, the reference standard may be imperfect or not available in all patients. We systematically reviewed the proposed solutions for these situations and generated methodological guidance. STUDY DESIGN AND SETTING Review of methodological articles. RESULTS We categorized the solutions into four main groups. The first group includes methods that impute or adjust for missing data on the reference standard. The second group consists of methods that correct estimates of accuracy obtained with an imperfect reference standard. In the third group a reference standard is constructed by combining multiple test results through a predefined rule, based on a consensus procedure, or through statistical modeling. In the fourth group, the diagnostic accuracy paradigm is abandoned in favor of validation studies that relate index test results to relevant clinical data, such as history, future clinical events, and response to therapy. CONCLUSION Most of the methods try to impute, adjust, or construct a reference standard. In situations that deviate only marginally from the classical diagnostic accuracy paradigm, these are valuable methods. In cases where an acceptable reference standard does not exist, the concept of clinical test validation may provide an alternative paradigm to evaluate a diagnostic test.
Collapse
Affiliation(s)
- Johannes B Reitsma
- Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands.
| | | | | | | | | |
Collapse
|
33
|
Computational methods for comparing two binary diagnostic tests in the presence of partial verification of the disease. Comput Stat 2009. [DOI: 10.1007/s00180-009-0155-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/30/2022]
|
34
|
Roldán Nofuentes J, Luna del Castillo J, Montero Alonso M. Determining sample size to evaluate and compare the accuracy of binary diagnostic tests in the presence of partial disease verification. Comput Stat Data Anal 2009. [DOI: 10.1016/j.csda.2008.07.030] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
35
|
de Groot JAH, Janssen KJM, Zwinderman AH, Moons KGM, Reitsma JB. Multiple imputation to correct for partial verification bias revisited. Stat Med 2008; 27:5880-9. [DOI: 10.1002/sim.3410] [Citation(s) in RCA: 36] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
36
|
Reiter JP. Multiple imputation when records used for imputation are not used or disseminated for analysis. Biometrika 2008. [DOI: 10.1093/biomet/asn042] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
|
37
|
Harel O, (Andrew) Zhou XH. Correction. Stat Med 2008. [DOI: 10.1002/sim.3322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
38
|
Hachamovitch R, Di Carli MF. Methods and limitations of assessing new noninvasive tests: part I: Anatomy-based validation of noninvasive testing. Circulation 2008; 117:2684-90. [PMID: 18490536 DOI: 10.1161/circulationaha.107.708586] [Citation(s) in RCA: 27] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Affiliation(s)
- Rory Hachamovitch
- Division of Nuclear Medicine/PET, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, Mass, USA.
| | | |
Collapse
|
39
|
Hanusa BH, Scholle SH, Haskett RF, Spadaro K, Wisner KL. Screening for depression in the postpartum period: a comparison of three instruments. J Womens Health (Larchmt) 2008; 17:585-96. [PMID: 18345995 PMCID: PMC7083208 DOI: 10.1089/jwh.2006.0248] [Citation(s) in RCA: 105] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022] Open
Abstract
OBJECTIVES Postpartum depression, the most prevalent complication of childbirth, is often unrecognized. Our objective was to compare the effectiveness of three screening instruments--Edinburgh Postnatal Depression Scale (EPDS), Patient Health Questionnaire (PHQ-9), and the 7-item screen of the Postpartum Depression Screening Scale (PDSS)--for identifying women with postpartum depression in the first 6 months after delivery. METHODS We administered the three instruments via telephone to women who were > or =18 years and had delivered infants 6-8 weeks earlier. We arranged home interviews to confirm DSM-IV criteria current major depressive disorder (MDD) in women who had an above-threshold score on any of the instruments. For women who screened negative on the 6-8 week call, we repeated the screening at 3 months and 6 months to identify emergent symptoms. The primary outcome measures were the screening scores and DSM-IV diagnoses. RESULTS Of 135 women reached, 123 (91%) were screened, 29 (24%) had home visits, and 13 (11%) had an MDD within 6 months of delivery. Analyses of the scores at 6-8 weeks postpartum and the DSM-IV diagnoses indicated the EPDS at a cutoff point of > or =10 identified 8 (62%) of cases, the PHQ-9 at a cutoff point of > or =10 identified 4 (31%), and the PDSS 7-item Short Form (PDSS_SF) at a cutoff point of > or =14 identified 12 (92%). However, 15 of 16 (94%) women without current MDD screened positive on the PDSS_SF. The EPDS was significantly more accurate (p = 0.01) than the PDSS_SF and PHQ-9 with the cutoff points used. After correcting for verification bias, we found the EPDS and the PDSS_SF were significantly more accurate than the PHQ-9 (p < 0.03). CONCLUSIONS Administering the EPDS by phone at 6-8 weeks postpartum is an efficient and accurate way to identify women at high risk for postpartum depression within the first 6 months after delivery.
Collapse
Affiliation(s)
- Barbara H Hanusa
- Department of Psychiatry, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania 15213, USA.
| | | | | | | | | |
Collapse
|
40
|
Morvan J, Coste J, Roux CH, Euller-Ziegler L, Saraux A, Guillemin F. Prevalence in Two-Phase Surveys: Accuracy of Screening Procedure and Corrected Estimates. Ann Epidemiol 2008; 18:261-9. [DOI: 10.1016/j.annepidem.2007.11.006] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2007] [Revised: 10/11/2007] [Accepted: 11/12/2007] [Indexed: 10/22/2022]
|
41
|
Chi YY, Zhou XH. Receiver operating characteristic surfaces in the presence of verification bias. J R Stat Soc Ser C Appl Stat 2008. [DOI: 10.1111/j.1467-9876.2007.00597.x] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
42
|
Abstract
Missing data is a common complication in data analysis. In many medical settings missing data can cause difficulties in estimation, precision and inference. Multiple imputation (MI) (Multiple Imputation for Nonresponse in Surveys. Wiley: New York, 1987) is a simulation-based approach to deal with incomplete data. Although there are many different methods to deal with incomplete data, MI has become one of the leading methods. Since the late 1980s we observed a constant increase in the use and publication of MI-related research. This tutorial does not attempt to cover all the material concerning MI, but rather provides an overview and combines together the theory behind MI, the implementation of MI, and discusses increasing possibilities of the use of MI using commercial and free software. We illustrate some of the major points using an example from an Alzheimer disease (AD) study. In this AD study, while clinical data are available for all subjects, postmortem data are only available for the subset of those who died and underwent an autopsy. Analysis of incomplete data requires making unverifiable assumptions. These assumptions are discussed in detail in the text. Relevant S-Plus code is provided.
Collapse
Affiliation(s)
- Ofer Harel
- Department of Statistics, University of Connecticut, 215 Glenbrook Road Unit 4120 Storrs, CT 06269-4120, USA.
| | | |
Collapse
|
43
|
|