Bias in estimating accuracy of a binary screening test with differential disease verification

Screening for Primary Aldosteronism by Mass Spectrometry Versus Immunoassay Measurements of Aldosterone: A Prospective Within-Patient Study

BACKGROUND

Measurements of aldosterone by mass spectrometry are more accurate and less prone to interferences than immunoassay measurements, and may produce a more accurate aldosterone:renin ratio (ARR) when screening for primary aldosteronism (PA).

METHODS

Differences in diagnostic performance of the ARR using mass spectrometry vs immunoassay measurements of aldosterone were examined in 710 patients screened for PA. PA was confirmed in 153 patients and excluded in 451 others. Disease classifications were not achieved in 106 patients. Areas under receiver-operating characteristic curves (AUROC) and other measures were used to compare diagnostic performance.

RESULTS

Mass spectrometry-based measurements yielded lower plasma aldosterone concentrations than immunoassay measurements. For the ARR based on immunoassay measurements of aldosterone, AUROCs were slightly lower (P = 0.018) than those using mass spectrometry measurements (0.895 vs 0.906). The cutoff for the ARR to reach a sensitivity of 95% was 30 and 21.5 pmol/mU by respective immunoassay and mass spectrometry-based measurements, which corresponded to specificities of 57% for both. With data restricted to patients with unilateral PA, diagnostic sensitivities of 94% with specificities >81% could be achieved at cutoffs of 68 and 52 pmol/mU for respective immunoassay and mass spectrometry measurements.

CONCLUSIONS

Mass spectrometry-based measurements of aldosterone for the ARR provide no clear diagnostic advantage over immunoassay-based measurements. Both approaches offer limited diagnostic accuracy for the ARR as a screening test. One solution is to employ the higher cutoffs to triage patients likely to have unilateral PA for further tests and possible adrenalectomy, while using the lower cutoffs to identify others for targeted medical therapy.German Clinical Trials Register ID: DRKS00017084.

Estimating Cancer Screening Sensitivity and Specificity Using Healthcare Utilization Data: Defining the Accuracy Assessment Interval

The effectiveness and efficiency of cancer screening in real-world settings depend on many factors, including test sensitivity and specificity. Outside of select experimental studies, not everyone receives a gold standard test that can serve as a comparator in estimating screening test accuracy. Thus, many studies of screening test accuracy use the passage of time to infer whether or not cancer was present at the time of the screening test, particularly for patients with a negative screening test. We define the accuracy assessment interval as the period of time after a screening test that is used to estimate the test's accuracy. We describe how the length of this interval may bias sensitivity and specificity estimates. We call for future research to quantify bias and uncertainty in accuracy estimates and to provide guidance on setting accuracy assessment interval lengths for different cancers and screening modalities.

Adjusting for verification bias in diagnostic accuracy measures when comparing multiple screening tests - an application to the IP1-PROSTAGRAM study

Introduction

Novel screening tests used to detect a target condition are compared against either a reference standard or other existing screening methods. However, as it is not always possible to apply the reference standard on the whole population under study, verification bias is introduced. Statistical methods exist to adjust estimates to account for this bias. We extend common methods to adjust for verification bias when multiple tests are compared to a reference standard using data from a prospective double blind screening study for prostate cancer.

Methods

Begg and Greenes method and multiple imputation are extended to include the results of multiple screening tests which determine condition verification status. These two methods are compared to the complete case analysis using the IP1-PROSTAGRAM study data. IP1-PROSTAGRAM used a paired-cohort double-blind design to evaluate the use of imaging as alternative tests to screen for prostate cancer, compared to a blood test called prostate specific antigen (PSA). Participants with positive imaging (index) and/or PSA (control) underwent a prostate biopsy (reference standard).

Results

When comparing complete case results to Begg and Greenes and methods of multiple imputation there is a statistically significant increase in the specificity estimates for all screening tests. Sensitivity estimates remained similar across the methods, with completely overlapping 95% confidence intervals. Negative predictive value (NPV) estimates were higher when adjusting for verification bias, compared to complete case analysis, even though the 95% confidence intervals overlap. Positive predictive value (PPV) estimates were similar across all methods.

Conclusion

Statistical methods are required to adjust for verification bias in accuracy estimates of screening tests. Expanding Begg and Greenes method to include multiple screening tests can be computationally intensive, hence multiple imputation is recommended, especially as it can be modified for low prevalence of the target condition.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01481-w.

Diagnostic test evaluation methodology: A systematic review of methods employed to evaluate diagnostic tests in the absence of gold standard - An update

OBJECTIVE

To systematically review methods developed and employed to evaluate the diagnostic accuracy of medical test when there is a missing or no gold standard.

STUDY DESIGN AND SETTINGS

Articles that proposed or applied any methods to evaluate the diagnostic accuracy of medical test(s) in the absence of gold standard were reviewed. The protocol for this review was registered in PROSPERO (CRD42018089349).

RESULTS

Identified methods were classified into four main groups: methods employed when there is a missing gold standard; correction methods (which make adjustment for an imperfect reference standard with known diagnostic accuracy measures); methods employed to evaluate a medical test using multiple imperfect reference standards; and other methods, like agreement studies, and a mixed group of alternative study designs. Fifty-one statistical methods were identified from the review that were developed to evaluate medical test(s) when the true disease status of some participants is unverified with the gold standard. Seven correction methods were identified and four methods were identified to evaluate medical test(s) using multiple imperfect reference standards. Flow-diagrams were developed to guide the selection of appropriate methods.

CONCLUSION

Various methods have been proposed to evaluate medical test(s) in the absence of a gold standard for some or all participants in a diagnostic accuracy study. These methods depend on the availability of the gold standard, its' application to the participants in the study and the availability of alternative reference standard(s). The clinical application of some of these methods, especially methods developed when there is missing gold standard is however limited. This may be due to the complexity of these methods and/or a disconnection between the fields of expertise of those who develop (e.g. mathematicians) and those who employ the methods (e.g. clinical researchers). This review aims to help close this gap with our classification and guidance tools.

Anticipating missing reference standard data when planning diagnostic accuracy studies

Results obtained using a reference standard may be missing for some participants in diagnostic accuracy studies. This paper looks at methods for dealing with such missing data when designing or conducting a prospective diagnostic accuracy study

Exploring the Underdiagnosis and Prevalence of Autism Spectrum Conditions in Beijing

Previous studies reported that the prevalence of Autism Spectrum Conditions (ASC) in mainland China is much lower than estimates from developed countries (around 1%). The aim of the study is to apply current screening and standardized diagnostic instruments to a Chinese population to establish a prevalence estimate of ASC in an undiagnosed population in mainland China. We followed the design development used previously in the UK published in 2009 by Baron‐Cohen and colleagues. The Mandarin Childhood Autism Spectrum Test (CAST) was validated by screening primary school pupils (n = 737 children age 6–10 years old) in Beijing and by conducting diagnostic assessments using the Autism Diagnostic Observation Schedule and the Autism Diagnostic Interview‐Revised. The prevalence estimate was generated after adjusting and imputing for missing values using the inverse probability weighting. Response was high (97%). Using the UK cutoff (≥15), CAST performance has 84% sensitivity and 96% specificity (95% confidence interval [CI]: 46, 98, and 96, 97, respectively). Six out of 103 children, not previously diagnosed, were found to the meet diagnostic criteria (8.5 after adjustment, 95% CI: 1.6, 15.4). The preliminary prevalence in an undiagnosed primary school population in mainland China was 119 per 10,000 (95% CI: 53, 265). The utility of CAST is acceptable as a screening instrument for ASC in large epidemiological studies in China. Using a comparable method, the preliminary prevalence estimate of ASC in mainland China is similar to that of those from developed countries. Autism Res2015, 8: 250–260. © 2015 The Authors. Autism Research published by Wiley Periodicals, Inc. on behalf of International Society for Autism Research

Personalized prostate cancer screening among men with high risk genetic predisposition- study protocol for a prospective cohort study

Background

Prostate cancer screening among the general population is highly debatable. Nevertheless, screening among high-risk groups is appealing. Prior data suggests that men carrying mutations in the BRCA1& 2 genes may be at increased risk of developing prostate cancer. Additionally, they appear to develop prostate cancer at a younger age and with a more aggressive course. However, prior studies did not systematically perform prostate biopsies and thus cannot determine the true prevalence of prostate cancer in this population.

Methods

This will be a prospective diagnostic trial of screening for prostate cancer among men with genetic predisposition. The target population is males (40–70 year old) carrying a BRCA1 and/or BRCA2 germ line mutation. They will be identified via our Genetic counseling unit. All men after signing an informed consent will undergo the following tests: PSA, free to total PSA, MRI of prostate and prostate biopsy. The primary endpoint will be to estimate the prevalence, stage and grade of prostate cancer in this population. Additionally, the study aims to estimate the impact of these germ line mutations on benign prostatic hyperplasia. Furthermore, this study aims to create a bio-bank of tissue, urine and serum of this unique cohort for future investigations. Finally, this study will identify an inception cohort for future interventional studies of primary and secondary prevention.

Discussion

The proposed research is highly translational and focuses not only on the clinical results, but on the future specimens that will be used to advance our understanding of prostate cancer patho-physiology. Most importantly, these high-risk germ-line mutation carriers are ideal candidates for primary and secondary prevention initiatives.

Trial registration

ClinicalTrials.gov: NCT02053805.

Estimation of diagnostic test accuracy without full verification: a review of latent class methods

The performance of a diagnostic test is best evaluated against a reference test that is without error. For many diseases, this is not possible, and an imperfect reference test must be used. However, diagnostic accuracy estimates may be biased if inaccurately verified status is used as the truth. Statistical models have been developed to handle this situation by treating disease as a latent variable. In this paper, we conduct a systematized review of statistical methods using latent class models for estimating test accuracy and disease prevalence in the absence of complete verification.

A systematic review classifies sources of bias and variation in diagnostic test accuracy studies

OBJECTIVE

To classify the sources of bias and variation and to provide an updated summary of the evidence of the effects of each source of bias and variation.

STUDY DESIGN AND SETTING

We conducted a systematic review of studies of any design with the main objective of addressing bias or variation in the results of diagnostic accuracy studies. We searched MEDLINE, EMBASE, BIOSIS, the Cochrane Methodology Register, and Database of Abstracts of Reviews of Effects (DARE) from 2001 to October 2011. Citation searches based on three key papers were conducted, and studies from our previous review (search to 2001) were eligible. One reviewer extracted data on the study design, objective, sources of bias and/or variation, and results. A second reviewer checked the extraction.

RESULTS

We summarized the number of studies providing evidence of an effect arising from each source of bias and variation on the estimates of sensitivity, specificity, and overall accuracy.

CONCLUSIONS

We found consistent evidence for the effects of case-control design, observer variability, availability of clinical information, reference standard, partial and differential verification bias, demographic features, and disease prevalence and severity. Effects were generally stronger for sensitivity than for specificity. Evidence for other sources of bias and variation was limited.

Estimating the relative utility of screening mammography

BACKGROUND

The concept of diagnostic utility is a fundamental component of signal detection theory, going back to some of its earliest works. Attaching utility values to the various possible outcomes of a diagnostic test should, in principle, lead to meaningful approaches to evaluating and comparing such systems. However, in many areas of medical imaging, utility is not used because it is presumed to be unknown.

METHODS

In this work, we estimate relative utility (the utility benefit of a detection relative to that of a correct rejection) for screening mammography using its known relation to the slope of a receiver operating characteristic (ROC) curve at the optimal operating point. The approach assumes that the clinical operating point is optimal for the goal of maximizing expected utility and therefore the slope at this point implies a value of relative utility for the diagnostic task, for known disease prevalence. We examine utility estimation in the context of screening mammography using the Digital Mammographic Imaging Screening Trials (DMIST) data.

RESULTS

We show how various conditions can influence the estimated relative utility, including characteristics of the rating scale, verification time, probability model, and scope of the ROC curve fit. Relative utility estimates range from 66 to 227.

CONCLUSIONS

We argue for one particular set of conditions that results in a relative utility estimate of 162 (±14%). This is broadly consistent with values in screening mammography determined previously by other means. At the disease prevalence found in the DMIST study (0.59% at 365-day verification), optimal ROC slopes are near unity, suggesting that utility-based assessments of screening mammography will be similar to those found using Youden's index.