Assessing the relative accuracies of two screening tests in the presence of verification bias

VALHUDES: A protocol for validation of human papillomavirus assays and collection devices for HPV testing on self-samples and urine samples

BACK GROUND

Systematic reviews have concluded that hrHPV DNA testing using target-amplification tests is as accurate on vaginal self-samples as on clinician-taken specimens for the detection of cervical precancer. However, insufficient evidence is available for specific HPV assay/self-sample device combinations.

OBJECTIVES

The VALHUDES protocol is designed as a diagnostic test accuracy study that aims to compare the clinical sensitivity and specificity of particular hrHPV assay(s) on vaginal self-samples and first-void-urine, collected in agreement with standardized protocols, with hrHPV testing on matched clinician-taken samples.

STUDY DESIGN

Five hundred enrolled women referred to a colposcopy clinic are invited to collect a first-void urine sample and one or more vaginal self-samples with particular devices before collection of a cervical sample by a clinician. Sample sets are subsequently analysed in a laboratory accredited for HPV testing. Disease verification for all enrolled patients is provided by colposcopy combined with histological assessment of biopsies.

RESULTS

A first VALHUDES study has started in Belgium in December 2017 with enrolment from four colposcopy centres. The following assays are foreseen to be evaluated: RealTime High Risk HPV assay (Abbott), cobas-4800 and -6800 (Roche), Onclarity (BD), Xpert HPV (Cepheid) and Anyplex II HPV HR (Seegene).

CONCLUSION

Given empirical evidence that the relative accuracy of HPV-testing on self- vs clinician-samples is robust across clinical settings, the VALHUDES protocol offers a framework for validation of HPV assay/self-sample device combinations that can be translated to a primary screening setting.

Xpert human papillomavirus test is a promising cervical cancer screening test for HIV-seropositive women

This study investigated the performance of Cepheid Xpert human papillomavirus (HPV) assay in South African human immunodeficiency virus (HIV)-infected women and compared its performance with that of hybrid capture-2 (hc2). Methods: Stored cervical specimens from HIV-infected women that had previously been tested using hc2 were tested using Xpert. Results: The overall HR-HPV prevalence was found to be 62.0% (720/1161) by Xpert and 61.2% (711/1161) by hc2. 13.6% (158/1161) were HPV16 positive, 18.8% (218/1161) were HPV18/45, 37.3% (434/1161) were HPV31/33/35/52/58, 12.7% (147/1161) were HPV51/59 and 23.3% (270/1161) were HPV39/68/56/66. Overall agreement with hc2 was 90%; Cohen’s kappa was 0.78 (95% CI 0.74–0.82) indicating substantial agreement. Detection of HPV16, HPV18/45, and HPV31/33/35/52/58 were independently associated with cervical intraepithelial neoplasia (CIN)−2+ (P<0.0001 for each); while HPV51/59 and HPV39/68/56/66 were not. Women infected with HPV16, HPV18/45 or HPV31/33/35/52/58 were found to have significantly higher amounts of HPV DNA detected for those with CIN2+ compared to those without CIN2+, P<0.0001 for each. Xpert and hc2 were similarly sensitive (88.3% and 91.5%, respectively) and specific (48.4% and 51.0%) for CIN2+ and CIN3 (sensitivity: 95.8% and 97.9%; specificity: 41.4% and 42.8%). Conclusions: Xpert is a promising screening test in HIV-infected women that performs similarly to hc2.

Evaluation of a cervicography-based program to ensure quality of visual inspection of the cervix in HIV-infected women in Johannesburg, South Africa

Quality assurance with cervicography and specialist review significantly improved the sensitivity of visual inspection with acetic acid to detect cervical intraepithelial neoplasia grade 2 or worse.

Objective

To determine whether a quality assurance (QA) program using digital cervicography improved the performance of a visual inspection with acetic acid (VIA) to detect cervical intraepithelial neoplasia grade 2 or worse (CIN 2+) in HIV-infected women in Johannesburg, South Africa.

Materials and Methods

Visual inspection with acetic acid was performed among HIV-infected women, aged 18 to 65 years, in Johannesburg, South Africa. Nurses received 2 weeks of training on the VIA procedure. The VIA interpretation was performed in real time. The VIA results were then photographed using a retail available digital camera. A gynecologist and medical officer reviewed the VIA digital images within 2 weeks of the procedure. Colposcopic biopsy was performed on all women with positive VIA and 25% negative VIA results. Sensitivity and specificity of VIA for the detection of CIN 2+ were compared between the nurses and physicians at the beginning and at the end of the study.

Results

Positive VIA results were found in 541 (45%) of the 1,202 participating women. The sensitivity of VIA to predict CIN 2+ was improved from 65% to 75% (p = .001) with the addition of digital cervicography and specialist review. There was no statistical difference in the sensitivity of the VIA readings when comparing the first 600 participants to the final 593 participants between the nurses (p = .613) and physicians (p = .624).

Conclusions

Quality assurance performed by specialists using digital cervicography improved the sensitivity of VIA. There was no difference in sensitivity in interpreting VIA between the beginning and the end of the study. Quality assurance should form a cornerstone of any VIA program to improve sensitivity in detecting CIN 2+ lesions.

Validation of cervical cancer screening methods in HIV positive women from Johannesburg South Africa

BACKGROUND

HIV-infected women are at increased risk for developing cervical cancer. Women living in resource-limited countries are especially at risk due to poor access to cervical cancer screening and treatment. We evaluated three cervical cancer screening methods to detect cervical intraepithelial neoplasia grade 2 and above (CIN 2+) in HIV-infected women in South Africa; Pap smear, visual inspection with 5% acetic acid (VIA) and human papillomavirus detection (HPV).

METHODS

HIV-infected women aged 18-65 were recruited in Johannesburg. A cross-sectional study evaluating three screening methods for the detection of the histologically-defined gold standard CIN-2 + was performed. Women were screened for cervical abnormalities with the Digene HC2 assay (HPV), Pap smear and VIA. VIA was performed by clinic nurses, digital photographs taken and then later reviewed by specialist physicians. The sensitivity, specificity and predictive valves for CIN-2 + were calculated using maximum likelihood estimators.

RESULTS

1,202 HIV-infected women participated, with a median age of 38 years and CD4 counts of 394 cells/mm(3). One third of women had a high grade lesion on cytology. VIA and HPV were positive in 45% and 61% of women respectively. Estimated sensitivity/specificity for HPV, Pap smear and VIA for CIN 2+ was 92%/51.4%, 75.8%/83.4% and 65.4/68.5% (nurse reading), respectively. Sensitivities were similar, and specificities appeared significantly lower for the HPV test, cytology and VIA among women with CD4 counts ≤200 cells/mm(3) as compared to CD4 counts >350 cells/mm(3).

CONCLUSIONS

Although HPV was the most sensitive screening method for detecting CIN 2+, it was less specific than conventional cytology and VIA with digital imaging review. Screening programs may need to be individualized in context of the resources and capacity in each area.

Estimating the relative utility of screening mammography

BACKGROUND

The concept of diagnostic utility is a fundamental component of signal detection theory, going back to some of its earliest works. Attaching utility values to the various possible outcomes of a diagnostic test should, in principle, lead to meaningful approaches to evaluating and comparing such systems. However, in many areas of medical imaging, utility is not used because it is presumed to be unknown.

METHODS

In this work, we estimate relative utility (the utility benefit of a detection relative to that of a correct rejection) for screening mammography using its known relation to the slope of a receiver operating characteristic (ROC) curve at the optimal operating point. The approach assumes that the clinical operating point is optimal for the goal of maximizing expected utility and therefore the slope at this point implies a value of relative utility for the diagnostic task, for known disease prevalence. We examine utility estimation in the context of screening mammography using the Digital Mammographic Imaging Screening Trials (DMIST) data.

RESULTS

We show how various conditions can influence the estimated relative utility, including characteristics of the rating scale, verification time, probability model, and scope of the ROC curve fit. Relative utility estimates range from 66 to 227.

CONCLUSIONS

We argue for one particular set of conditions that results in a relative utility estimate of 162 (±14%). This is broadly consistent with values in screening mammography determined previously by other means. At the disease prevalence found in the DMIST study (0.59% at 365-day verification), optimal ROC slopes are near unity, suggesting that utility-based assessments of screening mammography will be similar to those found using Youden's index.

Demographic corrections appear to compromise classification accuracy for severely skewed cognitive tests

Demographic corrections for cognitive tests should improve classification accuracy by reducing age or education biases, but empirical support has been equivocal. Using a simulation procedure, we show that creating moderate or extreme skewness in cognitive tests compromises the classification accuracy of demographic corrections, findings that appear replicated within clinical data for the few neuropsychological test scores with an extreme degree of skew. For most neuropsychological tests, the dementia classification accuracy of raw and demographically corrected scores was equivalent. These findings suggest that the dementia classification accuracy of demographic corrections is robust to slight degrees of skew (i.e., skewness <1.5).

Estimation of the disease-specific diagnostic marker distribution under verification bias

We consider the estimation of the parameters indexing a parametric model for the conditional distribution of a diagnostic marker given covariates and disease status. Such models are useful for the evaluation of whether and to what extent a marker's ability to accurately detect or discard disease depends on patient characteristics. A frequent problem that complicates the estimation of the model parameters is that estimation must be conducted from observational studies. Often, in such studies not all patients undergo the gold standard assessment of disease. Furthermore, the decision as to whether a patient undergoes verification is not controlled by study design. In such scenarios, maximum likelihood estimators based on subjects with observed disease status are generally biased. In this paper, we propose estimators for the model parameters that adjust for selection to verification that may depend on measured patient characteristics and additonally adjust for an assumed degree of residual association. Such estimators may be used as part of a sensitivity analysis for plausible degrees of residual association. We describe a doubly robust estimator that has the attractive feature of being consistent if either a model for the probability of selection to verification or a model for the probability of disease among the verified subjects (but not necessarily both) is correct.

Neurodegenerative memory disorders: a potential role of environmental toxins

The hypothesis that neurotoxins may play a role in neurodegenerative disorders remains an elusive one, given that epidemiologic studies often provide conflicting results. Although these conflicting results may result from methodological differences within and between studies, the complexity of chemical disruption of the central nervous system cannot be ignored in attempts to evaluate this hypothesis in different neurodegenerative disorders. Spencer provides a detailed review of the complex processes involved in defining the neurotoxic potential of naturally occurring and synthetic agents. Even concepts such as exposure and dose, as often reported in studies attempting to evaluate the risk imparted by a potential compound, can be deceptive. For example, although dose reflects "that amount of chemical transferred to the exposed subject", factors such as time and concentration in the organism, the ability to access the central nervous system, and how a compound reaches the central nervous system (routes of administration) or secondarily affects other organ systems leading to central nervous system disruption are clearly important to the concept of neurotoxic risk in neurodegenerative disorders. These factors would appear to explain the observed disagreements between studies using animal or neuronal models of neurotoxicity and population-based studies in humans. The importance of these factors and how a potential neurotoxin is investigated are clearly seen in the data on AD and aluminum. In contrast, the impact of MTPT on the central nervous system is more direct and compelling. Added complexity in the study of neurotoxins in human neurodegeneration is derived from data showing that agents may have additive, potentiating, synergistic, or antagonistic effects. Therefore, data from studies evaluating EMF risks could be readily confounded by the presence or absence of heavy metals (eg, arc welding). Other factors that may conceal neurotoxic causes for a given disorder focus on additional features such as genetic predispositions, physiologic changes that occur in aging, and even nutritional status that can support or hinder the affect of a given agent on the central nervous system. Finally, many studies that investigate exposure risk do not readily incorporate the five criteria proposed by Schaumburg for establishing causation. For example, if we apply Schaumburg's first criterion, epidemiologic studies often determines the presence of an agent through history, yet they cannot readily confirm exposure based on environmental or clinical chemical analyses to fulfill this criterion for causation. Additional limitations in research design along with the populations and methods that are sued to study neurotoxins in human neurodegenerative disorders often fail to meet other criteria such as linking the severity and onset with duration and exposure level. Therefore, although studies of agents such as MTPT provide compelling models of neurotoxins and neurodegeneration in humans, disorders such as ALS, PD, and particularly AD will require additional effort if research is to determine the contribution (presence or absence) of neurotoxins to these neurologic disorders.

Correcting the 3MS for bias does not improve accuracy when screening for cognitive impairment or dementia

We investigated the effects of correcting for demographic biases on the sensitivity and specificity of the Modified Mini Mental Status Exam (3MS) using a sample of English-speaking older adults (N=8901) from the Canadian Studies of Health and Aging. The sensitivity and specificity of the original 3MS were compared to the 3MS regression-adjusted for the influence of demographic variables and then to 3MS percentiles based on published normative data with age and education corrected cutoff scores. According to receiver operating characteristic curve analyses, the regression-adjusted 3MS was no more accurate than the original 3MS when screening for dementia, and it was less accurate when screening for cognitive impairment. The use of 3MS percentiles based on normative data with age and education corrected cut-off points were less accurate than the original 3MS when screening for both cognitive impairment and when screening for dementia.

The reliability of clinical methods in ophthalmology

The reliability of clinical methods is often taken for granted. There is considerable evidence, however, that variability of clinical measurements, interpretations, and judgements are greater than commonly thought. Inconsistency of clinical methods has profound and widespread implications in clinical practice. The inferential capacity of measurements depends on accuracy and reproducibility. Appropriate medical decisions cannot be reached without accurate and reliable data. This review will examine the conceptual relationship of reliability, accuracy, and validity. The role of observer reliability in ophthalmic test measurements and interpretations will be addressed in the context of three case reports.