Validation of Diagnostic Groups Based on Health Care Utilization Data Should Adjust for Sampling Strategy

Validating pertussis data measures using electronic medical record data in Ontario, Canada 1986-2016

Background

Pertussis is a reportable disease in many countries, but ascertainment bias has limited data accuracy. This study aims to validate pertussis data measures using a reference standard that incorporates different suspected case severities, allowing for the impact of case severity on accuracy and detection to be explored.

Methods

We evaluated 25 pertussis detection algorithms in a primary care electronic medical record database between January 1, 1986 and December 30, 2016. We estimated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). We used sensitivity analyses to explore areas of uncertainty and evaluated reasons for lack of detection.

Results

The algorithm including all data measures achieved the highest sensitivity at 20.6%. Sensitivity increased to 100% after reclassifying symptom-only cases as non-cases, but the PPV remained low. Age at first episode was significantly associated with detection in half of the tested scenarios, and false negatives often had some history of immunization.

Conclusions

Sensitivity improved by reclassifying symptom-only cases but remained low unless multiple data sources were used. Results demonstrate a trade-off between PPV and sensitivity. EMRs can enhance detection through patient history and clinical note data. It is essential to improve case identification of older individuals with vaccination history to reduce ascertainment bias.

Developing a reference standard for pertussis by applying a stratified sampling strategy to electronic medical record data

PURPOSE

Pertussis surveillance remains essential in Canada, but ascertainment bias limits the accuracy of surveillance data. Introducing other sources to improve detection has highlighted the importance of validation. However, challenges arise due to low prevalence, and oversampling suspected cases can introduce partial verification bias. The aim of this study was to build a reference standard for pertussis validation studies that provides adequate analytic precision and minimizes bias.

METHODS

We used a stratified strategy to sample the reference standard from a primary care electronic medical record cohort. We incorporated abstractor notes into definite, possible, ruled-out, and no mention of pertussis classifications which were based on surveillance case definitions.

RESULTS

We abstracted eight hundred records from the cohort of 404,922. There were 208 (26%) definite and 261 (32.6%) possible prevalent pertussis cases. Classifications demonstrated a wide variety of case severities. Abstraction reliability was moderate to substantial based on Cohen's kappa and raw percent agreement.

CONCLUSIONS

When conducting validation studies for pertussis and other low prevalence diseases, this stratified sampling strategy can be used to develop a reference standard using limited resources. This approach mitigates verification and spectrum bias while providing sufficient precision and incorporating a range of case severities.

Systematic review of the accuracy of antibody tests used to screen asymptomatic adults for hepatitis C infection

BACKGROUND

Several expert groups, including the United States Preventive Services Task Force and the Canadian Task Force on Preventive Health Care, have recently examined or are currently examining whether primary care physicians should screen asymptomatic adults for hepatitis C virus (HCV) infection. To inform decision-making on HCV screening, we performed a systematic review of the accuracy of antibody tests compared with other immunoassays and RNA detection for screening asymptomatic adults for HCV infection in Canada.

METHODS

MEDLINE and Embase databases were searched from 1990 to 2016; resulting citations were uploaded into DistillerSR and independently screened by 2 reviewers. Original research studies, systematic reviews and meta-analyses were eligible for inclusion. At least 80% of the study population had to be asymptomatic, nonpregnant, treatment-naïve adults with unknown liver enzyme values and unknown HCV status. Risk of bias was assessed with the use of the Quality Assessment of Diagnostic Accuracy Studies version 2 (QUADAS-2) tool; the quality of the body of evidence was assessed by means of GRADE (Grading of Recommendations Assessment, Development and Evaluation) methodology.

RESULTS

Of 1537 articles identified, 81 underwent full-text review, and 9 studies met the inclusion criteria. Compared with RNA detection, the sensitivity of the third-generation enzyme-linked immunosorbent assay was variable (61.0%-81.8%), and its specificity was high (97.5%-99.7%). As expected, there were more false-positive results when comparing antibody tests to RNA detection than to other immunoassays. Our GRADE assessment suggested that there was a high concern for risk of bias, particularly verification bias, and substantial inconsistency between studies in terms of their design.

INTERPRETATION

More research is needed to better characterize the accuracy of antibody tests used to screen for HCV infection in the general population. Jurisdictions that recently adopted birth cohort screening for HCV infection are encouraged to evaluate and report on the accuracy of HCV screening tests and screening benefits and harms. PROSPERO registration: no. CRD42016039710.