Quality Assessments of HMO Diagnosis Databases Used to Monitor Childhood Vaccine Safety

Objective: To assess the quality of automated diagnoses extracted from medical care databases by the Vaccine Safety Datalink (VSD) study.

Methods: Two methods are used to assess quality of VSD diagnosis data. The first method compares common automated and abstracted diagnostic categories (“outcomes”) in 1-2% simple random samples of study populations. The second method estimates positive predictive values of automated diagnosis codes used to identify potential cases of rare conditions (e.g., acute ataxia) for inclusion in nested case-control medical record abstraction studies.

Results: There was good agreement (64-68%) between automated and abstracted outcomes in the 1-2% simple random samples at 3 of the 4 VSD sites and poor agreement (44%) at 1 site. Overall at 3 sites, 56% of children with automated cerebella ataxia codes (ICD-9 = 334) and 22% with “lack of coordination” codes (ICD-9 = 781.3) met objective clinical criteria for acute ataxia.

Conclusions: The misclassification error rates for automated screening outcomes substantially reduce the power of screening analyses and limit usefulness of screening analyses to moderate to strong vaccine-outcome associations. Medical record verification of outcomes is needed for definitive assessments.

Quality assessments of HMO diagnosis databases used to monitor childhood vaccine safety

OBJECTIVE

To assess the quality of automated diagnoses extracted from medical care databases by the Vaccine Safety Datalink (VSD) study.

METHODS

Two methods are used to assess quality of VSD diagnosis data. The first method compares common automated and abstracted diagnostic categories ("outcomes") in 1-2% simple random samples of study populations. The second method estimates positive predictive values of automated diagnosis codes used to identify potential cases of rare conditions (e.g., acute ataxia) for inclusion in nested case-control medical record abstraction studies.

RESULTS

There was good agreement (64-68%) between automated and abstracted outcomes in the 1-2% simple random samples at 3 of the 4 VSD sites and poor agreement (44%) at 1 site. Overall at 3 sites, 56% of children with automated cerebella ataxia codes (ICD-9 = 334) and 22% with "lack of coordination" codes (ICD-9 = 781.3) met objective clinical criteria for acute ataxia.

CONCLUSIONS

The misclassification error rates for automated screening outcomes substantially reduce the power of screening analyses and limit usefulness of screening analyses to moderate to strong vaccine-outcome associations. Medical record verification of outcomes is needed for definitive assessments.

Quality of HMO vaccination databases used to monitor childhood vaccine safety. Vaccine Safety DataLink Team

The availability of large, population-based, automated, medical care databases provides unique opportunities for monitoring the safety of childhood vaccines. The authors assessed the quality of automated vaccination databases by comparing them with vaccinations documented in paper-based medical records at three large US West Coast health maintenance organizations (HMOs) participating in the Vaccine Safety DataLink (VSD) study, a Centers for Disease Control and Prevention collaborative study of childhood vaccine safety. The authors randomly selected 1% or 2% samples of VSD study populations (n = 1,224-2,577) for data quality analyses. Agreement between automated and abstracted vaccinations required identical triads of child identification number, vaccination date, and vaccine type. Separate analyses were conducted for each HMO and for each vaccine type administered between 1991 and 1995. Agreement was measured by three matching proportions: 1) the proportion of automated vaccinations present in the abstracted source, 2) the proportion of abstracted vaccinations present in the automated source, and 3) the proportion of vaccinations from either source present in both sources. Overall, for common childhood vaccines, proportion 1 ranged from 83% to 99%, proportion 2 ranged from 82% to 98%, and proportion 3 ranged from 70% to 97%. Lack of automated data was the most frequent type of discrepancy, followed by date mismatches and vaccine type mismatches. Vaccination exposure classification errors in the range reported here were found by mathematical modeling to only modestly bias measured medical outcome rate ratios toward the null hypothesis. The results of the data quality analyses support the usefulness of vaccination exposure data derived from these automated HMO vaccination databases.