Accuracy and usefulness of the HEDIS childhood immunization measures

Evaluating the vaccination coverage: validity of household-hold vaccination booklet and caregiver's recall

BACKGROUND

We compared results from household data sources to medical record sources by using data from a vaccination coverage survey.

METHODS

Vaccination coverage (VC) was calculated based on parental recall, household vaccination booklet, and Zhejiang provincial immunization information system (ZJIIS). We evaluated the accuracy of VC based on household sources (vaccination booklet and recall) assuming the medical record was accurate. Concordance, sensitivity, specificity, positive predictive value, and negative predictive value were estimated as well as the Kappa statistic was also used to evaluate the agreement between data sources.

RESULTS

Among the 1,800 children identified in the household survey, all were registered in ZJIIS. VC estimated using the vaccination booklet alone was substantially lower than that based on medical records (net bias 3.4-16.7% in different age groups). VC based on parental recall ranged from 2.5% below (among children aged 1 year) to 16.7% points above (among children aged 6 years) than those based on medical records. Concordance was lowest for card estimates (32.5-45.5%). Sensitivity was <60% for all household sources, except for recall source. Specificity was lowest for recall estimates (14.5-42.6%). Positive predictive value was >75%, while negative predictive value was <50%, for all household sources. Kappa statistics generally indicated poor agreement between household and medical record sources.

CONCLUSIONS

Household-retained vaccination booklets and parental recall were insufficient sources for evaluating the VC. Our findings emphasized the importance of taking interventions to make the vaccination booklet more consistent with the records from medical resource.

A pathway to developing and testing quality measures aimed at improving adult vaccination rates in the United States

Despite recommendations for vaccinating adults and widespread availability of immunization services (e.g., pharmacy venues, workplace wellness clinics), vaccination rates in the United States remain low. The U.S. National Adult Immunization Plan identified the development of quality measures as a priority and key strategy to address low adult vaccination coverage rates. The use of quality measures can provide incentives for increased utilization of preventive services. To address the lack of adult immunization measures, the National Adult and Influenza Immunization Summit, a coalition of adult immunization partners led by the Immunization Action Coalition, Centers for Disease Control and Prevention, and National Vaccine Program Office, spearheaded efforts to (1) identify gaps and priorities in adult immunization quality performance measurement; (2) explore feasibility of data collection on adult immunizations through pilot testing and engaging stakeholders; and (3) develop and test quality measure specifications. This paper outlines the process by which a public-private partnership drove the development of two adult immunization performance measures-an adult immunization status measure for influenza, tetanus and diphtheria (Td) and/or tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis (Tdap), herpes zoster and pneumococcal vaccines, and a prenatal immunization status measure for influenza and Tdap vaccinations in pregnant women. These measures have recently been added to the 2019 Healthcare Effectiveness Data and Information Set (HEDIS®), a widely used set of performance measures reportable by private health plans.

Electronic Immunization Alerts and Spillover Effects on Other Preventive Care

The impact of electronic health record (EHR) immunization clinical alert systems on the delivery of other preventive services remains unknown. We assessed for spillover effects of an EHR immunization alert on delivery of 6 other preventive services, in children 18 to 30 months of age needing immunizations. We conducted a secondary data analysis, with additional primary data collection, of a randomized, historically controlled trial to improve immunization rates with EHR alerts, in an urban, primary care clinic. No significant differences were found in screening for anemia, lead, development, nutrition, and injury prevention counseling in children prompting EHR immunization alerts (n = 129), compared with controls (n = 135). Significant increases in oral health screening in patients prompting EHR alerts (odds ratio = 4.8, 95% CI = 1.8-13.0) were likely due to practice changes over time. An EHR clinical alert system targeting immunizations did not have a spillover effect on the delivery of other preventive services.

Context Matters: Practitioner Perspectives on Immunization Delivery Quality Improvement Efforts

Uneven adherence to immunization guidelines might leave some communities vulnerable to outbreaks of vaccine-preventable diseases. To examine factors related to implementation of immunization delivery best practices, we analyzed responses to monthly surveys and debriefing interviews from 16 diverse pediatric practices engaged in a year-long virtual learning collaborative. The collaborative provided a toolkit, online learning sessions, performance feedback, and conference calls with a quality improvement coach. Participants used iterative plan-do-study-act cycles to implement self-selected changes in immunization practices. Descriptive statistics were applied to quantitative data; qualitative data were analyzed using a framework approach. Impediments to implementing guidelines included difficulties with electronic record systems, rigid management structures, competing priorities, and parental resistance. Facilitators included linkage with regional immunization registries, positive social interactions, and performance feedback. Findings suggest that improving immunization delivery requires not only awareness of recommendations and training in improvement methods but also efforts to ameliorate contextual factors that impede immunization delivery.

Improving immunization delivery using an electronic health record: the ImmProve project

OBJECTIVE

Though an essential pediatric preventive service, immunizations are challenging to deliver reliably. Our objective was to measure the impact on pediatric immunization rates of providing clinicians with electronic health record-derived immunization prompting.

METHODS

Operating in a large, urban, hospital-based pediatric primary care clinic, we evaluated 2 interventions to improve immunization delivery to children ages 2, 6, and 13 years: point-of-care, patient-specific electronic clinical decision support (CDS) when children overdue for immunizations presented for care, and provider-specific bulletins listing children overdue for immunizations.

RESULTS

Overall, the proportion of children up to date for a composite of recommended immunizations at ages 2, 6, and 13 years was not different in the intervention (CDS active) and historical control (CDS not active) periods; historical immunization rates were high. The proportion of children receiving 2 doses of hepatitis A immunization before their second birthday was significantly improved during the intervention period. Human papillomavirus (HPV) immunization delivery was low during both control and intervention periods and was unchanged for 13-year-olds. For 14-year-olds, however, 4 of the 5 highest quarterly rates of complete HPV immunization occurred in the final year of the intervention. Provider-specific bulletins listing children overdue for immunizations increased the likelihood of identified children receiving catch-up hepatitis A immunizations (hazard ratio 1.32; 95% confidence interval 1.12-1.56); results for HPV and the composite of recommended immunizations were of a similar magnitude but not statistically significant.

CONCLUSIONS

In our patient population, with high baseline uptake of recommended immunizations, electronic health record-derived immunization prompting had a limited effect on immunization delivery. Benefit was more clearly demonstrated for newer immunizations with lower baseline uptake.