Improving immunization data quality in Peru and Mexico: Two case studies highlighting challenges and lessons learned

Coping mechanisms and strategies adopted to improve the quality and timeliness of immunization data among lower-level private-for-profit service providers in Kampala Capital City, Uganda

BACKGROUND

Lower-level urban private-for-profit health service providers are actively engaged in the delivery of immunization services. However, not much is known about their everyday endeavours to improve data quality and ensure the submitted data meets the quality and timeliness requirements as per established guidelines. The objective of this paper was to examine the coping mechanisms and strategies adopted to improve the quality and timeliness of immunization data among lower-level private-for-profit service providers in Kampala Capital City, Uganda.

METHODS

A qualitative study design was adopted with in-depth interviews (n = 17) and key informant interviews (n = 8) completed among frontline health workers, district health managers and immunization implementing partners. Analysis followed a thematic approach with coding conducted using Atlas. ti, a qualitative data management software.

RESULTS

Overall, coping mechanisms and strategies adopted to improve the data quality among lower-level urban private-for-profit immunization service providers included; Experiential attachment for practical skills acquisition in data management, data quality peer-to-peer learning among private-for-profit immunization service providers, registration using cohort system for easy tracking of records during subsequent visits, separation of visiting service user records from regular attendants, service delivery modifications such as reward for better performance, engagement of Village Health Teams (VHTs) in outreaches and data completion, and data quality checks through review of monitoring charts to identify gaps in data quality.

CONCLUSIONS

Within the urban context, the delivery of immunization services by lower-level private-for-profit services faces data quality challenges some of which stem from the implementation context. Different coping strategies focusing on frontline health worker skills, enhanced experience sharing through peer-to-peer learning, modifications to registration and routine review of monitoring charts. However, these efforts were often faced with chronic barriers such as the high staff turnover calling for dedicated efforts to optimize the available implementation opportunities like guidelines mandating the public health facilities to supervise the lower-level private immunization service providers within their catchment areas to respond to the identified gaps.

Integrated Immunization Information System in Indonesia: Prototype Design Using Quantitative and Qualitative Data

BACKGROUND

As the volume of immunization records increases, problems with fragmented records arise, especially since the majority of records in developing countries, including Indonesia, remain paper based. Implementing an immunization information system (IIS) offers a solution to this problem.

OBJECTIVE

In this study, we designed an integrated IIS prototype in Indonesia using the design science research (DSR) methodology.

METHODS

The stages of the DSR methodology followed in this study included identifying problems and motivating and defining objectives for a solution, design and development, demonstration, evaluation, communication, and drawing conclusions and suggestions. Specifically, this study began with problem formulation and a literature review. We then applied quantitative (questionnaire with 305 members of the public) and qualitative (interviews with 15 health workers including nurses, midwives, and doctors) data collection approaches.

RESULTS

The resulting high-fidelity prototype follows the 8 golden rules. There are 2 IIS designs, one for the public as immunization recipients and another for health workers. The functionalities include immunization history, schedule, recommendations, verification, certificates, reminders and recalls, coverage, monitoring, news, and reports of adverse events. Evaluation of the prototype was carried out through interviews and a questionnaire designed according to the System Usability Scale (SUS) and Post-Study System Usability Questionnaire (PSSUQ). The SUS value was 72.5 or "Good (Acceptable)," while the system usefulness, information quality, interface quality, and overall value on the PSSUQ were 2.65, 2.94, 2.48, and 2.71, respectively, which indicate it has an effective design.

CONCLUSIONS

This provides a guide for health facilities, health regulators, and health application developers on how to implement an integrated IIS in Indonesia.

Comparative estimates of crude coverage of the Mexican immunization program: Findings from a national survey

The purpose of the study is to provide estimates for immunization coverage, considering single-dose and schemes (three or five vaccines), by comparing self-report method to immunization cards, while also assessing the timeliness of immunization in Mexico, with reference to Mexican Immunization Program guidelines. Data on immunization was obtained from the Mexican Immunization Survey conducted in 2017 that aimed to assess crude (card-based) coverage at the regional level. Timely immunization was defined with reference to National Immunization Program guidelines, and immunization coverage was defined as a three or five vaccine scheme, based on previous national reports of immunization coverage. Immunization coverage estimates account for sample weights from the complex survey design. We used weighted immunization coverage estimates to assess the extent to which immunization cards and self-reporting concurred. It was found that most Mexican children are not receiving their full vaccine schedule in a timely manner. Concerning children under twelve months of age, the coverage targets for National Immunization of 95 % was not reached for either vaccine, and only 2.94 % (95 % CI 0.92-9.01) who had been receiving a three-vaccine scheme were considered as fully immunized in a timely manner. In contrast, coverage increased to 33.94 % (95 % CI 26.99-41.66), when untimely immunizations were taken into account, and the 95 % target was reached for five vaccines. Likewise, there is little correlation between self-report and immunization cards but rates show more concurrence, when only considering the proportion of true positives. In conclusion it was find that children at a local level are vaccinated in an incomplete and untimely manner. In order to improve immunization systems, a nominal registry of administered doses is thus of paramount importance. There is a need to address underlying health inequalities, as well as the factors associated with these, resulting in improved chances of a disease-free childhood and healthy life.

Challenges of vaccination information system implementation: A systematic literature review

Globally, healthcare services have begun to show interest in switching from paper-based to electronic-based vaccination records through Vaccination Information Systems (VIS). VIS have been implemented in various countries, but the study on the challenges of implementing VIS in these countries is still limited. The challenges of implementing VIS need to be understood to become a subject of discussion and anticipation by other countries that are just starting to implement VIS. We analyzed 32 selected publications from 634 initially retrieved. Fourteen challenges were successfully identified when implementing VIS, including interoperability, data quality, security and privacy, standardization, usability, internet connectivity, infrastructure, workflow, funding, government regulations, awareness, skeptical response, computer literacy, and staff-related challenges. The challenges of interoperability and data quality were found to be the most widely discussed by previous studies. In addition to identifying the challenges, this study includes a series of solutions that can be applied to overcome each challenge.

Socioeconomic and geographic inequities in vaccination among children 12 to 59 months in Mexico, 2012 to 2021

Objective

To document the evolution of socioeconomic and geographical inequalities in childhood vaccination in Mexico from 2012 to 2021.

Methods

Repeated cross-sectional analysis using three rounds of National Health and Nutrition Surveys (2012, 2018, and 2021). Dichotomous variables were created to identify the proportion of children who received no dose of each vaccine included in the national immunization schedule (BCG; diphtheria, pertussis, and tetanus-containing; rotavirus; pneumococcal conjugate; and measles, mumps, and rubella [MMR]), and the proportion completely unvaccinated. The distribution of unvaccinated children was analyzed by state, and by socioeconomic status using the concentration index.

Results

The prevalence of completely unvaccinated children in Mexico was low, with 0.3% children in 2012 and 0.8% children in 2021 receiving no vaccines (p = 0.070). Notwithstanding, for each vaccine, an important proportion of children missed receiving any dose. Notably, the prevalence of MMR unvaccinated children was 10.2% (95% CI 9.2-11.1) in 2012, 22.3% (95% CI 20.9-23.8) in 2018, and 29.1% (95% CI 26.3-31.8) in 2021 (p < 0.001 for the difference between 2012 and 2021). The concentration index indicated pro-rich inequalities in non-vaccination for 2 of 5 vaccines in 2012, 3 of 5 vaccines in 2018, and 4 of 5 vaccines in 2021. There were marked subnational variations. The percentage of MMR unvaccinated children ranged from 3.3% to 17.9% in 2012, 5.5% to 36.5% in 2018, and 13.1% to 72.5% in 2021 across the 32 states of Mexico.

Conclusions

Equitable access to basic childhood vaccines in Mexico has deteriorated over the past decade. Vigilant equity monitoring coupled with tailored strategies to reach those left out is urgently required.

Spatial distribution, determinants and trends of full vaccination coverage in children aged 12-59 months in Peru: A subanalysis of the Peruvian Demographic and Health Survey

OBJECTIVE

To assess the spatial distribution, trends and determinants of crude full vaccination coverage (FVC) in children aged 12-59 months between 2010 and 2019 in Peru.

DESIGN, SETTING AND ANALYSIS

A cross-sectional study based on the secondary data analysis of the 2010 and 2019 Peruvian Demographic and Health Surveys (DHSs) was conducted. Logit based multivariate decomposition analysis was employed to identify factors contributing to differences in FVC between 2010 and 2019. The spatial distribution of FVC in 2019 was evaluated through spatial autocorrelation (Global Moran's I), ordinary kriging interpolation (Gaussian process regression) and Bernoulli-based purely spatial scan statistic.

OUTCOME MEASURE

FVC, as crude coverage, was defined as having completely received BCG; three doses of diphtheria, pertussis, and tetanus, and polio vaccines; and measles vaccine by 12 months of age.

PARTICIPANTS

A total of 5 751 and 14 144 children aged 12-59 months from 2010 and 2019 DHSs, respectively, were included.

RESULTS

FVC increased from 53.62% (95% CI 51.75% to 55.49%) in 2010 to 75.86% (95% CI 74.84% to 76.85%) in 2019. Most of the increase (70.39%) was attributable to differences in coefficients effects. Family size, visit of health workers in the last 12 months, age of the mother at first delivery, place of delivery and antenatal care follow-up were all significantly associated with the increase. The trend of FVC was non-linear and increased by 2.22% annually between 2010 and 2019. FVC distribution was heterogeneous at intradepartmental and interdepartmental level. Seven high-risk clusters of incomplete coverage were identified.

CONCLUSIONS

Although FVC has increased in Peru, it still remains below the recommended threshold. The increase of FVC was mainly attributed to the change in the effects of the characteristics of the population. There was high heterogeneity across Peruvian regions with the presence of high-risk clusters. Interventions must be redirected to reduce these geographical disparities.

A realist systematic review of evidence from low- and middle-income countries of interventions to improve immunization data use

BACKGROUND

The use of routine immunization data by health care professionals in low- and middle-income countries remains an underutilized resource in decision-making. Despite the significant resources invested in developing national health information systems, systematic reviews of the effectiveness of data use interventions are lacking. Applying a realist review methodology, this study synthesized evidence of effective interventions for improving data use in decision-making.

METHODS

We searched PubMed, POPLINE, Centre for Agriculture and Biosciences International Global Health, and African Journals Online for published literature. Grey literature was obtained from conference, implementer, and technical agency websites and requested from implementing organizations. Articles were included if they reported on an intervention designed to improve routine data use or reported outcomes related to data use, and targeted health care professionals as the principal data users. We developed a theory of change a priori for how we expect data use interventions to influence data use. Evidence was then synthesized according to data use intervention type and level of the health system targeted by the intervention.

RESULTS

The searches yielded 549 articles, of which 102 met our inclusion criteria, including 49 from peer-reviewed journals and 53 from grey literature. A total of 66 articles reported on immunization data use interventions and 36 articles reported on data use interventions for other health sectors. We categorized 68 articles as research evidence and 34 articles as promising strategies. We identified ten primary intervention categories, including electronic immunization registries, which were the most reported intervention type (n = 14). Among the research evidence from the immunization sector, 32 articles reported intermediate outcomes related to data quality and availability, data analysis, synthesis, interpretation, and review. Seventeen articles reported data-informed decision-making as an intervention outcome, which could be explained by the lack of consensus around how to define and measure data use.

CONCLUSIONS

Few immunization data use interventions have been rigorously studied or evaluated. The review highlights gaps in the evidence base, which future research and better measures for assessing data use should attempt to address.

Critical Review of Data Analytics Techniques used in the Expanded Program on Immunization (EPI)

BACKGROUND

Immunization is a significant public health intervention to reduce child mortality and morbidity. However, its coverage, in spite of free accessibility, is still very low in developing countries. One of the primary reasons for this low coverage is the lack of analysis and proper utilization of immunization data at various healthcare facilities.

PURPOSE

In this paper, the existing machine learning-based data analytics techniques have been reviewed critically to highlight the gaps where this high potential data could be exploited in a meaningful manner.

RESULTS

It has been revealed from our review that the existing approaches use data analytics techniques without considering the complete complexity of Expanded Program on Immunization which includes the maintenance of cold chain systems, proper distribution of vaccine and quality of data captured at various healthcare facilities. Moreover, in developing countries, there is no centralized data repository where all data related to immunization is being gathered to perform analytics at various levels of granularities.

CONCLUSION

We believe that the existing non-centralized immunization data with the right set of machine learning and Artificial Intelligence-based techniques will not only improve the vaccination coverage but will also help in predicting the future trends and patterns of its coverage in different geographical locations.

National vaccine coverage trends and funding in Latin America and the Caribbean

BACKGROUND

From 2006 to 2011 nearly 174,000 deaths were prevented in Latin America and the Caribbean through vaccination of children under five, which is widely attributed to the Expanded Program on Immunization (EPI). Despite near global adoption of EPI recommendations, vaccination coverage shows substantial variation across world regions. Causes for low immunizations within regions are multifaceted and include vaccination program costs. To date, publications regarding vaccine coverage across Latin America and the Caribbean are not readily available. This study aimed to: (1) compare vaccine coverage trends across nations within the region; and (2) assess whether national immunization program expenditures are correlated with vaccine coverage.

METHODS

Coverage for nine vaccines were collected by nation using publicly available data from WHO. National immunization program expenditures for each country were collected from the World Bank Index. The proportion of countries achieving 90% coverage in the years 2013 and 2017 for each vaccine were compared. Pearson correlation coefficients were calculated to measure the relationship between financing variables and DTP3 coverage for 2017.

RESULTS

In 2017, fewer Latin American and Caribbean nations were able to achieve 90% vaccine coverage for five vaccines compared to 2013. Mostly weak to moderate positive relationships were found between national immunization program expenditures and DTP3 coverage for 2017. Excluding Haiti, a weak negative relationship was found between total government expenditure on vaccines per infant and DTP3 coverage for 2017. Countries across Latin America and the Caribbean were largely self-reliant in funding vaccine expenditures.

CONCLUSIONS

Fewer countries across Latin America and the Caribbean are currently achieving optimum national vaccine coverage and weak to moderate relationships between routine immunization and vaccine expenditures and coverage were observed. Additional factors contributing to national vaccine coverage should be concomitantly examined to implement strategies which optimize delivery of childhood immunizations.

Improving the quality and use of immunization and surveillance data: Summary report of the Working Group of the Strategic Advisory Group of Experts on Immunization

Concerns about the quality and use of immunization and vaccine-preventable disease (VPD) surveillance data have been highlighted on the global agenda for over two decades. In August 2017, the Strategic Advisory Group of Experts (SAGE) established a Working Group (WG) onthe Quality and Use of Global Immunization and Surveillance Data to review the current status and evidence to make recommendations, which were presented to SAGE in October 2019. The WG synthesized evidence from landscape analyses, literature reviews, country case-studies, a data triangulation analysis, as well as surveys of experts. Data quality (DQ) was defined as data that are accurate, precise, relevant, complete, and timely enough for the intended purpose (fit-for-purpose), and data use as the degree to which data are actually used for defined purposes, e.g., immunization programme management, performance monitoring, decision-making. The WG outlined roles and responsibilities for immunization and surveillance DQ and use by programme level. The WG found that while DQ is dependent on quality data collection at health facilities, many interventions have targeted national and subnational levels, or have focused on new technologies, rather than the people and enabling environments required for functional information systems. The WG concluded that sustainable improvements in immunization and surveillance DQ and use will require efforts across the health system - governance, people, tools, and processes, including use of data for continuous quality improvement (CQI) - and that the approaches need to be context-specific, country-owned and driven from the frontline up. At the country level, major efforts are needed to: (1) embed monitoring DQ and use alongside monitoring of immunization and surveillance performance, (2) increase workforce capacity and capability for DQ and use, starting at the facility level, (3) improve the accuracy of immunization programme targets (denominators), (4) enhance use of existing data for tailored programme action (e.g., immunization programme planning, management and policy-change), (5) adopt a data-driven CQI approach as part of health system strengthening, (6) strengthen governance around piloting and implementation of new information and communication technology tools, and (7) improve data sharing and knowledge management across areas and organizations for improved transparency and efficiency. Global and regional partners are requested to support countries in adopting relevant recommendations for their setting and to continue strengthening the reporting and monitoring of immunization and VPD surveillance data through processes periodic needs assessment and revision processes. This summary of the WG's findings and recommendations can support "data-guided" implementation of the new Immunization Agenda 2030.

Improving quality and use of routine health information system data in low- and middle-income countries: A scoping review

BACKGROUND

A routine health information system is one of the essential components of a health system. Interventions to improve routine health information system data quality and use for decision-making in low- and middle-income countries differ in design, methods, and scope. There have been limited efforts to synthesise the knowledge across the currently available intervention studies. Thus, this scoping review synthesised published results from interventions that aimed at improving data quality and use in routine health information systems in low- and middle-income countries.

METHOD

We included articles on intervention studies that aimed to improve data quality and use within routine health information systems in low- and middle-income countries, published in English from January 2008 to February 2020. We searched the literature in the databases Medline/PubMed, Web of Science, Embase, and Global Health. After a meticulous screening, we identified 20 articles on data quality and 16 on data use. We prepared and presented the results as a narrative.

RESULTS

Most of the studies were from Sub-Saharan Africa and designed as case studies. Interventions enhancing the quality of data targeted health facilities and staff within districts, and district health managers for improved data use. Combinations of technology enhancement along with capacity building activities, and data quality assessment and feedback system were found useful in improving data quality. Interventions facilitating data availability combined with technology enhancement increased the use of data for planning.

CONCLUSION

The studies in this scoping review showed that a combination of interventions, addressing both behavioural and technical factors, improved data quality and use. Interventions addressing organisational factors were non-existent, but these factors were reported to pose challenges to the implementation and performance of reported interventions.

An assessment of the quality of vaccination data produced through smart paper technology in The Gambia

INTRODUCTION

MyChild Solution is an innovative Electronic Immunisation Register (EIR) reliant on Smart Paper Technology, thereby eliminating the need for electronic devices and internet connectivity at the point-of-care. The goal of this study is to characterise the quality of routine immunisation data generated using MyChild Solution compared to data obtained through the conventional health management information system (HMIS) used in The Gambia.

METHOD

We used the World Health Organization's (WHO) Data Quality Review (DQR) Toolkit to evaluate MyChild Solution's data quality in the 19 health facilities across two regions implementing MyChild Solution in The Gambia at the time of the evaluation. We evaluated all applicable data quality metrics as well as additional metrics of interest, including the incidence of recording errors, the incidence of incomplete indicator level data, and implausible dates. Where possible, we compared results to those of the conventional HMIS.

RESULTS

Both MyChild Solution and the conventional HMIS produced 100% complete and timely data in their reference years. Both systems had no moderate or extreme outliers and showed the expected Penta 1 to Penta 3 dropout direction. However, the proportion of verification factors that are not acceptable was higher in the conventional HMIS. MyChild Solution was found to near perfectly (99.98%) digitise scanned documents. These and other data quality indicators evaluated demonstrate that MyChild Solution produces high quality data with high completeness, timeliness, and consistency compared to the conventional HMIS system.

CONCLUSION

MyChild Solution produces high quality data as per the DQR Toolkit metrics and other metrics of interest of interest. The more internally consitent data produced through MyChild Solution compared to the conventional HMIS demonstrates its potential for supporting data-driven decision-making in immunisation.

A data-driven framework for introducing predictive analytics into expanded program on immunization in Pakistan

BACKGROUND

Pakistan has a nationwide expanded program on immunization (EPI), yet vaccination coverage in Pakistan is quite low. Recently, an analytical model has been proposed to improve the coverage by identifying children who are most likely to miss any of the vaccines included in the immunization schedule, known as defaulters; however, a number of limitations remain unresolved in the previously proposed model. Firstly, it only classified children into two stages: defaulters and non-defaulters, considering all children at high risk of defaulting even if only one dose is missed. Secondly, there was no categorisation of high and low coverage areas for prioritised vaccination. The aim of this study was to propose a prediction framework for the accurate identification of defaulters.

METHODS

We have utilised a sample dataset extracted from the Pakistan Demographic and Health Survey (PDHS, 2017-2018). This contained 7153 data records with 19 demographic and socioeconomic attributes, which were used for defaulter prediction and the identification of association rules to understand the relation between demographics of the child and the vaccination status.

RESULTS

Using a multilayer perceptron (MLP) classifier, the proposed model achieved 98% accuracy and 0.994 for the area under the curve (AUC), to correctly identify the children who are likely to default from immunization series at different risk stages.

CONCLUSION

The proposed framework in this study is a step forward towards a data-driven approach and provides a set of machine learning techniques to utilise predictive analytics. Hence, this can reinforce immunization programs by expediting targeted action to reduce drop-outs.

Redefining vaccination coverage and timeliness measures using electronic immunization registry data in low- and middle-income countries

Vaccine coverage is routinely used as a performance indicator for immunization programs both at local and global levels. For many national immunization programs, there are challenges with accurately estimating vaccination coverage based on available data sources, however an increasing number of low- and middle-income countries (LMICs) have begun implementing electronic immunization registries to replace health facilities’ paper-based tools and aggregate reporting systems. These systems allow for more efficient capture and use of routinely reported individual-level data that can be used to calculate dose-specific and cohort vaccination coverage, replacing the commonly used aggregate routine health information system data. With these individual-level data immunization programs have the opportunity to redefine performance measures to enhance programmatic decision-making at all levels of the health system. In this commentary, we discuss how measures for assessing vaccination status and program performance can be redefined and recalculated using these data when generated at the health facility level and the implications of the use and availability of electronic individual-level data.