Electronic immunization data collection systems: application of an evaluation framework

Availability of data for cost-effectiveness comparison of child vision and hearing screening programmes

OBJECTIVE

For cost-effectiveness comparison of child vision and hearing screening programmes, methods and data should be available. We assessed the current state of data collection and its availability in Europe.

METHODS

The EUSCREEN Questionnaire, conducted in 2017-2018, assessed paediatric vision and hearing screening programmes in 45 countries in Europe. For the current study, its items on data collection, monitoring and evaluation, and six of eleven items essential for cost-effectiveness analysis: prevalence, sensitivity, specificity, coverage, attendance and loss to follow-up, were reappraised with an additional questionnaire.

RESULTS

The practice of data collection in vision screening was reported in 36% (N = 42) of countries and in hearing screening in 81% (N = 43); collected data were published in 12% and 35%, respectively. Procedures for quality assurance in vision screening were reported in 19% and in hearing screening in 26%, research of screening effectiveness in 43% and 47%, whereas cost-effectiveness analysis was performed in 12% for both. Data on prevalence of amblyopia were reported in 40% and of hearing loss in 77%, on sensitivity of screening tests in 17% and 14%, on their specificity in 19% and 21%, on coverage of screening in 40% and 84%, on attendance in 21% and 37%, and on loss to follow-up in 12% and 40%, respectively.

CONCLUSIONS

Data collection is insufficient in hearing screening and even more so in vision screening: data essential for cost-effectiveness comparison could not be reported from most countries. When collection takes place, this is mostly at a local level for quality assurance or accountability, and data are often not accessible. The resulting inability to compare cost-effectiveness among screening programmes perpetuates their diversity and inefficiency.

Public health surveillance in the U.S. Department of Veterans Affairs: evaluation of the Praedico surveillance system

Background

Early threat detection and situational awareness are vital to achieving a comprehensive and accurate view of health-related events for federal, state, and local health agencies. Key to this are public health and syndromic surveillance systems that can analyze large data sets to discover patterns, trends, and correlations of public health significance. In 2020, Department of Veterans Affairs (VA) evaluated its public health surveillance system and identified areas for improvement.

Methods

Using the Centers for Disease Control and Prevention (CDC) Guidelines for Evaluating Public Health Surveillance Systems, we assessed the ability of the Praedico Surveillance System to perform public health surveillance for a variety of health issues and evaluated its performance compared to an enterprise data solution (VA Corporate Data Warehouse), legacy surveillance system (VA ESSENCE) and a national, collaborative syndromic surveillance platform (CDC NSSP BioSense).

Results

Review of system attributes found that the system was simple, flexible, and stable. Representativeness, timeliness, sensitivity, and Predictive Value Positive were acceptable but could be further improved. Data quality issues and acceptability present challenges that potentially affect the overall usefulness of the system.

Conclusions

Praedico is a customizable surveillance and data analytics platform built on big data technologies. Functionality is straightforward, with rapid query generation and runtimes. Data can be graphed, mapped, analyzed, and shared with key decision makers and stakeholders. Evaluation findings suggest that future development and system enhancements should focus on addressing Praedico data quality issues and improving user acceptability. Because Praedico is designed to handle big data queries and work with data from a variety of sources, it could be enlisted as a tool for interdepartmental and interagency collaboration and public health data sharing. We suggest that future system evaluations include measurements of value and effectiveness along with additional organizations and functional assessments.

Validation of 4D Components for Measuring Quality of the Public Health Data Collection Process: Elicitation Study

Background

Identification of the essential components of the quality of the data collection process is the starting point for designing effective data quality management strategies for public health information systems. An inductive analysis of the global literature on the quality of the public health data collection process has led to the formation of a preliminary 4D component framework, that is, data collection management, data collection personnel, data collection system, and data collection environment. It is necessary to empirically validate the framework for its use in future research and practice.

Objective

This study aims to obtain empirical evidence to confirm the components of the framework and, if needed, to further develop this framework.

Methods

Expert elicitation was used to evaluate the preliminary framework in the context of the Chinese National HIV/AIDS Comprehensive Response Information Management System. The research processes included the development of an interview guide and data collection form, data collection, and analysis. A total of 3 public health administrators, 15 public health workers, and 10 health care practitioners participated in the elicitation session. A framework qualitative data analysis approach and a quantitative comparative analysis were followed to elicit themes from the interview transcripts and to map them to the elements of the preliminary 4D framework.

Results

A total of 302 codes were extracted from interview transcripts. After iterative and recursive comparison, classification, and mapping, 46 new indicators emerged; 24.8% (37/149) of the original indicators were deleted because of a lack of evidence support and another 28.2% (42/149) were merged. The validated 4D component framework consists of 116 indicators (82 facilitators and 34 barriers). The first component, data collection management, includes data collection protocols and quality assurance. It was measured by 41 indicators, decreased from the original 49% (73/149) to 35.3% (41/116). The second component, data collection environment, was measured by 37 indicators, increased from the original 13.4% (20/149) to 31.9% (37/116). It comprised leadership, training, funding, organizational policy, high-level management support, and collaboration among parallel organizations. The third component, data collection personnel, includes the perception of data collection, skills and competence, communication, and staffing patterns. There was no change in the proportion for data collection personnel (19.5% vs 19.0%), although the number of its indicators was reduced from 29 to 22. The fourth component, the data collection system, was measured using 16 indicators, with a slight decrease in percentage points from 18.1% (27/149) to 13.8% (16/116). It comprised functions, system integration, technical support, and data collection devices.

Conclusions

This expert elicitation study validated and improved the 4D framework. The framework can be useful in developing a questionnaire survey instrument for measuring the quality of the public health data collection process after validation of psychometric properties and item reduction.

Critical Review of Data Analytics Techniques used in the Expanded Program on Immunization (EPI)

BACKGROUND

Immunization is a significant public health intervention to reduce child mortality and morbidity. However, its coverage, in spite of free accessibility, is still very low in developing countries. One of the primary reasons for this low coverage is the lack of analysis and proper utilization of immunization data at various healthcare facilities.

PURPOSE

In this paper, the existing machine learning-based data analytics techniques have been reviewed critically to highlight the gaps where this high potential data could be exploited in a meaningful manner.

RESULTS

It has been revealed from our review that the existing approaches use data analytics techniques without considering the complete complexity of Expanded Program on Immunization which includes the maintenance of cold chain systems, proper distribution of vaccine and quality of data captured at various healthcare facilities. Moreover, in developing countries, there is no centralized data repository where all data related to immunization is being gathered to perform analytics at various levels of granularities.

CONCLUSION

We believe that the existing non-centralized immunization data with the right set of machine learning and Artificial Intelligence-based techniques will not only improve the vaccination coverage but will also help in predicting the future trends and patterns of its coverage in different geographical locations.

Impact of COVID-19 on vaccination programs: adverse or positive?

The COVID-19 pandemic has posed multiple substantial challenges, affecting not only public health but also economic systems, socio-cultural patterns, and political institutions. Studies have focused on the relationships between complex emergencies and natural disasters with outbreaks of infectious diseases. However, there is a dearth of relevant literature on the impact of a global pandemic on vaccination programs - an important topic because delays or stops in such programs are likely to result in outbreaks and epidemics of other infectious diseases. Thus, this article discusses the negative and positive impacts that the COVID-19 pandemic may exert on vaccination for vaccine-preventable diseases (VPDs). Negative impacts include the increased risk of VPD outbreaks in low-resource countries where vaccination programs must be temporarily halted to prevent the spread of infection. Positive effects include the strong possibility that the universally-recognized need for a coronavirus vaccine may increase people's appreciation for vaccines in general, resulting in improved vaccination uptake once the pandemic passes. Concerned stakeholders, such as governments and the World Health Organization (WHO), should seize this moment to effectively build on these positive impacts by planning renewed and revitalized post-COVID vaccination programs.

Assessment of VaxTrac electronic immunization registry in an urban district in Sierra Leone: Implications for data quality, defaulter tracking, and policy

BACKGROUND

In 2016, the Sierra Leone Ministry of Health and Sanitation (MoHS) piloted VaxTrac, an electronic immunization registry (EIR), in an urban district to improve management of vaccination records and tracking of children who missed scheduled doses. We aimed to document lessons learned to inform decision-making on VaxTrac and similar EIRs' future use.

METHODS

Ten out of 50 urban health facilities that implemented VaxTrac were purposively selected for inclusion in a rapid mixed-method assessment from November to December 2017. For a one-month period, records of six scheduled vaccine doses among children < 2 years old in VaxTrac were abstracted and compared to three paper-based records (register of under-two children, daily tally sheet, and monthly summary form). We used the under-two register as the reference gold standard for comparison purposes. We interviewed and observed 10 heath workers, one from each selected facility, who were using VaxTrac.

RESULTS

Overall, VaxTrac captured < 65% of the vaccine doses reported in the paper-based sources, but in the largest health facility VaxTrac captured the highest number of doses. Two additional notable patterns emerged: 1) the aggregated data sources reported higher doses administered compared to the under-two register and VaxTrac; 2) data sources that need real-time data capture during the vaccination session reported fewer doses administered compared to the monthly HF2 summary form. Health workers expressed that the EIR helped them to shorten the time to manage, summarize, and report vaccination records. Workflows for data entry in VaxTrac were inconsistent among facilities and rarely integrated into existing processes. Data sharing restrictions contributed to duplicate records.

CONCLUSION

Although VaxTrac helped to shorten the time to manage, summarize, and report vaccination records, data sharing restrictions coupled with inconsistent and inefficient workflows were major implementation challenges. Readiness-to-introduce and sustainability should be carefully considered before implementing an EIR.

Identification of the essential components of quality in the data collection process for public health information systems

This study identifies essential components in the data collection process for public health information systems based on appraisal and synthesis of the reported factors affecting this process in the literature. Extant process assessment instruments and studies of public health data collection from electronic databases and the relevant institutional websites were reviewed and analyzed following a five-stage framework. Four dimensions covering 12 factors and 149 indicators were identified. The first dimension, data collection management, includes data collection system and quality assurance. The second dimension, data collector, is described by staffing pattern, skill or competence, communication and attitude toward data collection. The third, information system, is assessed by function and technology support, integration of different data collection systems, and device. The fourth dimension, data collection environment, comprises training, leadership, and funding. With empirical testing and contextual analysis, these essential components can be further used to develop a framework for measuring the quality of the data collection process for public health information systems.

Methods used for immunization coverage assessment in Canada, a Canadian Immunization Research Network (CIRN) study

Accurate and complete immunization data are necessary to assess vaccine coverage, safety and effectiveness. Across Canada, different methods and data sources are used to assess vaccine coverage, but these have not been systematically described. Our primary objective was to examine and describe the methods used to determine immunization coverage in Canada. The secondary objective was to compare routine infant and childhood coverage estimates derived from the Canadian 2013 Childhood National Immunization Coverage Survey (cNICS) with estimates collected from provinces and territories (P/Ts). We collected information from key informants regarding their provincial, territorial or federal methods for assessing immunization coverage. We also collected P/T coverage estimates for select antigens and birth cohorts to determine absolute differences between these and estimates from cNICS. Twenty-six individuals across 16 public health organizations participated between April and August 2015. Coverage surveys are conducted regularly for toddlers in Quebec and in one health authority in British Columbia. Across P/Ts, different methodologies for measuring coverage are used (e.g., valid doses, grace periods). Most P/Ts, except Ontario, measure up-to-date (UTD) coverage and 4 P/Ts also assess on-time coverage. The degree of concordance between P/T and cNICS coverage estimates varied by jurisdiction, antigen and age group. In addition to differences in the data sources and processes used for coverage assessment, there are also differences between Canadian P/Ts in the methods used for calculating immunization coverage. Comparisons between P/T and cNICS estimates leave remaining questions about the proportion of children fully vaccinated in Canada.

Comparing laboratory surveillance with the notifiable diseases surveillance system in South Africa

OBJECTIVE

The aim of this study was to compare laboratory surveillance with the notifiable diseases surveillance system (NDSS) in South Africa.

METHODS

Data on three tracer notifiable diseases - measles, meningococcal meningitis, and typhoid - were compared to assess data quality, stability, representativeness, sensitivity and positive predictive value (PPV), using the Wilcoxon and Chi-square tests, at the 5% significance level.

RESULTS

For all three diseases, fewer cases were notified than confirmed in the laboratory. Completeness for the laboratory system was higher for measles (63% vs. 47%, p<0.001) and meningococcal meningitis (63% vs. 57%, p<0.001), but not for typhoid (60% vs. 63%, p=0.082). Stability was higher for the laboratory (all 100%) compared to notified measles (24%, p<0.001), meningococcal meningitis (74%, p<0.001), and typhoid (36%, p<0.001). Representativeness was also higher for the laboratory (all 100%) than for notified measles (67%, p=0.058), meningococcal meningitis (56%, p=0.023), and typhoid (44%, p=0.009). The sensitivity of the NDSS was 50%, 98%, and 93%, and the PPV was 20%, 57%, and 81% for measles, meningococcal meningitis, and typhoid, respectively.

CONCLUSIONS

Compared to laboratory surveillance, the NDSS performed poorly on most system attributes. Revitalization of the NDSS in South Africa is recommended to address the completeness, stability, and representativeness of the system.

Immunization information systems in Canada: Attributes, functionality, strengths and challenges. A Canadian Immunization Research Network study

OBJECTIVES

Canada does not have a national immunization registry. Diverse systems to record vaccine uptake exist, but these have not been systematically described. Our objective was to describe the immunization information systems (IISs) and non-IIS processes used to record childhood and adolescent vaccinations, and to outline the strengths and limitations of the systems and processes.

METHODS

We collected information from key informants regarding their provincial, territorial or federal organization's surveillance systems for assessing immunization coverage. Information collection consisted of a self-administered questionnaire and a follow-up interview. We evaluated systems against attributes derived from the literature using content analysis.

RESULTS

Twenty-six individuals across 16 public health organizations participated over the period of April to August 2015. Twelve of Canada's 13 provinces and territories (P/Ts) and two organizations involved in health service delivery for on-reserve First Nations people participated. Across systems, there were differences in data collection processes, reporting capabilities and advanced functionality. Commonly cited challenges included timeliness and data completeness of records, particularly for physician-administered immunizations. Privacy considerations and the need for data standards were stated as challenges to the goal of information sharing across P/T systems. Many P/Ts have recently implemented new systems and, in some cases, legislation to improve timeliness and/or completeness.

CONCLUSION

Considerable variability exists among IISs and non-IIS processes used to assess immunization coverage in Canada. Although some P/Ts have already pursued legislative or policy initiatives to address the completeness and timeliness of information, many additional opportunities exist in the information technology realm.

Effectiveness of Implementation of Electronic Malaria Information System as the National Malaria Surveillance System in Thailand

Background

In moving toward malaria elimination, one strategy is to implement an active surveillance system for effective case management. Thailand has developed and implemented the electronic Malaria Information System (eMIS) capturing individualized electronic records of suspected or confirmed malaria cases.

Objective

The main purpose of this study was to determine how well the eMIS improves the quality of Thailand’s malaria surveillance system. In particular, the focus of the study was to evaluate the effectiveness of the eMIS in terms of the system users’ perception and the system outcomes (ie, quality of data) regarding the management of malaria patients.

Methods

A mixed-methods technique was used with the framework based on system effectiveness attributes: data quality, timeliness, simplicity, acceptability, flexibility, stability, and usefulness. Three methods were utilized: data records review, survey of system users, and in-depth interviews with key stakeholders. From the two highest endemic provinces, paper forms matching electronic records of 4455 noninfected and 784 malaria-infected cases were reviewed. Web-based anonymous questionnaires were distributed to all 129 eMIS data entry staff throughout Thailand, and semistructured interviews were conducted with 12 management-level officers.

Results

The eMIS is well accepted by system users at both management and operational levels. The data quality has enabled malaria personnel to perform more effective prevention and control activities. There is evidence of practices resulting in inconsistencies and logical errors in data reporting. Critical data elements were mostly completed, except for a few related to certain dates and area classifications. Timeliness in reporting a case to the system was acceptable with a delay of 3-4 days. The evaluation of quantitative and qualitative data confirmed that the eMIS has high levels of simplicity, acceptability, stability, and flexibility.

Conclusions

Overall, the system implemented has achieved its objective. The results of the study suggested that the eMIS helps improve the quality of Thailand’s malaria surveillance system. As the national malaria surveillance system, the eMIS’s functionalities have provided the malaria staff working at the point of care with close-to-real-time case management data quality, covering case detection, case investigation, drug compliance, and follow-up visits. Such features has led to an improvement in the quality of the malaria control program; the government officials now have quicker access to both individual and aggregated data to promptly react to possible outbreak. The eMIS thus plays one of the key roles in moving toward the national goal of malaria elimination by the next decade.