The Value Added of Incorporating Qualitative Approaches into Malaria Surveillance, Monitoring, and Evaluation

Progress toward malaria elimination and improvements in the performance of national malaria control programs (NMCPs) have stalled in recent years. The current COVID-19 pandemic further threatens building on previous gains. Surveillance, monitoring, and evaluation (SME) are critical for the continued success of NMCPs because they provide the information necessary for effective program planning and management. Interventions aimed at strengthening NMCPs focus on both the target population and the program provider. Qualitative approaches are often used to understand the target population and barriers to intervention success. Although there is growing emphasis on qualitative approaches in provider-focused SME, metrics of success tend to focus on quantitative measures. The integration of qualitative approaches offers added value because they provide additional data points to facilitate the understanding of barriers that impede sustaining the gains made from provider-focused capacity-building efforts. Qualitative approaches focus on understanding program implementation and interventions, but the systematic integration of qualitative data is limited. Qualitative approaches provide avenues to strengthen SME efforts, can lead to subsequent improvement for NMCPs, and fuel progress toward malaria elimination.

Evaluating malaria programmes in moderate- and low-transmission settings: practical ways to generate robust evidence

BACKGROUND

Many countries have made substantial progress in scaling-up and sustaining malaria intervention coverage, leading to more focalized and heterogeneous transmission in many settings. Evaluation provides valuable information for programmes to understand if interventions have been implemented as planned and with quality, if the programme had the intended impact on malaria burden, and to guide programmatic decision-making. Low-, moderate-, and heterogeneous-transmission settings present unique evaluation challenges because of dynamic and targeted intervention strategies. This paper provides illustration of evaluation approaches and methodologies for these transmission settings, and suggests how to answer evaluation questions specific to the local context.

METHODS

The Roll Back Malaria Monitoring and Evaluation Reference Group formed a task force in October 2017 to lead development of this framework. The task force includes representatives from National Malaria Programmes, funding agencies, and malaria research and implementing partners. The framework builds on existing guidance for process and outcome evaluations and impact evaluations specifically in high transmission settings.

RESULTS

The theory of change describes how evaluation questions asked by national malaria programmes in different contexts influence evaluation design. The transmission setting, existing stratification, and data quality and availability are also key considerations. The framework is intended for adaption by countries to their local context, and use for evaluation at sub-national level. Confirmed malaria incidence is recommended as the primary impact indicator due to its sensitivity to detect changes in low-transmission settings. It is expected that process evaluations provide sufficient evidence for programme monitoring and improvement, while impact evaluations are needed following adoption of new mixes of interventions, operational strategies, tools or policies, particularly in contexts of changing malaria epidemiology. Impact evaluations in low-, moderate-, or heterogeneous-transmission settings will likely use plausibility designs, and methods highlighted by the framework include interrupted time series, district-level dose-response analyses, and constructed control methods. Triangulating multiple data sources and analyses is important to strengthen the plausibility argument.

CONCLUSIONS

This framework provides a structure to assist national malaria programmes and partners to design evaluations in low-, moderate- or heterogeneous-transmission settings. Emphasizing a continuous cycle along the causal pathway linking process evaluation to impact evaluation and then programmatic decision-making, the framework provides practical guidance in evaluation design, analysis, and interpretation to ensure that the evaluation meets national malaria programme priority questions and guides decision-making at national and sub-national levels.

Evidence of Impact: iCCM as a strategy to save lives of children under five

Background

In 2013, the World Health Organization (WHO) launched the Rapid Access Expansion (RAcE) programme in the Democratic Republic of Congo, Malawi, Mozambique, Niger, and Nigeria to increase coverage of diagnostic, treatment, and referral services for malaria, pneumonia, and diarrhea among children ages 2-59 months. In 2017, a final evaluation of the six RAcE sites was conducted to determine whether the programme goal was reached. A key evaluation objective was to estimate the reduction in childhood mortality and the number of under-five lives saved over the project period in the RAcE project areas.

Methods

The Lives Saved Tool (LiST) was used to estimate reductions in all-cause child mortality due to changes in coverage of treatment for the integrated community case management (iCCM) illnesses – malaria, pneumonia, and diarrhea – while accounting for other changes in maternal and child health interventions in each RAcE project area. Data from RAcE baseline and endline household surveys, Demographic and Health Surveys, and routine health service data were used in each LiST model. The models yielded estimated change in under-five mortality rates, and estimated number of lives saved per year by malaria, pneumonia and diarrhea treatment. We adjusted the results to estimate the number of lives saved by community health worker (CHW)-provided treatment.

Results

The LiST model accounts for coverage changes in iCCM intervention coverage and other health trends in each project area to estimate mortality reduction and child lives saved. Under five mortality declined in all six RAcE sites, with an average decline of 10 percent. An estimated 6200 under-five lives were saved by malaria, pneumonia, and diarrhea treatment in the DRC, Malawi, Niger, and Nigeria, of which approximately 4940 (75 percent) were saved by treatment provided by CHWs. This total excludes Mozambique, where there were no estimated under-five lives saved likely due to widespread stockouts of key medications. In all other project areas, lives saved by CHW-provided treatment contributed substantially to the estimated decline in under-five mortality.

Conclusions

Our results suggest that iCCM is a strategy that can save lives and measurably decrease child mortality in settings where access to health facility services is low and adequate resources for iCCM implementation are provided for CHW services.

Data quality assessments stimulate improvements to health management information systems: evidence from five African countries

Background

Health service data are used to inform decisions about planning and implementation, as well as to evaluate performance and outcomes, and the quality of those data are important. Data quality assessments (DQA) afford the opportunity to collect information about health service data. Through its Rapid Access Expansion Programme (RAcE), the World Health Organization (WHO) funded non-governmental organizations (NGO) to support Ministries of Health (MOH) in implementing integrated community case management (iCCM) programs in the Democratic Republic of Congo, Malawi, Mozambique, Niger and Nigeria. WHO contracted ICF to support grantee monitoring and evaluation efforts, part of which was to conduct DQAs to enhance program monitoring and decision making. The contribution of DQAs to data-driven decision making has been documented and the purpose of this paper is to describe how DQAs contributed to health management information system (HMIS) strengthening and the findings of subsequent DQAs in those areas.

Methods

ICF created a mixed-methods DQA for iCCM data, comprising a review of the data collection and management system, a data tracing component and key informant interviews. The DQA was applied twice in each RAcE site, which enables a general comparison of system-level attributes before and after the first DQA application. For this qualitative assessment, we reviewed DQA reports to collate information about DQA recommendations and how they were addressed before a subsequent DQA, along with the findings of the second DQA.

Results

Findings from the first DQA in each RAcE site stimulated NGO and MOH efforts to strengthen different aspects of the HMIS in each country, including modifying data collection tools in the Democratic Republic of Congo; training community health workers (CHWs) and supervisors in Malawi; strengthening supervision in Mozambique; improving CHW registers and strengthening staff capacity at all levels to report data in Niger; establishing a data review system in Abia State, Nigeria; and, establishing processes to improve data use and quality in Niger State, Nigeria.

Conclusion

Data quality assessments stimulated context-specific efforts by NGOs and MOHs to improve iCCM data quality. DQAs can serve as a collaborative and evidence-based activity to influence discussions of data quality and stimulate HMIS strengthening efforts.

Achievements and challenges of implementation in a mature iCCM programme: Malawi case study

Background

Malawi has a mature integrated community case management (iCCM) programme that is led by the Ministry of Health (MOH) but that still relies on donor support. From 2013 until 2017, under the Rapid Access Expansion (RAcE) programme, the World Health Organization supported the MOH to expand and strengthen iCCM services in four districts. This paper examines Malawi’s iCCM programme performance and implementation strength in RAcE districts to further strengthen the broader programme.

Methods

Baseline and endline household surveys were conducted in iCCM-eligible areas of RAcE districts. Primary caregivers of recently-sick children under five were interviewed to assess changes in care-seeking and treatment over the project period. Health surveillance assistants (HSAs) were surveyed at endline to assess iCCM implementation strength.

Results

Care-seeking from HSAs and treatment of fever improved over the project period. At endline, however, less than half of sick children were brought to an HSA, many caregivers reported a preference for providers other than HSAs, and perceptions of HSAs as trusted providers of high-quality, convenient care had decreased. HSA supervision and mentorship were below MOH targets. Stockouts of malaria medicines were associated with decreased care-seeking from HSAs. Thirty percent of clusters had limited or no access to iCCM (no HSA or an HSA providing iCCM services less than 2 days per week); 50% had moderate access (an HSA providing iCCM services 2 to 4 days per week; and 20% had high access (a resident HSA providing iCCM services 5 or more days per week). Moderate access to iCCM was associated with increased care-seeking from HSAs, increased treatment by HSAs, and more positive perceptions of HSAs compared to areas with limited or no access. Areas with high access to iCCM did not show further improvements above areas with moderate access.

Conclusions

Availability of well-equipped and supported HSAs is critical to the provision of iCCM services. Additional qualitative research is needed to examine challenges and to inform potential solutions. Malawi’s mature iCCM programme has a strong foundation but can be improved to strengthen the continuity of care from communities to facilities and to ultimately improve child health outcomes.

iCCM data quality: an approach to assessing iCCM reporting systems and data quality in 5 African countries

BACKGROUND

Ensuring the quality of health service data is critical for data-driven decision-making. Data quality assessments (DQAs) are used to determine if data are of sufficient quality to support their intended use. However, guidance on how to conduct DQAs specifically for community-based interventions, such as integrated community case management (iCCM) programs, is limited. As part of the World Health Organization's (WHO) Rapid Access Expansion (RAcE) Programme, ICF conducted DQAs in a unique effort to characterize the quality of community health worker-generated data and to use DQA findings to strengthen reporting systems and decision-making.

METHODS

We present our experience implementing assessments using standardized DQA tools in the six RAcE project sites in the Democratic Republic of Congo, Malawi, Mozambique, Niger, and Nigeria. We describe the process used to create the RAcE DQA tools, adapt the tools to country contexts, and develop the iCCM DQA Toolkit, which enables countries to carry out regular and rapid DQAs. We provide examples of how we used results to generate recommendations.

RESULTS

The DQA tools were customized for each RAcE project to assess the iCCM data reporting system, trace iCCM indicators through this system, and to ensure that DQAs were efficient and generated useful recommendations. This experience led to creation of an iCCM DQA Toolkit comprised of simplified versions of RAcE DQA tools and a guidance document. It includes system assessment questions that elicit actionable responses and a simplified data tracing tool focused on one treatment indicator for each iCCM focus illness: diarrhea, malaria, and pneumonia. The toolkit is intended for use at the national or sub-national level for periodic data quality checks.

CONCLUSIONS

The iCCM DQA Toolkit was designed to be easily tailored to different data reporting system structures because iCCM data reporting tools and data flow vary substantially. The toolkit enables countries to identify points in the reporting system where data quality is compromised and areas of the reporting system that require strengthening, so that countries can make informed adjustments that improve data quality, strengthen reporting systems, and inform decision-making.

Effect of community-based intervention on improving access to treatment for sick under-five children in hard-to-reach communities in Niger State, Nigeria

BACKGROUND

Access to prompt and appropriate treatment is key to survival for children with malaria, pneumonia and diarrhoea. Community-based services are vital to extending care to remote populations. Malaria Consortium supported Niger state Ministry of Health, Nigeria, to introduce and implement an integrated community case management (iCCM) programme for four years in six local government areas (LGAs). The objective was to increase coverage of effective treatment for malaria, pneumonia and diarrhoea among children aged 2-59 months.

METHODS

The programme involved training, equipping, ongoing support and supervision of 1320 community volunteers (CORPs) to provide iCCM services to their communities in all six LGAs. Demand creation activities were also conducted; these included community dialogues, household mobilization, sensitization and mass media campaigns targeted at programme communities. To assess the level of changes in care seeking and treatment, baseline and endline household surveys were conducted in 2014 and 2017 respectively. For both surveys, a 30×30 multi-stage cluster sampling method was used, the sampling frame being RAcE programme communities.

RESULTS

Care-seeking from an appropriate provider increased overall and for each iCCM illness from 78% to 94% for children presenting with fever (P < 0.01), from 72% to 91% for diarrhoea cases (P < 0.01), and from 76% to 89% for cases of cough with difficult or fast breathing (P < 0.05). For diagnosis and treatment, the coverage of fevers tested for malaria increased from 34% to 77% (P < 0.001) and ACT treatments from 57% to 73% (<0.005); 56% of cases of cough or fast breathing who sought care from a CORP, had their respiratory rate counted and 61% with cough or fast breathing received amoxicillin. At endline caregivers sought care from CORPs in their communities for most cases of childhood illnesses (84%) compared to other providers at hospitals (1%) or health centres (9%).This aligns with caregivers' belief that CORPs are trusted providers (94%) who provide quality services (96%).

CONCLUSION

Implementation of iCCM with focused demand creation activities can improve access to quality lifesaving interventions from frontline community providers in Nigeria. This can contribute towards achieving SDGs if iCCM is scaled up to hard-to-reach areas of all states in the country.

How do we know? An assessment of integrated community case management data quality in four districts of Malawi

The World Health Organization contracted annual data quality assessments of Rapid Access Expansion (RAcE) projects to review integrated community case management (iCCM) data quality and the monitoring and evaluation (M&E) system for iCCM, and to suggest ways to improve data quality. The first RAcE data quality assessment was conducted in Malawi in January 2014 and we present findings pertaining to data from the health management information system at the community, facility and other sub-national levels because RAcE grantees rely on that for most of their monitoring data. We randomly selected 10 health facilities (10% of eligible facilities) from the four RAcE project districts, and collected quantitative data with an adapted and comprehensive tool that included an assessment of Malawi's M&E system for iCCM data and a data verification exercise that traced selected indicators through the reporting system. We rated the iCCM M&E system across five function areas based on interviews and observations, and calculated verification ratios for each data reporting level. We also conducted key informant interviews with Health Surveillance Assistants and facility, district and central Ministry of Health staff. Scores show a high-functioning M&E system for iCCM with some deficiencies in data management processes. The system lacks quality controls, including data entry verification, a protocol for addressing errors, and written procedures for data collection, entry, analysis and management. Data availability was generally high except for supervision data. The data verification process identified gaps in completeness and consistency, particularly in Health Surveillance Assistants' record keeping. Staff at all levels would like more training in data management. This data quality assessment illuminates where an otherwise strong M&E system for iCCM fails to ensure some aspects of data quality. Prioritizing data management with documented protocols, additional training and approaches to create efficient supervision practices may improve iCCM data quality.

Community-based intervention packages facilitated by NGOs demonstrate plausible evidence for child mortality impact

INTRODUCTION

Evidence exists that community-based intervention packages can have substantial child and newborn mortality impact, and may help more countries meet Millennium Development Goal 4 (MDG 4) targets. A non-governmental organization (NGO) project using such programming in Mozambique documented an annual decline in under-five mortality rate (U5MR) of 9.3% in a province in which Demographic and Health Survey (DHS) data showed a 4.2% U5MR decline during the same period. To test the generalizability of this finding, the same analysis was applied to a group of projects funded by the US Agency for International Development. Projects supported implementation of community-based intervention packages aimed at increasing use of health services while improving preventive and home-care practices for children under five.

METHODS

All projects collect baseline and endline population coverage data for key child health interventions. Twelve projects fitted the inclusion criteria. U5MR decline was estimated by modelling these coverage changes in the Lives Saved Tool (LiST) and comparing with concurrent measured DHS mortality data.

RESULTS

Average coverage changes for all interventions exceeded average concurrent trends. When population coverage changes were modelled in LiST, they were estimated to give a child mortality improvement in the project area that exceeded concurrent secular trend in the subnational DHS region in 11 of 12 cases. The average improvement in modelled U5MR (5.8%) was more than twice the concurrent directly measured average decline (2.5%).

CONCLUSIONS

NGO projects implementing community-based intervention packages appear to be effective in reducing child mortality in diverse settings. There is plausible evidence that they raised coverage for a variety of high-impact interventions and improved U5MR by more than twice the concurrent secular trend. All projects used community-based strategies that achieved frequent interpersonal contact for health behaviour change. Further study of the effectiveness and scalability of similar packages should be part of the effort to accelerate progress towards MDG 4.