Strengthening effectiveness evaluations to improve programs for women, children and adolescents

Evaluation of a comprehensive maternal newborn health intervention in rural Tanzania: single-arm pre-post coverage survey results

Background

In Tanzania, maternal and newborn deaths can be prevented via quality facility-based antenatal care (ANC), delivery, and postnatal care (PNC). Scalable, integrated, and comprehensive interventions addressing demand and service-side care-seeking barriers are needed.

Objective

Assess coverage survey indicators before and after a comprehensive maternal newborn health (MNH) intervention in Misungwi District, Tanzania.

Methods

A prospective, single-arm, pre- (2016) and post-(2019) coverage survey (ClinicalTrials.gov #NCT02506413) was used to assess key maternal and newborn health (MNH) outcomes. The Mama na Mtoto intervention included district activities (planning, leadership training, supportive supervision), health facility activities (training, equipment, infrastructure upgrades), and plus community health worker mobilization. Implementation change strategies, a process model, and a motivational framework incorporated best practices from a similar Ugandan intervention. Cluster sampling randomized hamlets then used ‘wedge sampling’ protocol as an alternative to full household enumeration. Key outcomes included: four or more ANC visits (ANC4+); skilled birth attendant (SBA); PNC for mother within 48 hours (PNC-woman); health facility delivery (HFD); and PNC for newborn within 48 hours (PNC-baby). Trained interviewers administered the ‘Real Accountability: Data Analysis for Results Coverage Survey to women 15–49 years old. Descriptive statistics incorporated design effect; the Lives Saved Tool estimated deaths averted based on ANC4+/HFD.

Results

Between baseline (n = 2,431) and endline (n = 2,070), surveys revealed significant absolute percentage increases for ANC4+ (+11.6, 95% CI [5.4, 17.7], p < 0.001), SBA (+16.6, 95% CI [11.1, 22.0], p < 0.001), PNC-woman (+9.2, 95% CI [3.2, 15.2], p = 0.002), and HFD (+17.2%, 95% CI [11.3, 23.1], p < 0.001). A PNC-baby increase (+6.1%, 95% CI [−0.5, 12.8], p = 0.07) was not statistically significant. An estimated 121 neonatal and 20 maternal lives were saved between 2016 and 2019.

Conclusions

Full-district scale-up of a comprehensive MNH package embedded government health system was successfully implemented over a short time and associated with significant maternal care-seeking improvements and potential for lives saved.

The role of modeling in evaluation of maternal and child health programs: using the lives saved tool to help answer core evaluation questions

This paper explains how The Lives Saved Tool (LiST), a computer-based model that estimates the impact of scaling up interventions on stillbirths, maternal, neonatal and child health, can contribute to evaluations of programs being delivered at scale to improve maternal and child health. LiST can be used to estimate the impact of a program in advance, allowing planners to refine, streamline and set appropriate program targets. LiST can also be used to estimate the impact of a program, which is particularly useful given the high costs of measuring changes in population health. Finally, LiST can be used to estimate the relative contributions of different interventions or sets of interventions within programs that are found to have a positive impact. The latest version of LiST allows users to manipulate both utilization and quality of service to generate estimates of effective coverage. In addition, a new, web-based version of LiST is now available, with a simpler and more streamlined interface designed to increase accessibility to beginning users. LiST modeling can help program planners, evaluators and funders respond to core evaluation questions related to program design and impact, providing evidence to support decisions about how best to use available resources to save the lives of women and children.

The RADAR coverage tool: developing a toolkit for rigorous household surveys for reproductive, maternal, newborn, and child health & nutrition indicators

Population-based intervention coverage data are used to inform the design of projects, programs, and policies and to evaluate their impact. In low- and middle-income countries (LMICs), household surveys are the primary source of coverage data. Many coverage surveys are implemented by organizations with limited experience or resources in population-based data collection. We developed a streamlined survey and set of supporting materials to facilitate rigorous survey design and implementation. The RADAR coverage survey tool aimed to 1) rigorously measure priority reproductive, maternal, newborn, child health & nutrition coverage indicators, and allow for equity and gender analyses; 2) use standard, valid questions, to the extent possible; 3) be as light as possible; 4) be flexible to address users’ needs; and 5) be compatible with the Lives Saved Tool for analysis of program impact. Early interactions with stakeholders also highlighted survey planning, implementation, and analysis as challenging areas. We therefore developed a suite of resources to support implementers in these areas. The toolkit was piloted by implementers in Tanzania and in Burkina Faso. Although the toolkit was successfully implemented in these settings and facilitated survey planning and implementation, we found that implementers must still have access to sufficient resources, time, and technical expertise in order to use the tool appropriately. This potentially limits the use of the tool to situations where high-quality surveys or evaluations have been prioritized and adequately resourced.

Strengthening effectiveness evaluations through gender integration to improve programs for women, newborn, child, and adolescent health

Over the past decade, there has been substantial progress towards integrating our understanding of social determinants of reproductive, maternal, newborn, child, and adolescent health (RMNCAH) into health planning and programs. For these programs, gender inequity remains one of the most harmful factors for women’s access to healthcare. Designing RMNCAH programs to be gender-responsive through active engagement with and overcoming of gender-related barriers remains paramount to increasing women’s access to and use of health programs. However, the integration of gender within RMNCAH programs and their evaluation is often non-existent or is limited in scope. Building on a prior framework for comprehensive gender analysis in RMNCAH, we discuss key steps used to incorporate a gender lens and analytical approach in the Real Accountability: Data Analysis for Results (RADAR) evaluation framework. In order to effectively address these key areas, gender must be integrated into all stages of the evaluation, including tool development and programmatic activities that are associated with each question. Our paper includes practical activities and tools that evaluators may use to integrate gender into their evaluation tools.

Measuring Quality of Maternal, Neonatal, Child, Reproductive Health and Nutrition Care with tools developed by the RADAR project and tested in Sub Saharan Africa

Increasing coverage of evidence-based maternal, neonatal, child, reproductive health and nutrition (MNCRHN) programs in low- and middle-income countries has coincided with dramatic improvements in health despite variable quality of implementation. Comprehensive evaluation to inform program improvement requires standardized but adaptable tools, which the Real Accountability, Data Analysis for Results (RADAR) project has developed. To inform selection of tools and methods packages (‘packages’) to measure program quality of care (QoC), we documented experiences testing the packages, which were developed and adapted based on global and local expertise, and pre- and pilot-testing. We conducted cross-sectional studies in 2018–2019 on the quality of 1) integrated community case management, 2) counseling on maternal, infant, and young child feeding, 3) intrapartum care, and 4) family planning counseling in Mali, Mozambique, Tanzania, and Malawi. Herein we describe package performance and highlight experiences that inform their selection and use. Direct observation packages provided high-quality, immediately applicable results but they required specialized expertise, in-person collection, adequate patient volume, reasonable wait times, and unambiguously ‘correct’ provision of care. General satisfaction questions from exit interview packages produced unvaryingly positive responses despite variable observed quality of care. Variation increased when questions were more targeted, but findings on caregiver and client’s recall of recommendations were more actionable. When interactive, clinical vignettes can capture knowledge of clinical care. But for conditions that can be simulated, like provision of family planning counseling, we could capture provider practice from simulated clients. Clinicians could more easily demonstrate tactile aspects of intrapartum care using observed structured clinical examinations, but this method required storage and transport of the required mannequins. Based on our findings we recommend ten questions upon which evaluators can base package selection. Findings from these packages inform programs and, in the context of comprehensive program evaluation enable us to link programs with impact.