A common evaluation framework for the African Health Initiative

Health system description and assessment: a scoping review of templates for systematic analyses

BACKGROUND

Understanding and comparing health systems is key for cross-country learning and health system strengthening. Templates help to develop standardised and coherent descriptions and assessments of health systems, which then allow meaningful analyses and comparisons. Our scoping review aims to provide an overview of existing templates, their content and the way data is presented.

MAIN BODY

Based on the WHO building blocks framework, we defined templates as having (1) an overall framework, (2) a list of indicators or topics, and (3) instructions for authors, while covering (4) the design of the health system, (5) an assessment of health system performance, and (6) should cover the entire health system. We conducted a scoping review of grey literature published between 2000 and 2023 to identify templates. The content of the identified templates was screened, analyzed and compared. We found 12 documents that met our inclusion criteria. The building block `health financing´ is covered in all 12 templates; and many templates cover ´service delivery´ and ´health workforce'. Health system performance is frequently assessed with regard to 'access and coverage', 'quality and safety', and 'financial protection'. Most templates do not cover 'responsiveness' and 'efficiency'. Seven templates combine quantitative and qualitative data, three are mostly quantitative, and two are primarily qualitative. Templates cover data and information that is mostly relevant for specific groups of countries, e.g. a particular geographical region, or for high or for low and middle-income countries (LMICs). Templates for LMICs rely more on survey-based indicators than administrative data.

CONCLUSIONS

This is the first scoping review of templates for standardized descriptions of health systems and assessments of their performance. The implications are that (1) templates can help analyze health systems across countries while accounting for context; (2) template-guided analyses of health systems could underpin national health policies, strategies, and plans; (3) organizations developing templates could learn from approaches of other templates; and (4) more research is needed on how to improve templates to better achieve their goals. Our findings provide an overview and help identify the most important aspects and topics to look at when comparing and analyzing health systems, and how data are commonly presented. The templates were created by organizations with different agendas and target audiences, and with different end products in mind. Comprehensive health systems analyses and comparisons require production of quantitative indicators and complementing them with qualitative information to build a holistic picture.

CLINICAL TRIAL REGISTRATION

  Not applicable.

Quality improvement in public-private partnerships in low- and middle-income countries: a systematic review

BACKGROUND

Public-private partnerships (PPP) are often how health improvement programs are implemented in low-and-middle-income countries (LMICs). We therefore aimed to systematically review the literature about the aim and impacts of quality improvement (QI) approaches in PPP in LMICs.

METHODS

We searched SCOPUS and grey literature for studies published before March 2022. One reviewer screened abstracts and full-text studies for inclusion. The study characteristics, setting, design, outcomes, and lessons learned were abstracted using a standard tool and reviewed in detail by a second author.

RESULTS

We identified 9,457 citations, of which 144 met the inclusion criteria and underwent full-text abstraction. We identified five key themes for successful QI projects in LMICs: 1) leadership support and alignment with overarching priorities, 2) local ownership and engagement of frontline teams, 3) shared authentic learning across teams, 4) resilience in managing external challenges, and 5) robust data and data visualization to track progress. We found great heterogeneity in QI tools, study designs, participants, and outcome measures. Most studies had diffuse aims and poor descriptions of the intervention components and their follow-up. Few papers formally reported on actual deployment of private-sector capital, and either provided insufficient information or did not follow the formal PPP model, which involves capital investment for a explicit return on investment. Few studies discussed the response to their findings and the organizational willingness to change.

CONCLUSIONS

Many of the same factors that impact the success of QI in healthcare in high-income countries are relevant for PPP in LMICs. Vague descriptions of the structure and financial arrangements of the PPPs, and the roles of public and private entities made it difficult to draw meaningful conclusions about the impacts of the organizational governance on the outcomes of QI programs in LMICs. While we found many articles in the published literature on PPP-funded QI partnerships in LMICs, there is a dire need for research that more clearly describes the intervention details, implementation challenges, contextual factors, leadership and organizational structures. These details are needed to better align incentives to support the kinds of collaboration needed for guiding accountability in advancing global health. More ownership and power needs to be shifted to local leaders and researchers to improve research equity and sustainability.

Tracking progress towards equitable maternal and child health in Yunnan: a systematic assessment for the Health Programme for Poverty Alleviation in China during 2015-2020

OBJECTIVES

To inform the impacts of health programmes which aimed at preventing women and children from being trapped in or returning to poverty because of illness in Yunnan, the main battlefield against poverty in China.

DESIGN

The longitudinal comparative evaluation design.

DATA COLLECTION AND ANALYSIS

National and Yunnan policy documents related to maternal and child health programmes for poverty alleviation during 2015-2020 were analysed. The changes in disparities in maternal and child health system inputs, service coverage, and health outcomes between poor and non-poor areas, as well as out-of-pocket payments between poor and non-poor populations were assessed before and after 2017.

RESULTS

In total 12 policies and 15 programmes related to poverty alleviation for poor women and children in Yunnan were summarised. As a result of health system strengthening in Yunnan, the densities of licensed doctors, nurses, obstetricians, midwives, township health workers and female village doctors had been increased substantially in poor areas, with the annual rates of 14.3%, 22.5%, 21.8%, 23.9%, 14.1% and 7.1% separately. Although disparities existed in some of service coverage between poor and non-poor areas, the health programmes had narrowed the gaps in utilisation of facility birth, caesarean section, prenatal screening and newborn screening across Yunnan (p<0.01). The out-of-pocket payments for inpatient care for serious illnesses among women and children with poverty registration had been considerably decreased to 10.0%. Paralleling the universal coverage, maternal deaths per 100 000 livebirths and child deaths per 1000 livebirths had further declined in both poor and non-poor areas, and the impacts of health programmes on closing the gaps in child survivals across Yunnan were significant (p<0.01).

CONCLUSIONS

Remarkable progress in equitable maternal and child survival has been achieved in Yunnan. The practices in Yunnan have shown the Chinese model in ending poverty by strengthening health system and implementing universal coverage with firm commitment, determined leadership, detailed blueprint and social participation.

The African Health Initiative's Role in Advancing the Use of Embedded Implementation Research for Health Systems Strengthening

The African Health Initiative has demonstrated the feasibility of changing the traditional knowledge generation paradigm by using an embedded implementation research approach to improve health systems’ performance and strengthen capacity for knowledge generation and use.

Embedding Research on Implementation of Primary Health Care Systems Strengthening: A Commentary on Collaborative Experiences in Ethiopia, Ghana, and Mozambique

Achieving universal health care coverage requires the adoption of primary health care policies and delivery strategies that are evidence based. Although this has been confronted by manifold challenges, particularly in the health systems of sub-Saharan Africa, there are promising approaches for accomplishing this objective. Salient among these is embedding implementation research (i.e., the study of methods to promote the systematic uptake of evidence-based interventions (EBIs) into routine practice) into policy making and implementation processes. Since 2007, the African Health Initiative of the Doris Duke Charitable Foundation supported partnerships that strengthened primary health systems and policy implementation in 7 countries in sub-Saharan Africa using the embedded implementation research as a core strategy. This programmatic review and analysis aims to identify the core features and processes that characterized how the partnerships operationalized the embedded implementation research approach and understand the factors that helped and constrained partnerships' effective use of this approach. For this, we drew upon findings from a desk review that consisted of 30 examples of embedded implementation research conducted by 3 African Health Initiative partnerships between 2016 and 2021 in Ethiopia, Ghana, and Mozambique. In addition, we conducted and analyzed 13 in-depth interviews with embedded implementation research stakeholders of the 3 projects. Core features and processes of embedded implementation research were: (1) the leadership role of policy decision makers and implementation leaders; (2) positioning research with program implementation at multiple levels of health systems; (3) multidisciplinary and multisectoral partnerships; (4) focus on research capacity building; and (5) real-time feedback loops and knowledge translation. Factors influencing the effectiveness of the embedded implementation research experiences involved: (1) the implementation climate and leadership; (2) opportunities and capacities to circulate and absorb new information; and (3) stakeholders' baseline knowledge and embedded scientists' identification within their organizations.

Learning Health Systems to Bridge the Evidence-Policy-Practice Gap in Primary Health Care: Lessons From the African Health Initiative

The compilation of lessons in this supplement on the Doris Duke Charitable Foundation’s African Health Initiative’s work in the application of implementation research in primary health care in sub-Saharan Africa reflects the evolution of the discipline that is now increasingly recognized as integral to health systems strengthening.

Strengthening effectiveness evaluations to improve programs for women, children and adolescents

A full understanding of the pathways from efficacious interventions to population impact requires rigorous effectiveness evaluations conducted under realistic scale-up conditions at country level. In this paper, we introduce a deductive framework that underpins effectiveness evaluations. This framework forms the theoretical and conceptual basis for the 'Real Accountability: Data Analysis for Results' (RADAR) project, intended to address gaps in guidance and tools for the evaluation of projects being implemented at scale to reduce mortality among women and children. These gaps include needs for a framework to guide decisions about evaluations and practical measurement tools, as well as increased capacity in evaluation practice among donors and program planners at global, national and project levels. RADAR aimed to improve the evidence base for program and policy decisions in reproductive, maternal, newborn and child health and nutrition (RMNCH&N). We focus on five linked methodological steps - presented as core evaluation questions - for designing and implementing effectiveness evaluation of large-scale programs that support both the needs of program managers to improve their programs and the needs of donors to meet their accountability responsibilities. RADAR has operationalized each step with a tool to facilitate its application. We also describe cross-cutting methodological issues and broader contextual factors that affect the planning and implementation of such evaluations. We conclude with proposals for how the global RMNCH&N community can support rigorous program evaluations and make better use of the resulting evidence.

Japan's development cooperation for health in Vietnam: a first holistic assessment on Japan's ODA and non-ODA public resources cooperation

Background

Japan strives to strengthen its development cooperation by mobilizing various resources to assist partner countries advance on Universal Health Coverage by 2030. However, the involvement and roles of various actors for health are not clear. This study is the first to map Japan’s publicly funded projects by both Official Development Assistance (ODA) and other non-ODA public funds, and to describe the intervention areas. Further, the policy implications for country-specific cooperation strategies are discussed. The development cooperation for health in Vietnam is used as a case in this study.

Methods

A cross-sectional analysis of the Japanese publicly funded health projects that were being implemented in Vietnam during December 2016 was conducted. A framework of analysis based on the World Health Organization six health systems building blocks was adopted. The projects’ qualitative information was also assessed.

Results

Overall, 68 projects implemented through Japanese public funding were analyzed. These 68 projects under 15 types of schemes were managed by seven different scheme-operating organizations and funded by five ministries. Of these 44 (64.7%) were ODA and 24 (35.3%) were non-ODA projects. Among the recategorized six building blocks of the health system, the largest proportion of projects was health service delivery (44%), followed by health workforces (25%), and health information systems (15%). Almost half the projects were implemented together with the central hospitals as Vietnamese counterparts, which suggests that this is one area in which the specificities of Japanese cooperation are demonstrated. No synergetic effects of potential collaboration or harmonization among Japanese funded projects were captured.

Conclusions

Several Japanese-funded projects addressed a wide range of health issues across all six building blocks of the health system in Vietnam. However, there is room for improvement in developing coordination and harmonization among the diversified Japanese projects. Establishing a country-specific mechanism for strategic coordination across Japanese ministries’ schemes can yield efficient and effective development cooperation for health. While Vietnam’s dependence on external funding is low, the importance of coordination across domestic actors of the donor countries can serve as an important lesson, especially in beneficiary countries with high external funding dependency.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-021-12170-0.

A framework for identifying and learning from countries that demonstrated exemplary performance in improving health outcomes and systems

This paper introduces a framework for conducting and disseminating mixed methods research on positive outlier countries that successfully improved their health outcomes and systems. We provide guidance on identifying exemplar countries, assembling multidisciplinary teams, collecting and synthesising pre-existing evidence, undertaking qualitative and quantitative analyses, and preparing dissemination products for various target audiences. Through a range of ongoing research studies, we illustrate application of each step of the framework while highlighting key considerations and lessons learnt. We hope uptake of this comprehensive framework by diverse stakeholders will increase the availability and utilisation of rigorous and comparable insights from global health success stories.

Improving National Intelligence for Public Health Preparedness: a methodological approach to finding local multi-sector indicators for health security

The COVID-19 epidemic is the latest evidence of critical gaps in our collective ability to monitor country-level preparedness for health emergencies. The global frameworks that exist to strengthen core public health capacities lack coverage of several preparedness domains and do not provide mechanisms to interface with local intelligence. We designed and piloted a process, in collaboration with three National Public Health Institutes (NPHIs) in Ethiopia, Nigeria and Pakistan, to identify potential preparedness indicators that exist in a myriad of frameworks and tools in varying local institutions. Following a desk-based systematic search and expert consultations, indicators were extracted from existing national and subnational health security-relevant frameworks and prioritised in a multi-stakeholder two-round Delphi process. Eighty-six indicators in Ethiopia, 87 indicators in Nigeria and 51 indicators in Pakistan were assessed to be valid, relevant and feasible. From these, 14–16 indicators were prioritised in each of the three countries for consideration in monitoring and evaluation tools. Priority indicators consistently included private sector metrics, subnational capacities, availability and capacity for electronic surveillance, measures of timeliness for routine reporting, data quality scores and data related to internally displaced persons and returnees. NPHIs play an increasingly central role in health security and must have access to data needed to identify and respond rapidly to public health threats. Collecting and collating local sources of information may prove essential to addressing gaps; it is a necessary step towards improving preparedness and strengthening international health regulations compliance.

Measuring child survival for the Millennium Development Goals in Africa: what have we learned and what more is needed to evaluate the Sustainable Development Goals?

Reducing child mortality is a key global health challenge. We examined reasons for greater or lesser success in meeting under-five mortality rate reductions, i.e. Millennium Development Goal #4, between 1990 and 2015 in Sub-Saharan Africa where child mortality remains high. We first examined factors associated with child mortality from all World Health Organization African Region nations during the Millennium Development Goal period. This analysis was followed by case studies of the facilitators and barriers to Millennium Development Goal #4 in four countries – Kenya, Liberia, Zambia, and Zimbabwe. Quantitative indicators, policy documents, and qualitative interviews and focus groups were collected from each country to examine factors within and across countries related to child mortality. We found familiar themes that highlighted the need for both specific services (e.g. primary care access, emergency obstetric and neonatal care) and general management (e.g. strong health governance and leadership, increasing community health workers, quality of care). We also identified methodological opportunities and challenges to assessing progress in child health, which can provide insights to similar efforts during the Sustainable Development Goal period. Specifically, it is important for countries to adapt general international goals and measurements to their national context, considering baseline mortality rates and health information systems, to develop country-specific goals. It will also be critical to develop more rigorous measurement tools and indicators to accurately characterize maternal, neonatal, and child health systems, particularly in the area of governance and leadership. Valuable lessons can be learned from Millennium Development Goal successes and failures, as well as how they are evaluated. As countries seek to lower child mortality further during the Sustainable Development Goal period, it will be necessary to prioritize and support countries in quantitative and qualitative data collection to assess and contextualize progress, identifying areas needing improvement.

Operationalising health systems thinking: a pathway to high effective coverage

Background

The global health community has recognised the importance of defining and measuring the effective coverage of health interventions and their implementation strength to monitor progress towards global mortality and morbidity targets. Existing health system models and frameworks guide thinking around these measurement areas; however, they fall short of adequately capturing the dynamic and multi-level relationships between different components of the health system. These relationships must be articulated for measurement and managed to effectively deliver health interventions of sufficient quality to achieve health impacts. Save the Children’s Saving Newborn Lives programme and EnCompass LLC, its evaluation partner, developed and applied the Pathway to High Effective Coverage as a health systems thinking framework (hereafter referred to as the Pathway) in its strategic planning, monitoring and evaluation.

Methods

We used an iterative approach to develop, test and refine thinking around the Pathway. The initial framework was developed based on existing literature, then shared and vetted during consultations with global health thought leaders in maternal and newborn health.

Results

The Pathway is a robust health systems thinking framework that unpacks system, policy and point of intervention delivery factors, thus encouraging specific actions to address gaps in implementation and facilitate the achievement of high effective coverage. The Pathway includes six main components – (1) national readiness; (2) system structures; (3) management capacity; (4) implementation strength; (5) effective coverage; and (6) impact. Each component is comprised of specific elements reflecting the range of facility-, community- and home-based interventions. We describe applications of the Pathway and results for in-country strategic planning, monitoring of progress and implementation strength, and evaluation.

Conclusions

The Pathway provides a cohesive health systems thinking framework that facilitates assessment and coordinated action to achieve high coverage and impact. Experiences of its application show its utility in guiding strategic planning and in more comprehensive and effective monitoring and evaluation as well as its potential adaptability for use in other health areas and sectors.

Challenges and facilitators to evidence-based decision-making for maternal and child health in Mozambique: district, municipal and national case studies

BACKGROUND

The need for evidence-based decision-making in the health sector is well understood in the global health community. Yet, gaps persist between the availability of evidence and the use of that evidence. Most research on evidence-based decision-making has been carried out in higher-income countries, and most studies look at policy-making rather than decision-making more broadly. We conducted this study to address these gaps and to identify challenges and facilitators to evidence-based decision-making in Maternal, Newborn and Child Health and Nutrition (MNCH&N) at the municipality, district, and national levels in Mozambique.

METHODS

We used a case study design to capture the experiences of decision-makers and analysts (n = 24) who participated in evidence-based decision-making processes related to health policies and interventions to improve MNCH&N in diverse decision-making contexts (district, municipality, and national levels) in 2014-2017, in Mozambique. We examined six case studies, at the national level, in Maputo City and in two districts of Sofala Province and two of Zambézia Province, using individual in-depth interviews with key informants and a document review, for three weeks, in July 2018.

RESULTS

Our analysis highlighted various challenges for evidence-based decision-making for MNCH&N, at national, district, and municipality levels in Mozambique, including limited demand for evidence, limited capacity to use evidence, and lack of trust in the available evidence. By contrast, access to evidence, and availability of evidence were viewed positively and seen as potential facilitators. Organizational capacity for the demand and use of evidence appears to be the greatest challenge; while individual capacity is also a barrier.

CONCLUSION

Evidence-based decision-making requires that actors have access to evidence and are empowered to act on that evidence. This, in turn, requires alignment between those who collect data, those who analyze and interpret data, and those who make and implement decisions. Investments in individual, organizational, and systems capacity to use evidence are needed to foster practices of evidence-based decision-making for improved maternal and child health in Mozambique.

Advancing research on emergency care systems in low-income and middle-income countries: ensuring high-quality care delivery systems

Emergency care systems (ECS) address a wide range of acute conditions, including emergent conditions from communicable diseases, non-communicable diseases, pregnancy and injury. Together, ECS represent an area of great potential for reducing morbidity and mortality in low-income and middle-income countries (LMICs). It is estimated that up to 54% of annual deaths in LMICs could be addressed by improved prehospital and facility-based emergency care. Research is needed to identify strategies for enhancing ECS to optimise prevention and treatment of conditions presenting in this context, yet significant gaps persist in defining critical research questions for ECS studies in LMICs. The Collaborative on Enhancing Emergency Care Research in LMICs seeks to promote research that improves immediate and long-term outcomes for clients and populations with emergent conditions. The objective of this paper is to describe systems approaches and research strategies for ECS in LMICs, elucidate priority research questions and methodology, and present a selection of studies addressing the operational, implementation, policy and health systems domains of health systems research as an approach to studying ECS. Finally, we briefly discuss limitations and the next steps in developing ECS-oriented interventions and research.

[Construction of a monitoring framework for universal healthConstrução de um quadro de monitoramento para saúde universal]

El objetivo del trabajo es construir un marco conceptual de monitoreo sobre los avances de políticas y acciones orientadas a alcanzar la salud universal. Se revisaron modelos conceptuales y propuestas metodológicas relacionados con el monitoreo del acceso y la cobertura universal de salud. Se realizó también una revisión de la literatura para seleccionar indicadores relevantes. Esta revisión fue complementada con un proceso de consulta con expertos en sistemas de salud de la Región de las Américas. Se desarrolló un marco integral para el monitoreo de políticas y acciones para el acceso y la cobertura universal de salud. El marco de monitoreo contiene cuatro componentes (acciones estratégicas, resultados inmediatos, resultados intermedios y resultados de impacto) e identifica un conjunto de opciones políticas para guiar la transformación de los sistemas de salud hacia el acceso y la cobertura universal de salud. Se eligieron 64 indicadores entre un total de 500 indicadores para la evaluación de los componentes del marco de monitoreo. El abordaje propuesto para la utilización del marco se basa en la medición de inequidades en las condiciones de acceso y cobertura, así como en la recolección de evidencia cualitativa sobre el grado de ejecución de políticas y acciones. El marco propuesto podría contribuir a fortalecer los procesos de transformación de los sistemas de salud para avanzar hacia el acceso y la cobertura universal de salud.

Progress and challenges in improving maternal health in the Tibet Autonomous Region, China

Background

The maternal mortality ratio (MMR) of the Tibet Autonomous Region (hereinafter “Tibet”) is still five times higher than the national average. This study aims to identify the successes and pitfalls of the health system that might be related to the high mortality rate based on the WHO health system building blocks, focusing on human resources for health and health infrastructure and the impact on maternal health and outcomes.

Methods

Sources of information include China’s central government and Tibet’s local government policies and regulations, health statistical yearbooks, maternal and child health routine reporting system, and English and Chinese online research articles. Joinpoint analysis was applied for MMR and maternal health service trends, and correlation test was used to test the relationship between maternal health services and outcomes.

Results

Between 2000 and 2015, public health spending in Tibet increased 67-fold, the hospital delivery rate increased 70.1%, and the MMR dropped from 466.9 to 100.1 per 100,000 live births. However, the total number of health workers, qualified medical doctors, and registered nurses per 1,000 people were 4.4, 1.4, and 1.0, respectively, much lower than the national average (5.8, 1.8, and 2.4). In Tibet, there were 80 basic and 16 comprehensive emergency obstetric care (EOC) centers. On average, there were 12 basic and 2.5 comprehensive EOC centers per 500,000 of the population. Though it met the WHO’s recommendation, it might remain inadequate in the low population density of the area like Tibet.

Conclusion

The shortage of health professionals and EOC centers and health information in predominantly remote rural areas with a scattered population still needs to be rectified.

Baseline population health conditions ahead of a health system strengthening program in rural Madagascar

Background: A model health district was initiated through a program of health system strengthening (HSS) in Ifanadiana District of southeastern Madagascar in 2014. We report population health indicators prior to initiation of the program.

Methods: A representative household survey based on the Demographic Health Survey was conducted using a two-stage cluster sampling design in two strata – the initial program catchment area and the future catchment area. Chi-squared and t-tests were used to compare data by stratum, using appropriate sampling weights. Madagascar data for comparison were taken from a 2013 national study.

Results: 1522 households were surveyed, representing 8310 individuals including 1635 women ages 15–49, 1685 men ages 15–59 and 1251 children under age 5. Maternal mortality rates in the district are 1044/100,000. 81% of women’s last childbirth deliveries were in the home; only 20% of deliveries were attended by a doctor or nurse/midwife (not different by stratum). 9.3% of women had their first birth by age 15, and 29.5% by age 18. Under-5 mortality rate is high: 145/1000 live births vs. 62/1000 nationally. 34.6% of children received all recommended vaccines by age 12 months (compared to 51.5% in Madagascar overall). In the 2 weeks prior to interview, approximately 28% of children under age 5 had acute respiratory infections of whom 34.7% were taken for care, and 14% of children had diarrhea of whom 56.6% were taken for care. Under-5 mortality, illness, care-seeking and vaccination rates were not significantly different between strata.

Conclusions: Indicators of population health and health care-seeking reveal low use of the formal health system, which could benefit from HSS. Data from this survey and from a longitudinal follow-up study will be used to target needed interventions, to assess change in the district and the impact of HSS on individual households and the population of the district.

Early changes in intervention coverage and mortality rates following the implementation of an integrated health system intervention in Madagascar

INTRODUCTION

The Sustainable Development Goals framed an unprecedented commitment to achieve global convergence in child and maternal mortality rates through 2030. To meet those targets, essential health services must be scaled via integration with strengthened health systems. This is especially urgent in Madagascar, the country with the lowest level of financing for health in the world. Here, we present an interim evaluation of the first 2 years of a district-level health system strengthening (HSS) initiative in rural Madagascar, using estimates of intervention coverage and mortality rates from a district-wide longitudinal cohort.

METHODS

We carried out a district representative household survey at baseline of the HSS intervention in over 1500 households in Ifanadiana district. The first follow-up was after the first 2 years of the initiative. For each survey, we estimated maternal, newborn and child health (MNCH) coverage, healthcare inequalities and child mortality rates both in the initial intervention catchment area and in the rest of the district. We evaluated changes between the two areas through difference-in-differences analyses. We estimated annual changes in health centre per capita utilisation from 2013 to 2016.

RESULTS

The intervention was associated with 19.1% and 36.4% decreases in under-five and neonatal mortality, respectively, although these were not statistically significant. The composite coverage index (a summary measure of MNCH coverage) increased by 30.1%, with a notable 63% increase in deliveries in health facilities. Improvements in coverage were substantially larger in the HSS catchment area and led to an overall reduction in healthcare inequalities. Health centre utilisation rates in the catchment tripled for most types of care during the study period.

CONCLUSION

At the earliest stages of an HSS intervention, the rapid improvements observed for Ifanadiana add to preliminary evidence supporting the untapped and poorly understood potential of integrated HSS interventions on population health.

Data-informed decision-making for life-saving commodities investments in Malawi: A qualitative case study

Background

During the last 15 years, Malawi has made remarkable progress in reducing child mortality. However, maternal and newborn mortality remains persistently high. To help address these entrenched challenges, the Reproductive, Maternal, Newborn and Child Health (RMNCH) Trust Fund provided short-term catalytic financing of $11.5 million (2013-2016) to support country plans to advance the RMNCH and commodity agenda.

Objectives

(1) To document how Malawi (ministries, partners, working groups) used evidence to inform decision-making and RMNCH investments, (2) To identify barriers to utilizing information and evidence in the planning and prioritization process at national and sub-national levels, and (3) To assess the utility of the RMNCH Landscape Synthesis, which uses existing information to review life-saving RMNCH commodities and services.

Methods

This was a qualitative case study utilizing a Rapid Appraisal (RA) approach, where semi-structured interviews were conducted with staff members from UN agencies, development partners and the Ministry of Health (MoH) at national and district level. The analysis enlists a framework approach for manual qualitative content analysis.

Results

Led by the MoH, the RMNCH Trust Fund grant proposal utilized an evidence-based and equity-focused process for prioritization of investments. Data-informed decision-making permeates similar commodity-focused working groups. However, common health information system (HIS) weaknesses, such as data quality and collection burden, persist and are more prevalent at district-level. The collation of evidence in the RMNCH Landscape Synthesis was a useful and sustainable tool to support planning.

Conclusions

The evidence-based, equity-focused decision-making process for the RMNCH Trust Fund proposal provides an effective model for inter-agency investment prioritization. Strengthening data-informed decision-making will require financial and political commitments to HIS and capacity building for data use, particularly at the district-level. New initiatives (e.g. Health Data Collaborative and QED Network to Improve Quality of Care) provide opportunities to further improve evidence-informed decision-making.

Impact of a health system strengthening intervention on maternal and child health outputs and outcomes in rural Rwanda 2005-2010

Introduction

Although Rwanda’s health system underwent major reforms and improvements after the 1994 Genocide, the health system and population health in the southeast lagged behind other areas. In 2005, Partners In Health and the Rwandan Ministry of Health began a health system strengthening intervention in this region. We evaluate potential impacts of the intervention on maternal and child health indicators.

Methods

Combining results from the 2005 and 2010 Demographic and Health Surveys with those from a supplemental 2010 survey, we compared changes in health system output indicators and population health outcomes between 2005 and 2010 as reported by women living in the intervention area with those reported by the pooled population of women from all other rural areas of the country, controlling for potential confounding by economic and demographic variables.

Results

Overall health system coverage improved similarly in the comparison groups between 2005 and 2010, with an indicator of composite coverage of child health interventions increasing from 57.9% to 75.0% in the intervention area and from 58.7% to 73.8% in the other rural areas. Under-five mortality declined by an annual rate of 12.8% in the intervention area, from 229.8 to 83.2 deaths per 1000 live births, and by 8.9% in other rural areas, from 157.7 to 75.8 deaths per 1000 live births. Improvements were most marked among the poorest households.

Conclusion

We observed dramatic improvements in population health outcomes including under-five mortality between 2005 and 2010 in rural Rwanda generally and in the intervention area specifically.

Contextual factors in maternal and newborn health evaluation: a protocol applied in Nigeria, India and Ethiopia

Background

Understanding the context of a health programme is important in interpreting evaluation findings and in considering the external validity for other settings. Public health researchers can be imprecise and inconsistent in their usage of the word “context” and its application to their work. This paper presents an approach to defining context, to capturing relevant contextual information and to using such information to help interpret findings from the perspective of a research group evaluating the effect of diverse innovations on coverage of evidence-based, life-saving interventions for maternal and newborn health in Ethiopia, Nigeria, and India.

Methods

We define “context” as the background environment or setting of any program, and “contextual factors” as those elements of context that could affect implementation of a programme. Through a structured, consultative process, contextual factors were identified while trying to strike a balance between comprehensiveness and feasibility. Thematic areas included demographics and socio-economics, epidemiological profile, health systems and service uptake, infrastructure, education, environment, politics, policy and governance. We outline an approach for capturing and using contextual factors while maximizing use of existing data. Methods include desk reviews, secondary data extraction and key informant interviews. Outputs include databases of contextual factors and summaries of existing maternal and newborn health policies and their implementation. Use of contextual data will be qualitative in nature and may assist in interpreting findings in both quantitative and qualitative aspects of programme evaluation.

Discussion

Applying this approach was more resource intensive than expected, in part because routinely available information was not consistently available across settings and more primary data collection was required than anticipated. Data was used only minimally, partly due to a lack of evaluation results that needed further explanation, but also because contextual data was not available for the precise units of analysis or time periods of interest. We would advise others to consider integrating contextual factors within other data collection activities, and to conduct regular reviews of maternal and newborn health policies. This approach and the learnings from its application could help inform the development of guidelines for the collection and use of contextual factors in public health evaluation.

Electronic supplementary material

The online version of this article (10.1186/s12982-018-0071-0) contains supplementary material, which is available to authorized users.

Research capacity building integrated into PHIT projects: leveraging research and research funding to build national capacity

Background

Inadequate research capacity impedes the development of evidence-based health programming in sub-Saharan Africa. However, funding for research capacity building (RCB) is often insufficient and restricted, limiting institutions’ ability to address current RCB needs. The Doris Duke Charitable Foundation’s African Health Initiative (AHI) funded Population Health Implementation and Training (PHIT) partnership projects in five African countries (Ghana, Mozambique, Rwanda, Tanzania and Zambia) to implement health systems strengthening initiatives inclusive of RCB.

Methods

Using Cooke’s framework for RCB, RCB activity leaders from each country reported on RCB priorities, activities, program metrics, ongoing challenges and solutions. These were synthesized by the authorship team, identifying common challenges and lessons learned.

Results

For most countries, each of the RCB domains from Cooke’s framework was a high priority. In about half of the countries, domain specific activities happened prior to PHIT. During PHIT, specific RCB activities varied across countries. However, all five countries used AHI funding to improve research administrative support and infrastructure, implement research trainings and support mentorship activities and research dissemination. While outcomes data were not systematically collected, countries reported holding 54 research trainings, forming 56 mentor-mentee relationships, training 201 individuals and awarding 22 PhD and Masters-level scholarships. Over the 5 years, 116 manuscripts were developed. Of the 59 manuscripts published in peer-reviewed journals, 29 had national first authors and 18 had national senior authors. Trainees participated in 99 conferences and projects held 37 forums with policy makers to facilitate research translation into policy.

Conclusion

All five PHIT projects strongly reported an increase in RCB activities and commended the Doris Duke Charitable Foundation for prioritizing RCB, funding RCB at adequate levels and time frames and for allowing flexibility in funding so that each project could implement activities according to their trainees’ needs. As a result, many common challenges for RCB, such as adequate resources and local and international institutional support, were not identified as major challenges for these projects. Overall recommendations are for funders to provide adequate and flexible funding for RCB activities and for institutions to offer a spectrum of RCB activities to enable continued growth, provide adequate mentorship for trainees and systematically monitor RCB activities.

Electronic supplementary material

The online version of this article (10.1186/s12913-017-2657-6) contains supplementary material, which is available to authorized users.

Tackling the hard problems: implementation experience and lessons learned in newborn health from the African Health Initiative

Background

The Doris Duke Charitable Foundation’s African Health Initiative supported the implementation of Population Health Implementation and Training (PHIT) Partnership health system strengthening interventions in designated areas of five countries: Ghana, Mozambique, Rwanda, Tanzania, and Zambia. All PHIT programs included health system strengthening interventions with child health outcomes from the outset, but all increasingly recognized the need to increase focus to improve health and outcomes in the first month of life. This paper uses a case study approach to describe interventions implemented in newborn health, compare approaches, and identify lessons learned across the programs’ collective implementation experience.

Methods

Case studies were built using quantitative and qualitative methods, applying the World Health Organization Health Systems Strengthening Framework, and maternal, newborn and child health continuum of care framework. We identified the following five primary themes in health systems strengthening intervention strategies used to target improvement in newborn health, which were incorporated by all PHIT projects with varying results: health service delivery at the community level (Tanzania), combining community and health facility level interventions (Zambia), participatory information feedback and clinical training (Ghana), performance review and enhancement (Mozambique), and integrated clinical and system-level improvement (Rwanda), and used individual case studies to illustrate each of these themes.

Results

Tanzania and Zambia included significant community-based components, including mobilization and sensitization for increased uptake of essential services, while Ghana, Mozambique, and Rwanda focused more efforts on improving the quality of services delivered once a patient enters a health facility. All countries included aspects that improved communication across levels of the health system, whether through district-wide data sharing and peer learning networks in Mozambique and Rwanda, or improved referral processes and systems in Tanzania, Zambia, and Ghana.

Conclusion

Key lessons learned include the importance of focusing intervention components on addressing drivers of neonatal mortality across the maternal and newborn care continuum at all levels of the health system, matching efforts to improve service utilization with provision of high quality facility-based services, and the critical role of leadership to catalyze improvements in newborn health.

Health system strengthening: a qualitative evaluation of implementation experience and lessons learned across five African countries

BACKGROUND

Achieving the United Nations Sustainable Development Goals in sub-Saharan Africa will require substantial improvements in the coverage and performance of primary health care delivery systems. Projects supported by the Doris Duke Charitable Foundation's (DDCF) African Health Initiative (AHI) created public-private-academic and community partnerships in five African countries to implement and evaluate district-level health system strengthening interventions. In this study, we captured common implementation experiences and lessons learned to understand core elements of successful health systems interventions.

METHODS

We used qualitative data from key informant interviews and annual progress reports from the five Population Health Implementation and Training (PHIT) partnership projects funded through AHI in Ghana, Mozambique, Rwanda, Tanzania, and Zambia.

RESULTS

Four major overarching lessons were highlighted. First, variety and inclusiveness of concerned key players (public, academic and private) are necessary to address complex health system issues at all levels. Second, a learning culture that promotes evidence creation and ability to efficiently adapt were key in order to meet changing contextual needs. Third, inclusion of strong implementation science tools and strategies allowed informed and measured learning processes and efficient dissemination of best practices. Fourth, five to seven years was the minimum time frame necessary to effectively implement complex health system strengthening interventions and generate the evidence base needed to advocate for sustainable change for the PHIT partnership projects.

CONCLUSION

The AHI experience has raised remaining, if not overlooked, challenges and potential solutions to address complex health systems strengthening intervention designs and implementation issues, while aiming to measurably accomplish sustainable positive change in dynamic, learning, and varied contexts.

Measuring health systems strength and its impact: experiences from the African Health Initiative

Background

Health systems are essential platforms for accessible, quality health services, and population health improvements. Global health initiatives have dramatically increased health resources; however, funding to strengthen health systems has not increased commensurately, partially due to concerns about health system complexity and evidence gaps demonstrating health outcome improvements. In 2009, the African Health Initiative of the Doris Duke Charitable Foundation began supporting Population Health Implementation and Training Partnership projects in five sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) to catalyze significant advances in strengthening health systems. This manuscript reflects on the experience of establishing an evaluation framework to measure health systems strength, and associate measures with health outcomes, as part of this Initiative.

Methods

Using the World Health Organization’s health systems building block framework, the Partnerships present novel approaches to measure health systems building blocks and summarize data across and within building blocks to facilitate analytic procedures. Three Partnerships developed summary measures spanning the building blocks using principal component analysis (Ghana and Tanzania) or the balanced scorecard (Zambia). Other Partnerships developed summary measures to simplify multiple indicators within individual building blocks, including health information systems (Mozambique), and service delivery (Rwanda). At the end of the project intervention period, one to two key informants from each Partnership’s leadership team were asked to list – in rank order – the importance of the six building blocks in relation to their intervention.

Results

Though there were differences across Partnerships, service delivery and information systems were reported to be the most common focus of interventions, followed by health workforce and leadership and governance. Medical products, vaccines and technologies, and health financing, were the building blocks reported to be of lower focus.

Conclusion

The African Health Initiative experience furthers the science of evaluation for health systems strengthening, highlighting areas for further methodological development – including the development of valid, feasible measures sensitive to interventions in multiple contexts (particularly in leadership and governance) and describing interactions across building blocks; in developing summary statistics to facilitate testing intervention effects on health systems and associations with health status; and designing appropriate analytic models for complex, multi-level open health systems.

Mentorship and coaching to support strengthening healthcare systems: lessons learned across the five Population Health Implementation and Training partnership projects in sub-Saharan Africa

Background

Despite global efforts to increase health workforce capacity through training and guidelines, challenges remain in bridging the gap between knowledge and quality clinical practice and addressing health system deficiencies preventing health workers from providing high quality care. In many developing countries, supervision activities focus on data collection, auditing and report completion rather than catalyzing learning and supporting system quality improvement. To address this gap, mentorship and coaching interventions were implemented in projects in five African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) as components of health systems strengthening (HSS) strategies funded through the Doris Duke Charitable Foundation’s African Health Initiative. We report on lessons learned from a cross-country evaluation.

Methods

The evaluation was designed based on a conceptual model derived from the project-specific interventions. Semi-structured interviews were administered to key informants to capture data in six categories: 1) mentorship and coaching goals, 2) selection and training of mentors and coaches, 3) integration with the existing systems, 4) monitoring and evaluation, 5) reported outcomes, and 6) challenges and successes. A review of project-published articles and technical reports from the individual projects supplemented interview information.

Results

Although there was heterogeneity in the approaches to mentorship and coaching and targeted areas of the country projects, all led to improvements in core health system areas, including quality of clinical care, data-driven decision making, leadership and accountability, and staff satisfaction. Adaptation of approaches to reflect local context encouraged their adoption and improved their effectiveness and sustainability.

Conclusion

We found that incorporating mentorship and coaching activities into HSS strategies was associated with improvements in quality of care and health systems, and mentorship and coaching represents an important component of HSS activities designed to improve not just coverage, but even further effective coverage, in achieving Universal Health Care.

Progress and challenges in maternal health in western China: a Countdown to 2015 national case study

BACKGROUND

China is one of the few Countdown countries to have achieved Millennium Development Goal 5 (75% reduction in maternal mortality ratio between 1990 and 2015). We aimed to examine the health systems and contextual factors that might have contributed to the substantial decline in maternal mortality between 1997 and 2014. We chose to focus on western China because poverty, ethnic diversity, and geographical access represent particular challenges to ensuring universal access to maternal care in the region.

METHODS

In this systematic assessment, we used data from national census reports, National Statistical Yearbooks, the National Maternal and Child Health Routine Reporting System, the China National Health Accounts report, and National Health Statistical Yearbooks to describe changes in policies, health financing, health workforce, health infrastructure, coverage of maternal care, and maternal mortality by region between 1997 and 2014. We used a multivariate linear regression model to examine which contextual and health systems factors contributed to the regional variation in maternal mortality ratio in the same period. Using data from a cross-sectional survey in 2011, we also examined equity in access to maternity care in 42 poor counties in western China.

FINDINGS

Maternal mortality declined by 8·9% per year between 1997 and 2014 (geometric mean ratio for each year 0·91, 95% CI 0·91-0·92). After adjusting for GDP per capita, length of highways, female illiteracy, the number of licensed doctors per 1000 population, and the proportion of ethnic minorities, the maternal mortality ratio was 118% higher in the western region (2·18, 1·44-3·28) and 41% higher in the central region (1·41, 0·99-2·01) than in the eastern region. In the rural western region, the proportion of births in health facilities rose from 41·9% in 1997 to 98·4% in 2014. Underpinning such progress was the Government's strong commitment to long-term strategies to ensure access to delivery care in health facilities-eg, professionalisation of maternity care in large hospitals, effective referral systems for women medically or socially at high risk, and financial subsidies for antenatal and delivery care. However, in the poor western counties, substantial disparity by education level of the mother existed in access to health facility births (44% of illiterate women vs 100% of those with college or higher education), antenatal care (17% vs 69%) had at least four visits), and caesarean section (8% vs 44%).

INTERPRETATION

Despite remarkable progress in maternal survival in China, substantial disparities remain, especially for the poor, less educated, and ethnic minority groups in remote areas in western China. Whether China's highly medicalised model of maternity care will be an answer for these populations is uncertain. A strategy modelled after China's immunisation programme, whereby care is provided close to the women's homes, might need to be explored, with township hospitals taking a more prominent role.

FUNDING

Government of Canada, UNICEF, and the Bill & Melinda Gates Foundation.

Subnational variation for care at birth in Tanzania: is this explained by place, people, money or drugs?

BACKGROUND

Tanzania achieved the Millennium Development Goal for child survival, yet made insufficient progress for maternal and neonatal survival and stillbirths, due to low coverage and quality of services for care at birth, with rural women left behind. Our study aimed to evaluate Tanzania's subnational (regional-level) variations for rural care at birth outcomes, i.e., rural women giving birth in a facility and by Caesarean section (C-section), and associations with health systems inputs (financing, health workforce, facilities, and commodities), outputs (readiness and quality of care) and context (education and GDP).

METHODS

We undertook correlation analyses of subnational-level associations between health system inputs, outputs, context, and rural care at birth outcomes; and constructed implementation readiness barometers using benchmarks for each health system input indicator. We used geographical information system (GIS) mapping to visualise subnational variations in care at birth for rural women, with a focus on service availability and readiness, and collected qualitative data to investigate financial flows from national to council level to understand variation in financing inputs.

RESULTS

We found wide subnational variation for rural care at birth outcomes, health systems inputs, and contextual indicators. There was a positive association between rural women giving birth in a facility and by C-section; maternal education; workforce and facility density; and quality of care. There was a negative association between these outcomes and proportion of all births to rural women, total fertility rate, and availability of essential commodities at facilities. Per capita recurrent expenditure was positively associated with facility births (correlation coefficient = 0.43; p = 0.05) but not with C-section. Qualitative results showed that the health financing system is complex and insufficient for providing care at birth services. Bottlenecks for care at birth included low density of health workers, poor availability of essential commodities, and low health financing in Lake and Western Zones.

CONCLUSIONS

No region meets the benchmarks for the four health systems building blocks including health finance, health workforce, health facilities, and commodities. Strategies for addressing health system inequities, including overall increases in health expenditure, are needed in rural populations and areas of highest unmet need for family planning to improve coverage of care at birth for rural women in Tanzania.

Tanzania's countdown to 2015: an analysis of two decades of progress and gaps for reproductive, maternal, newborn, and child health, to inform priorities for post-2015

BACKGROUND

Tanzania is on track to meet Millennium Development Goal (MDG) 4 for child survival, but is making insufficient progress for newborn survival and maternal health (MDG 5) and family planning. To understand this mixed progress and to identify priorities for the post-2015 era, Tanzania was selected as a Countdown to 2015 case study.

METHODS

We analysed progress made in Tanzania between 1990 and 2014 in maternal, newborn, and child mortality, and unmet need for family planning, in which we used a health systems evaluation framework to assess coverage and equity of interventions along the continuum of care, health systems, policies and investments, while also considering contextual change (eg, economic and educational). We had five objectives, which assessed each level of the health systems evaluation framework. We used the Lives Saved Tool (LiST) and did multiple linear regression analyses to explain the reduction in child mortality in Tanzania. We analysed the reasons for the slower changes in maternal and newborn survival and family planning, to inform priorities to end preventable maternal, newborn, and child deaths by 2030.

FINDINGS

In the past two decades, Tanzania's population has doubled in size, necessitating a doubling of health and social services to maintain coverage. Total health-care financing also doubled, with donor funding for child health and HIV/AIDS more than tripling. Trends along the continuum of care varied, with preventive child health services reaching high coverage (≥85%) and equity (socioeconomic status difference 13-14%), but lower coverage and wider inequities for child curative services (71% coverage, socioeconomic status difference 36%), facility delivery (52% coverage, socioeconomic status difference 56%), and family planning (46% coverage, socioeconomic status difference 22%). The LiST analysis suggested that around 39% of child mortality reduction was linked to increases in coverage of interventions, especially of immunisation and insecticide-treated bednets. Economic growth was also associated with reductions in child mortality. Child health programmes focused on selected high-impact interventions at lower levels of the health system (eg, the community and dispensary levels). Despite its high priority, implementation of maternal health care has been intermittent. Newborn survival has gained attention only since 2005, but high-impact interventions are already being implemented. Family planning had consistent policies but only recent reinvestment in implementation.

INTERPRETATION

Mixed progress in reproductive, maternal, newborn, and child health in Tanzania indicates a complex interplay of political prioritisation, health financing, and consistent implementation. Post-2015 priorities for Tanzania should focus on the unmet need for family planning, especially in the Western and Lake regions; addressing gaps for coverage and quality of care at birth, especially in rural areas; and continuation of progress for child health.

FUNDING

Government of Canada, Foreign Affairs, Trade, and Development; US Fund for UNICEF; and the Bill & Melinda Gates Foundation.

Measuring implementation strength: lessons from the evaluation of public health strategies in low- and middle-income settings

Evaluation of strategies to ensure evidence-based, low-cost interventions reach those in need is critical. One approach is to measure the strength, or intensity, with which packages of interventions are delivered, in order to explore the association between implementation strength and public health gains. A recent systematic review suggested methodological guidance was needed. We described the approaches used in three examples of measures of implementation strength in evaluation. These addressed important public health topics with a substantial disease burden in low-and middle-income countries; they involved large-scale implementation; and featured evaluation designs without comparison areas. Strengths and weaknesses of the approaches were discussed. In the evaluation of Ethiopia’s Health Extension Programme, implementation strength scoring for each kebele (ward) was based on aggregated data from interviews with mothers of children aged 12–23 months, reflecting their reports of contact with four elements of the programme. An evaluation of the Avahan HIV prevention programme in India used the cumulative amount of Avahan funding per HIV-infected person spent each year in each district. In these cases, a single measure was developed and the association with hypothesised programme outcomes presented. In the evaluation of the Affordable Medicines Facility—malaria, several implementation strength measures were developed based on the duration of activity of the programme and the level of implementation of supporting interventions. Measuring the strength of programme implementation and assessing its association with outcomes is a promising approach to strengthen pragmatic impact evaluation. Five key aspects of developing an implementation strength measure are to: (a) develop a logic model; (b) identify aspects of implementation to be assessed; (c) design and implement data collection from a range of data sources; (d) decide whether and how to combine data into a single measure; and, (e) plan whether and how to use the measure(s) in outcome analysis.

Health systems guidance appraisal--a critical interpretive synthesis

BACKGROUND

Health systems guidance (HSG) are systematically developed statements that assist with decisions about options for addressing health systems challenges, including related changes in health systems arrangements. However, the development, appraisal, and reporting of HSG poses unique conceptual and methodological challenges related to the varied types of evidence that are relevant, the complexity of health systems, and the pre-eminence of contextual factors. To address this gap, we are conducting a program of research that aims to create a tool to support the appraisal of HSG and further enhance HSG development and reporting. The focus of this paper was to conduct a knowledge synthesis of the published and grey literatures to determine quality criteria (concepts) relevant for this process.

METHODS

We applied a critical interpretive synthesis (CIS) approach to knowledge synthesis that enabled an iterative, flexible, and dynamic analysis of diverse bodies of literature in order to generate a candidate list of concepts that will constitute the foundational components of the HSG tool. Using our review questions as compasses, we were able to guide the search strategy to look for papers based on their potential relevance to HSG appraisal, development, and reporting. The search strategy included various electronic databases and sources, subject-specific journals, conference abstracts, research reports, book chapters, unpublished data, dissertations, and policy documents. Screening the papers and data extraction was completed independently and in duplicate, and a narrative approach to data synthesis was executed.

RESULTS

We identified 43 papers that met eligibility criteria. No existing review was found on this topic, and no HSG appraisal tool was identified. Over one third of the authors implicitly or explicitly identified the need for a high-quality tool aimed to systematically evaluate HSG and contribute to its development/reporting. We identified 30 concepts that may be relevant to the appraisal of HSG and were able to cluster them into three meaningful domains: process principles, content, and context principles.

CONCLUSIONS

Our study showed the role that the quality criteria play in the development, appraisal, and reporting of HSG and demonstrated the link and resonance within and between the various concepts in the three domains.

A proposed model to conduct process and outcome evaluations and implementation research of child health programs in Africa using integrated community case management as an example

Aim

To use a newly devised set of criteria to review the study design and scope of collection of process, outcomes and contextual data for evaluations and implementation research of integrated community case management (iCCM) in Sub–Saharan African.

Methods

We examined 24 program evaluations and implementation research studies of iCCM in sub–Saharan Africa conducted in the last 5 years (2008–2013), assessed the design used and categorized them according to whether or not they collected sufficient information to conduct process and outcome evaluations.

Results

Five of the 24 studies used a stepped wedge design and two were randomized control trials. The remaining 17 were quasi–experimental of which 10 had comparison areas; however, not all comparison areas had a pre and post household survey. With regard to process data, 22 of the studies collected sufficient information to report on implementation strength, and all, except one, could report on program implementation. Most common missing data elements were health facility treatments, service costs, and qualitative data to assess demand. For the measurement of program outcomes, 7 of the 24 studies had a year or less of implementation at scale before the endline survey, 6 of the household surveys did not collect point of service, 10 did not collect timeliness (care seeking within 24 hours of symptoms) and 12 did not have socioeconomic (SES) information. Among the 16 studies with comparison areas, only 5 randomly selected comparison areas, while 10 had appropriate comparison areas.

Conclusions

Several evaluations were done too soon after implementation, lacked information on health facility treatments, costs, demand, timeliness or SES and/or did not have a counterfactual. We propose several study designs and minimal data elements to be collected to provide sufficient information to assess whether iCCM increased timely coverage of treatment for the neediest children in a cost–efficient manner.

The Lives Saved Tool (LiST) as a model for diarrhea mortality reduction

BACKGROUND

Diarrhea is a leading cause of morbidity and mortality among children under five years of age. The Lives Saved Tool (LiST) is a model used to calculate deaths averted or lives saved by past interventions and for the purposes of program planning when costly and time consuming impact studies are not possible.

DISCUSSION

LiST models the relationship between coverage of interventions and outputs, such as stunting, diarrhea incidence and diarrhea mortality. Each intervention directly prevents a proportion of diarrhea deaths such that the effect size of the intervention is multiplied by coverage to calculate lives saved. That is, the maximum effect size could be achieved at 100% coverage, but at 50% coverage only 50% of possible deaths are prevented. Diarrhea mortality is one of the most complex causes of death to be modeled. The complexity is driven by the combination of direct prevention and treatment interventions as well as interventions that operate indirectly via the reduction in risk factors, such as stunting and wasting. Published evidence is used to quantify the effect sizes for each direct and indirect relationship. Several studies have compared measured changes in mortality to LiST estimates of mortality change looking at different sets of interventions in different countries. While comparison work has generally found good agreement between the LiST estimates and measured mortality reduction, where data availability is weak, the model is less likely to produce accurate results. LiST can be used as a component of program evaluation, but should be coupled with more complete information on inputs, processes and outputs, not just outcomes and impact.

SUMMARY

LiST is an effective tool for modeling diarrhea mortality and can be a useful alternative to large and expensive mortality impact studies. Predicting the impact of interventions or comparing the impact of more than one intervention without having to wait for the results of large and expensive mortality studies is critical to keep programs focused and results oriented for continued reductions in diarrhea and all-cause mortality among children under five years of age.

Stock-outs of essential health products in Mozambique - longitudinal analyses from 2011 to 2013

OBJECTIVES

To assess the relationship between health system factors and facility-level EHP stock-outs in Mozambique.

METHODS

Service provisions were assessed in 26 health facilities and 13 district warehouses in Sofala Province, Mozambique, from July to August in 2011-2013. Generalised estimating equations were used to model factors associated with facility-level availability of essential drugs, supplies and equipment.

RESULTS

Stock-out rates for drugs ranged from 1.3% for oral rehydration solution to 20.5% for Depo-Provera and condoms, with a mean stock-out rate of 9.1%; mean stock-out rates were 15.4% for supplies and 4.1% for equipment. Stock-outs at the district level accounted for 27.1% (29/107) of facility-level drug stock-outs and 44.0% (37/84) of supply stock-outs. Each 10-km increase in the distance from district distribution warehouses was associated with a 31% (CI: 22-42%), 28% (CI: 17-40%) or 27% (CI: 7-50%) increase in rates of drug, supply or equipment stock-outs, respectively. The number of heath facility staff was consistently negatively associated with the occurrence of stock-outs.

CONCLUSIONS

Facility-level stock-outs of EHPs in Mozambique are common and appear to disproportionately affect those living far from district capitals and near facilities with few health staff. The majority of facility-level EHP stock-outs in Mozambique occur when stock exists at the district distribution centre. Innovative methods are urgently needed to improve EHP supply chains, requesting and ordering of drugs, facility and district communication, and forecasting of future EHP needs in Mozambique. Increased investments in public-sector human resources for health could potentially decrease the occurrence of EHP stock-outs.

Approaches to ensuring and improving quality in the context of health system strengthening: a cross-site analysis of the five African Health Initiative Partnership programs

BACKGROUND

Integrated into the work in health systems strengthening (HSS) is a growing focus on the importance of ensuring quality of the services delivered and systems which support them. Understanding how to define and measure quality in the different key World Health Organization building blocks is critical to providing the information needed to address gaps and identify models for replication.

DESCRIPTION OF APPROACHES

We describe the approaches to defining and improving quality across the five country programs funded through the Doris Duke Charitable Foundation African Health Initiative. While each program has independently developed and implemented country-specific approaches to strengthening health systems, they all included quality of services and systems as a core principle. We describe the differences and similarities across the programs in defining and improving quality as an embedded process essential for HSS to achieve the goal of improved population health. The programs measured quality across most or all of the six WHO building blocks, with specific areas of overlap in improving quality falling into four main categories: 1) defining and measuring quality; 2) ensuring data quality, and building capacity for data use for decision making and response to quality measurements; 3) strengthened supportive supervision and/or mentoring; and 4) operational research to understand the factors associated with observed variation in quality.

CONCLUSIONS

Learning the value and challenges of these approaches to measuring and improving quality across the key components of HSS as the projects continue their work will help inform similar efforts both now and in the future to ensure quality across the critical components of a health system and the impact on population health.

Improving health information systems for decision making across five sub-Saharan African countries: Implementation strategies from the African Health Initiative

BACKGROUND

Weak health information systems (HIS) are a critical challenge to reaching the health-related Millennium Development Goals because health systems performance cannot be adequately assessed or monitored where HIS data are incomplete, inaccurate, or untimely. The Population Health Implementation and Training (PHIT) Partnerships were established in five sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) to catalyze advances in strengthening district health systems. Interventions were tailored to the setting in which activities were planned.

COMPARISONS ACROSS STRATEGIES

All five PHIT Partnerships share a common feature in their goal of enhancing HIS and linking data with improved decision-making, specific strategies varied. Mozambique, Ghana, and Tanzania all focus on improving the quality and use of the existing Ministry of Health HIS, while the Zambia and Rwanda partnerships have introduced new information and communication technology systems or tools. All partnerships have adopted a flexible, iterative approach in designing and refining the development of new tools and approaches for HIS enhancement (such as routine data quality audits and automated troubleshooting), as well as improving decision making through timely feedback on health system performance (such as through summary data dashboards or routine data review meetings). The most striking differences between partnership approaches can be found in the level of emphasis of data collection (patient versus health facility), and consequently the level of decision making enhancement (community, facility, district, or provincial leadership).

DISCUSSION

Design differences across PHIT Partnerships reflect differing theories of change, particularly regarding what information is needed, who will use the information to affect change, and how this change is expected to manifest. The iterative process of data use to monitor and assess the health system has been heavily communication dependent, with challenges due to poor feedback loops. Implementation to date has highlighted the importance of engaging frontline staff and managers in improving data collection and its use for informing system improvement. Through rigorous process and impact evaluation, the experience of the PHIT teams hope to contribute to the evidence base in the areas of HIS strengthening, linking HIS with decision making, and its impact on measures of health system outputs and impact.

Strengthening integrated primary health care in Sofala, Mozambique

Background

Large increases in health sector investment and policies favoring upgrading and expanding the public sector health network have prioritized maternal and child health in Mozambique and, over the past decade, Mozambique has achieved substantial improvements in maternal and child health indicators. Over this same period, the government of Mozambique has continued to decentralize the management of public sector resources to the district level, including in the health sector, with the aim of bringing decision-making and resources closer to service beneficiaries. Weak district level management capacity has hindered the decentralization process, and building this capacity is an important link to ensure that resources translate to improved service delivery and further improvements in population health. A consortium of the Ministry of Health, Health Alliance International, Eduardo Mondlane University, and the University of Washington are implementing a health systems strengthening model in Sofala Province, central Mozambique.

Description of implementation

The Mozambique Population Health Implementation and Training (PHIT) Partnership focuses on improving the quality of routine data and its use through appropriate tools to facilitate decision making by health system managers; strengthening management and planning capacity and funding district health plans; and building capacity for operations research to guide system-strengthening efforts. This seven-year effort covers all 13 districts and 146 health facilities in Sofala Province.

Evaluation design

A quasi-experimental controlled time-series design will be used to assess the overall impact of the partnership strategy on under-5 mortality by examining changes in mortality pre- and post-implementation in Sofala Province compared with neighboring Manica Province. The evaluation will compare a broad range of input, process, output, and outcome variables to strengthen the plausibility that the partnership strategy led to health system improvements and subsequent population health impact.

Discussion

The Mozambique PHIT Partnership expects to provide evidence on the effect of efforts to improve data quality coupled with the introduction of tools, training, and supervision to improve evidence-based decision making. This contribution to the knowledge base on what works to enhance health systems is highly replicable for rapid scale-up to other provinces in Mozambique, as well as other sub-Saharan African countries with limited resources and a commitment to comprehensive primary health care.

Improving health information systems for decision making across five sub-Saharan African countries: Implementation strategies from the African Health Initiative

BACKGROUND

Weak health information systems (HIS) are a critical challenge to reaching the health-related Millennium Development Goals because health systems performance cannot be adequately assessed or monitored where HIS data are incomplete, inaccurate, or untimely. The Population Health Implementation and Training (PHIT) Partnerships were established in five sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) to catalyze advances in strengthening district health systems. Interventions were tailored to the setting in which activities were planned.

COMPARISONS ACROSS STRATEGIES

All five PHIT Partnerships share a common feature in their goal of enhancing HIS and linking data with improved decision-making, specific strategies varied. Mozambique, Ghana, and Tanzania all focus on improving the quality and use of the existing Ministry of Health HIS, while the Zambia and Rwanda partnerships have introduced new information and communication technology systems or tools. All partnerships have adopted a flexible, iterative approach in designing and refining the development of new tools and approaches for HIS enhancement (such as routine data quality audits and automated troubleshooting), as well as improving decision making through timely feedback on health system performance (such as through summary data dashboards or routine data review meetings). The most striking differences between partnership approaches can be found in the level of emphasis of data collection (patient versus health facility), and consequently the level of decision making enhancement (community, facility, district, or provincial leadership).

DISCUSSION

Design differences across PHIT Partnerships reflect differing theories of change, particularly regarding what information is needed, who will use the information to affect change, and how this change is expected to manifest. The iterative process of data use to monitor and assess the health system has been heavily communication dependent, with challenges due to poor feedback loops. Implementation to date has highlighted the importance of engaging frontline staff and managers in improving data collection and its use for informing system improvement. Through rigorous process and impact evaluation, the experience of the PHIT teams hope to contribute to the evidence base in the areas of HIS strengthening, linking HIS with decision making, and its impact on measures of health system outputs and impact.