Square peg in a round hole: re-thinking our approach to evaluating health system strengthening in low-income and middle-income countries

Identifying vulnerabilities in essential health services: Analysing the effects of system shocks on childhood vaccination delivery in Lebanon

Shocks effects are under-theorised in the growing literature on health system resilience. Existing work has focused on the effects of single shocks on discrete elements within the health system, typically at national level. Using qualitative system dynamics, we explored how effects of multiple shocks interacted across system levels and combined with existing vulnerabilities to produce effects on essential health services delivery, through the prism of a case study on childhood vaccination in Lebanon. Lebanon has experienced a series of shocks in recent years, including large-scale refugee arrivals from neighbouring Syria, the COVID-19 pandemic and a political-economic crisis. We developed a causal loop diagram (CLD) to explore the effects of each shock individually, and in combination. The CLD was developed and validated using qualitative data from interviews with 38 stakeholders working in Lebanon's vaccination delivery system, in roles ranging from national level policy to facility-level service delivery, conducted between February 2020 and January 2022. We found that each of the shocks had different effects on service demand- and supply-side dynamics. These effects cascaded from national through to local levels. Both Syrian refugee movement and the COVID-19 pandemic primarily exposed vulnerabilities in service demand, mainly through slowly emerging knock-on effects on vaccination uptake behaviour among host communities, and fear of contracting infection in crowded health facilities respectively. The economic crisis exposed wider system vulnerabilities, including demand for vaccination as household income collapsed, and supply-side effects such as reduced clinic time for vaccination, declining workforce retention, and reduced availability of viable vaccine doses, among others. Finally, important pathways of interaction between shocks were identified, particularly affecting the balance between demand for vaccination through publicly supported facilities and private clinics. Future research should incorporate dynamic approaches to identifying within-system vulnerabilities and their potential impacts under different scenarios, as a precursor to improved resilience measurement, system preparedness, and intervention targeting.

Assessing the Implementation Determinants of Pilot Malaria Vaccination Programs in Ghana, Kenya, and Malawi through a Complexity Lens: A Rapid Review Using a Consolidated Framework for Implementation Research

In 2019, national immunization programs in Ghana, Kenya, and Malawi commenced the implementation of RTS,S/AS01 vaccination in large-scale pilot schemes. Understanding the implementation context of this malaria vaccination in the pilot countries can provide useful insights for enhancing implementation outcomes in new countries. There has not yet been a proper synthesis of the implementation determinants of malaria vaccination programs. A rapid review was conducted to identify the implementation determinants of the pilot malaria vaccination programs in Ghana, Kenya, and Malawi, and describe the mechanism by which these determinants interact with each other. A literature search was conducted in November 2023 in PubMed and Google Scholar to identify those studies that described the factors affecting malaria vaccine implementation in Ghana, Kenya, and Malawi. Thirteen studies conducted between 2021 and 2023 were included. A total of 62 implementation determinants of malaria vaccination across all five domains of the consolidated framework for implementation research (CFIR) were identified. A causal loop diagram showed that these factors are interconnected and interrelated, identifying nine reinforcing loops and two balancing loops. As additional countries in Africa prepare for a malaria vaccine roll-out, it is pertinent to ensure that they have access to adequate information about the implementation context of countries that are already implementing malaria vaccination programs so that they understand the potential barriers and facilitators. This information can be used to inform context-specific systems enhancement to maximize implementation success. Going forward, primary implementation studies that incorporate the causal loop diagram should be integrated into the malaria vaccine implementation program to enable immunization program managers and other key stakeholders to identify and respond to emerging implementation barriers in a timely and systematic manner, to improve overall implementation performance.

Cost-effectiveness of tuberculosis infection prevention and control interventions in South African clinics: a model-based economic evaluation informed by complexity science methods

INTRODUCTION

Nosocomial Mycobacterium tuberculosis (Mtb) transmission substantially impacts health workers, patients and communities. Guidelines for tuberculosis infection prevention and control (TB IPC) exist but implementation in many settings remains suboptimal. Evidence is needed on cost-effective investments to prevent Mtb transmission that are feasible in routine clinic environments.

METHODS

A set of TB IPC interventions was codesigned with local stakeholders using system dynamics modelling techniques that addressed both core activities and enabling actions to support implementation. An economic evaluation of these interventions was conducted at two clinics in KwaZulu-Natal, employing agent-based models of Mtb transmission within the clinics and in their catchment populations. Intervention costs included the costs of the enablers (eg, strengthened supervision, community sensitisation) identified by stakeholders to ensure uptake and adherence.

RESULTS

All intervention scenarios modelled, inclusive of the relevant enablers, cost less than US$200 per disability-adjusted life-year (DALY) averted and were very cost-effective in comparison to South Africa's opportunity cost-based threshold (US$3200 per DALY averted). Two interventions, building modifications to improve ventilation and maximising use of the existing Central Chronic Medicines Dispensing and Distribution system to reduce the number of clinic attendees, were found to be cost saving over the 10-year model time horizon. Incremental cost-effectiveness ratios were sensitive to assumptions on baseline clinic ventilation rates, the prevalence of infectious TB in clinic attendees and future HIV incidence but remained highly cost-effective under all uncertainty analysis scenarios.

CONCLUSION

TB IPC interventions in clinics, including the enabling actions to ensure their feasibility, afford very good value for money and should be prioritised for implementation within the South African health system.

How to do (or not to do)…Using Causal Loop Diagrams for Health System Research in Low- and Middle-Income Settings

Causal loop diagrams (CLDs) are a systems thinking method that can be used to visualize and unpack complex health system behaviour. They can be employed prospectively or retrospectively to identify the mechanisms and consequences of policies or interventions designed to strengthen health systems and inform discussion with policymakers and stakeholders on actions that may alleviate sub-optimal outcomes. Whilst the use of CLDs in health systems research has generally increased, there is still limited use in low- and middle-income settings. In addition to their suitability for evaluating complex systems, CLDs can be developed where opportunities for primary data collection may be limited (such as in humanitarian or conflict settings) and instead be formulated using secondary data, published or grey literature, health surveys/reports and policy documents. The purpose of this paper is to provide a step-by-step guide for designing a health system research study that uses CLDs as their chosen research method, with particular attention to issues of relevance to research in low- and middle-income countries (LMICs). The guidance draws on examples from the LMIC literature and authors’ own experience of using CLDs in this research area. This paper guides researchers in addressing the following four questions in the study design process; (1) What is the scope of this research? (2) What data do I need to collect or source? (3) What is my chosen method for CLD development? (4) How will I validate the CLD? In providing supporting information to readers on avenues for addressing these key design questions, authors hope to promote CLDs for wider use by health system researchers working in LMICs.

Taking Systems Thinking to the Global Level: Using the WHO Building Blocks to Describe and Appraise the Global Health System in Relation to COVID-19

Adequately preparing for and containing global shocks, such as COVID-19, is a key challenge facing health systems globally. COVID-19 highlights that health systems are multilevel systems, a continuum from local to global. Goals and monitoring indicators have been key to strengthening national health systems but are missing at the supranational level. A framework to strengthen the global system-the global health actors and the governance, finance, and delivery arrangements within which they operate-is urgently needed. In this article, we illustrate how the World Health Organization Building Blocks framework, which has been used to monitor the performance of national health systems, can be applied to describe and appraise the global health system and its response to COVID-19, and identify potential reforms. Key weaknesses in the global response included: fragmented and voluntary financing; non-transparent pricing of medicines and supplies, poor quality standards, and inequities in procurement and distribution; and weak leadership and governance. We also identify positive achievements and identify potential reforms of the global health system for greater resilience to future shocks. We discuss the limitations of the Building Blocks framework and future research directions and reflect on political economy challenges to reform.

Long-term effects of payment for performance on maternal and child health outcomes: evidence from Tanzania

BACKGROUND

The success of payment for performance (P4P) schemes relies on their ability to generate sustainable changes in the behaviour of healthcare providers. This paper examines short-term and longer-term effects of P4P in Tanzania and the reasons for these changes.

METHODS

We conducted a controlled before and after study and an embedded process evaluation. Three rounds of facility, patient and household survey data (at baseline, after 13 months and at 36 months) measured programme effects in seven intervention districts and four comparison districts. We used linear difference-in-difference regression analysis to determine programme effects, and differential effects over time. Four rounds of qualitative data examined evolution in programme design, implementation and mechanisms of change.

RESULTS

Programme effects on the rate of institutional deliveries and antimalarial treatment during antenatal care reduced overtime, with stock out rates of antimalarials increasing over time to baseline levels. P4P led to sustained improvements in kindness during deliveries, with a wider set of improvements in patient experience of care in the longer term. A change in programme management and funding delayed incentive payments affecting performance on some indicators. The verification system became more integrated within routine systems over time, reducing the time burden on managers and health workers. Ongoing financial autonomy and supervision sustained motivational effects in those aspects of care giving not reliant on funding.

CONCLUSION

Our study adds to limited and mixed evidence documenting how P4P effects evolve over time. Our findings highlight the importance of undertaking ongoing assessment of effects over time.

Implementation science protocol for a participatory, theory-informed implementation research programme in the context of health system strengthening in sub-Saharan Africa (ASSET-ImplementER)

OBJECTIVES

ASSET (Health System Strengthening in sub-Saharan Africa) is a health system strengthening (HSS) programme involving eight work-packages (ie, a research study that addresses a specific need for HSS) that aims to develop solutions that support high-quality care. Here we present the protocol for the implementation science (IS) theme within ASSET (ASSET-ImplmentER) that aims to understand what HSS interventions work, for whom and how, and how IS methodologies can be adapted to improve the HSS interventions within resource-poor contexts.

SETTINGS

Publicly funded health facilities in rural and urban areas in in Ethiopia, South Africa, Sierra Leone, and Zimbabwe.

PARTICIPANTS

Research staff including principal investigators, coinvestigators, field staff, PhD students, and research assistants.

INTERVENTIONS

Work-packages use a mixed-methods effectiveness-effectiveness hybrid designs. At the end of the pre-implementation phase, a workshop is held whereby the IS theme, jointly with ASSET work-packages apply IS determinant frameworks to research findings to identify factors that influence the effectiveness of delivering evidence-informed care. Determinants are used to select a set of HSS interventions for further evaluation, where work-packages also theorise selective mechanisms.In the piloting and rolling implementation phase, work-packages pilot the HSS interventions. An iterative process then begins involving evaluation, reflection and adaptation. Throughout this phase, IS determinant frameworks are applied to monitor and identify barriers/enablers to implementation. Selective mechanisms of action are also investigated. Implementation outcomes are evaluated using qualitative and quantitative methods. The psychometric properties of outcome measures including acceptability, appropriateness and feasibility are also evaluated. In a final workshop, work-packages come together, to reflect and explore the utility of the selected IS methods and provide suggestions for future use.Structured templates are used to organise and analyse common and heterogeneous patterns across work-packages. Qualitative data are analysed using thematic analysis and quantitative data are analysed using means and proportions.

CONCLUSIONS

We use a novel combination of IS methods at a programmatic level to facilitate comparisons of determinants and mechanisms that influence the effectiveness of HSS interventions in achieving implementation outcomes across different contexts. The study also contributes conceptual development and clarification at the underdeveloped interface of IS, HSS and global health.The ASSET-ImplementER theme is considered minimal risk as we only interview researchers involved in the different work-packages. To this effect we have received approval from King's College London Ethics Committee for research that is considered minimal risk (Reference number: MRA-20/21-21772).

Forty years after Alma-Ata: primary health-care preparedness for chronic diseases in Mozambique, Nepal and Peru

Background

Four decades after the Alma-Ata Declaration, strengthening primary health care (PHC) remains a priority for health systems, especially in low- and middle-income countries (LMICs). Given the prominence of chronic diseases as a global health issue, PHC must include a wide range of components in order to provide adequate care.

Objective

To assess PHC preparedness to provide chronic care in Mozambique, Nepal and Peru, we used, as ‘tracer conditions’, diabetes, hypertension and a country-specific neglected tropical disease with chronic sequelae in each country.

Methods

By implementing a health system assessment, we collected quantitative and qualitative data from primary and secondary sources, including interviews of key informants at three health-system levels (macro, meso and micro). The World Health Organization’s health-system building blocks provided the basis for content analysis.

Results

In total, we conducted 227 interviews. Our findings show that the ambitious policies targeting specific diseases lack the support of technical, administrative and financial resources. Data collection systems do not allow the monitoring of individual patients or provide the health system with the information it requires. Patients receive limited disease-specific information. Clinical guidelines and training are either non-existent or not adapted to local contexts. Availability of medicines and diagnostic tests at the PHC level is an issue. Although medicines available through the public health care system are affordable, some essential medicines suffer shortages or are not available to PHC providers. This need, along with a lack of clear referral procedures and available transportation, generates financial issues for individuals and affects access to health care.

Conclusion

PHC in these LMICs is not well prepared to provide adequate care for chronic diseases. Improving PHC to attain universal health coverage requires strengthening the identified weaknesses across health-system building blocks.

A realist review to assess for whom, under what conditions and how pay for performance programmes work in low- and middle-income countries

Pay for performance (P4P) programmes are popular health system-focused interventions aiming to improve health outcomes in low-and middle-income countries (LMICs). This realist review aims to understand how, why and under what circumstance P4P works in LMICs.We systematically searched peer-reviewed and grey literature databases, and examined the mechanisms underpinning P4P effects on: utilisation of services, patient satisfaction, provider productivity and broader health system, and contextual factors moderating these. This evidence was then used to construct a causal loop diagram.We included 112 records (19 grey literature; 93 peer-reviewed articles) assessing P4P schemes in 36 countries. Although we found mixed evidence of P4P's effects on identified outcomes, common pathways to improved outcomes include: community outreach; adherence to clinical guidelines, patient-provider interactions, patient trust, facility improvements, access to drugs and equipment, facility autonomy, and lower user fees. Contextual factors shaping the system response to P4P include: degree of facility autonomy, efficiency of banking, role of user charges in financing public services; staffing levels; staff training and motivation, quality of facility infrastructure and community social norms. Programme design features supporting or impeding health system effects of P4P included: scope of incentivised indicators, fairness and reach of incentives, timely payments and a supportive, robust verification system that does not overburden staff. Facility bonuses are a key element of P4P, but rely on provider autonomy for maximum effect. If health system inputs are vastly underperforming pre-P4P, they are unlikely to improve only due to P4P. This is the first realist review describing how and why P4P initiatives work (or fail) in different LMIC contexts by exploring the underlying mechanisms and contextual and programme design moderators. Future studies should systematically examine health system pathways to outcomes for P4P and other health system strengthening initiatives, and offer more understanding of how programme design shapes mechanisms and effects.

Priority setting for health system strengthening in low income countries. A qualitative case study illustrating the complexities

Health systems are critical to the realisation of Universal Health Coverage. There has been insufficient attention to the evaluation of priority setting for health system strengthening within low income countries, including evaluation of the local capacity to implement priorities. This study evaluated the extent to which health system strengthening was prioritized in Uganda. The Kapiriri & Martin framework was used to evaluate health system priority setting from 2005-2015. A document analysis was triangulated with interview data (n = 67) from global, national and subnational stakeholders and analysed using content analysis. Health system strengthening was perceived to be circumvented by a lack of resources as well as influential actors with disease focused, rather than system-oriented, interests. There were defined processes with explicit criteria for identifying priorities and evidence was highly valued. But sub-optimal transparency and weak accountability often compromised the integrity of priority setting and contributed to stalling progress on health system strengthening and achieving health system outcomes. The strengths in the current planning processes should be harnessed. In addition, a systematic approach to priority setting, potentially through the establishment of an independent body, and stronger oversight mechanisms, would strengthen health system planning in this setting.

The implementation of the free maternal health policy in rural Northern Ghana: synthesised results and lessons learnt

Objective

A free maternal health policy was implemented under Ghana’s National Health Insurance Scheme to promote the use of maternal health services. Under the policy, women are entitled to free services throughout pregnancy and at childbirth. A mixed methods study involving women, providers and insurance managers was carried out in the Kassena-Nankana municipality of Ghana. It explored the affordability, availability, acceptability and quality of services. In this manuscript, we present synthesised results categorised as facilitators and barriers to access as well as lessons learnt (implications).

Results

Reasonable waiting times, cleanliness of facilities as well as good interpersonal relationships with providers were the facilitators to access. Barriers included out of pocket payments, lack of, or inadequate supply of drugs and commodities, equipment, water, electricity and emergency transport. Four lessons (implications) were identified. Firstly, out of pocket payments persisted. Secondly, the health system was not strengthened before implementing the free maternal health policy. Thirdly, lower level facilities were poorly resourced. Finally, the lack of essential inputs and infrastructure affected quality of care and therefore, access to care. It is suggested that the Government of Ghana, the Health Insurance Scheme and other stakeholders improve the provision of resources to facilities.