Understanding causal pathways within health systems policy evaluation through mediation analysis: an application to payment for performance (P4P) in Tanzania

Impact of health system governance on healthcare quality in low-income and middle-income countries: a scoping review

INTRODUCTION

Improving healthcare quality in low-/middle-income countries (LMICs) is a critical step in the pathway to Universal Health Coverage and health-related sustainable development goals. This study aimed to map the available evidence on the impacts of health system governance interventions on the quality of healthcare services in LMICs.

METHODS

We conducted a scoping review of the literature. The search strategy used a combination of keywords and phrases relevant to health system governance, quality of healthcare and LMICs. Studies published in English until August 2023, with no start date limitation, were searched on PubMed, Cochrane Library, CINAHL, Web of Science, Scopus, Google Scholar and ProQuest. Additional publications were identified by snowballing. The effects reported by the studies on processes of care and quality impacts were reviewed.

RESULTS

The findings from 201 primary studies were grouped under (1) leadership, (2) system design, (3) accountability and transparency, (4) financing, (5) private sector partnerships, (6) information and monitoring; (7) participation and engagement and (8) regulation.

CONCLUSIONS

We identified a stronger evidence base linking improved quality of care with health financing, private sector partnerships and community participation and engagement strategies. The evidence related to leadership, system design, information and monitoring, and accountability and transparency is limited.

Back to basics: A mediation analysis approach to addressing the fundamental questions of integrated care evaluations

Health systems around the world are aiming to improve the integration of health and social care services to deliver better care for patients. Existing evaluations have focused exclusively on the impact of care integration on health outcomes and found little effect. That suggests the need to take a step back and ask whether integrated care programmes actually lead to greater clinical integration of care and indeed whether greater integration is associated with improved health outcomes. We propose a mediation analysis approach to address these two fundamental questions when evaluating integrated care programmes. We illustrate our approach by re-examining the impact of an English integrated care program on clinical integration and assessing whether greater integration is causally associated with fewer admissions for ambulatory care sensitive conditions. We measure clinical integration using a concentration index of outpatient referrals at the general practice level. While we find that the scheme increased integration of primary and secondary care, clinical integration did not mediate a decrease in unplanned hospital admissions. Our analysis emphasizes the need to better understand the hypothesized causal impact of integration on health outcomes and demonstrates how mediation analysis can inform future evaluations and program design.

A critical interpretive synthesis of power and mistreatment of women in maternity care

Labouring women may be subjected to physical and verbal abuse that reflects dynamics of power, described as Mistreatment of Women (MoW). This Critical Interpretive Synthesis on power and MoW consolidates current research and advances theory and practice through inter-disciplinary literature exploration. The review was undertaken in 3 phases. Phase 1 consisted of topic scoping; phase 2 entailed exploration of key power-related drivers emerging from the topic scoping; and phase 3 entailed data synthesis and analysis, with a particular focus on interventions. We identified 63 papers for inclusion in Phase 1. These papers utilized a variety of methods and approaches and represented a wide range of geographic regions. The power-related drivers of mistreatment in these articles span multiple levels of the social ecological model, including intrapersonal (e.g. lack of knowledge about one's rights), interpersonal (e.g. patient-provider hierarchy), community (e.g. widespread discrimination against indigenous women), organizational (e.g. pressure to achieve performance goals), and law/policy (e.g. lack of accountability for rights violations). Most papers addressed more than one level of the social-ecological model, though a significant minority were focused just on interpersonal factors. During Phase 1, we identified priority themes relating to under-explored power-related drivers of MoW for exploration in Phase 2, including lack of conscientization and normalization of MoW; perceptions of fitness for motherhood; geopolitical and ethnopolitical projects related to fertility; and pressure to achieve quantifiable performance goals. We ultimately included 104 papers in Phase 2. The wide-ranging findings from Phase 3 (synthesis and analysis) coalesce in several key meta-themes, each with their own evidence-base for action. Consistent with the notion that research on power can point us to "drivers of the drivers," the paper includes some intervention-relevant insights for further exploration, including as relating to broader social norms, health systems design, and the utility of multi-level strategies.

Implementing essential diagnostics-learning from essential medicines: A scoping review

The World Health Organization (WHO) model list of Essential In vitro Diagnostic (EDL) introduced in 2018 complements the established Essential Medicines List (EML) and improves its impact on advancing universal health coverage and better health outcomes. We conducted a scoping review of the literature on implementing the WHO essential lists in Africa to inform the implementation of the recently introduced EDL. We searched eight electronic databases for studies reporting on implementing the WHO EDL and EML in Africa. Two authors independently conducted study selection and data extraction, with disagreements resolved through discussion. We used the Supporting the Use of Research Evidence (SURE) framework to extract themes and synthesised findings using thematic content analysis. We used the Mixed Method Appraisal Tool (MMAT) version 2018 to assess the quality of included studies. We included 172 studies reporting on EDL and EML after screening 3,813 articles titles and abstracts and 1,545 full-text papers. Most (75%, n = 129) studies were purely quantitative in design, comprising descriptive cross-sectional designs (60%, n = 104), 15% (n = 26) were purely qualitative, and 10% (n = 17) had mixed-methods approaches. There were no qualitative or randomised experimental studies about EDL. The main barrier facing the EML and EDL was poorly equipped health facilities-including unavailability or stock-outs of essential in vitro diagnostics and medicines. Financial and non-financial incentives to health facilities and workers were key enablers in implementing the EML; however, their impact differed from one context to another. Only fifty-six (33%) of the included studies were of high quality. Poorly equipped and stocked health facilities remain an implementation barrier to essential diagnostics and medicines. Health system interventions such as financial and non-financial incentives to improve their availability can be applied in different contexts. More implementation study designs, such as experimental and qualitative studies, are required to evaluate the effectiveness of essential lists.

Implementing value-based healthcare using a digital health exchange platform to improve pregnancy and childbirth outcomes in urban and rural Kenya

Maternal and neonatal mortality rates in many low- and middle-income countries (LMICs) are still far above the targets of the United Nations Sustainable Development Goal 3. Value-based healthcare (VBHC) has the potential to outperform traditional supply-driven approaches in changing this dismal situation, and significantly improve maternal, neonatal and child health (MNCH) outcomes. We developed a theory of change and used a cohort-based implementation approach to create short and long learning cycles along which different components of the VBHC framework were introduced and evaluated in Kenya. At the core of the approach was a value-based care bundle for maternity care, with predefined cost and quality of care using WHO guidelines and adjusted to the risk profile of the pregnancy. The care bundle was implemented using a digital exchange platform that connects pregnant women, clinics and payers. The platform manages financial transactions, enables bi-directional communication with pregnant women via SMS, collects data from clinics and shares enriched information via dashboards with payers and clinics. While the evaluation of health outcomes is ongoing, first results show improved adherence to evidence-based care pathways at a predictable cost per enrolled person. This community case study shows that implementation of the VBHC framework in an LMIC setting is possible for MNCH. The incremental, cohort-based approach enabled iterative learning processes. This can support the restructuring of health systems in low resource settings from an output-driven model to a value based financing-driven model.

Examining the Implementation of the Performance-Based Financing Equity Strategy in Improving Access and Utilization of Maternal Health Services in Cameroon: A Qualitative Study

Performance-based financing (PBF)-a supply-side strategy that incentivizes health providers based on predefined quality and quantity criteria-introduced an innovative approach to reaching the poor by means of using PBF equity instruments. These PBF equity instruments include paying providers more to reach out to poor women, selecting services used by the poor, subsidizing user fees to reduce out-of-pocket expenses, and adding complementary demand-side intervention. Before the implementation of the PBF equity instrument in Cameroon, there were few initiatives/schemes to enable the poor to access maternal health services. Moreover, there is a significant research gap on how the equity elements are defined and implemented across contexts. This study aims to understand (i) how health facilities define and classify the poor and vulnerable in the context of PBF, (ii) how the equity elements are implemented at the community and facility levels, and (iii) the potential impact on access to and the use of maternal health services at the facility level and challenges in the implementation process. We used key informant interviews and focus group discussions (FGDs) based on a grounded theory approach to gain an understanding of the social processes and experiences. Data were collected from three districts in the Southwest region of Cameroon from April 2021 to August 2021. Data were transcribed and analyzed using MaxQDA. The thematic analysis approach/technique was used to analyze data. Key informant interviews and focus groups were conducted with 79 participants, including 28 health professionals and service administrators, 27 pregnant women, and 24 community health workers in three districts. Health facilities employed various subjective approaches to assess and define poor and vulnerable (PAV) persons. Home visits were reported to have an impact in reaching the poor and vulnerable to improve access to maternal services. Meanwhile, a delay in the payment of PBF incentives was reported to be the main challenge that had a negative relationship with the consistent provision of care to the poor and vulnerable, especially in private health facilities. The theory generated from our findings suggests that the impact of the PBF equity elements specific to maternal health depends on (i) a shared understanding of the definition of PAV among different stakeholders, including providers and users, as well as how the PAV is operationalized (structure), and (ii) the appropriate and timely payment of incentives to health facilities and health providers.

Developing an implementation research logic model: using a multiple case study design to establish a worked exemplar

BACKGROUND

Implementation science frameworks explore, interpret, and evaluate different components of the implementation process. By using a program logic approach, implementation frameworks with different purposes can be combined to detail complex interactions. The Implementation Research Logic Model (IRLM) facilitates the development of causal pathways and mechanisms that enable implementation. Critical elements of the IRLM vary across different study designs, and its applicability to synthesizing findings across settings is also under-explored. The dual purpose of this study is to develop an IRLM from an implementation research study that used case study methodology and to demonstrate the utility of the IRLM to synthesize findings across case sites.

METHOD

The method used in the exemplar project and the alignment of the IRLM to case study methodology are described. Cases were purposely selected using replication logic and represent organizations that have embedded exercise in routine care for people with cancer or mental illness. Four data sources were selected: semi-structured interviews with purposely selected staff, organizational document review, observations, and a survey using the Program Sustainability Assessment Tool (PSAT). Framework analysis was used, and an IRLM was produced at each case site. Similar elements within the individual IRLM were identified, extracted, and re-produced to synthesize findings across sites and represent the generalized, cross-case findings.

RESULTS

The IRLM was embedded within multiple stages of the study, including data collection, analysis, and reporting transparency. Between 33-44 determinants and 36-44 implementation strategies were identified at sites that informed individual IRLMs. An example of generalized findings describing "intervention adaptability" demonstrated similarities in determinant detail and mechanisms of implementation strategies across sites. However, different strategies were applied to address similar determinants. Dependent and bi-directional relationships operated along the causal pathway that influenced implementation outcomes.

CONCLUSIONS

Case study methods help address implementation research priorities, including developing causal pathways and mechanisms. Embedding the IRLM within the case study approach provided structure and added to the transparency and replicability of the study. Identifying the similar elements across sites helped synthesize findings and give a general explanation of the implementation process. Detailing the methods provides an example for replication that can build generalizable knowledge in implementation research.

Revisiting concepts of evidence in implementation science

BACKGROUND

Evidence, in multiple forms, is a foundation of implementation science. For public health and clinical practice, evidence includes the following: type 1 evidence on etiology and burden; type 2 evidence on effectiveness of interventions; and type 3: evidence on dissemination and implementation (D&I) within context. To support a vision for development and use of evidence in D&I science that is more comprehensive and equitable (particularly for type 3 evidence), this article aims to clarify concepts of evidence, summarize ongoing debates about evidence, and provide a set of recommendations and tools/resources for addressing the "how-to" in filling evidence gaps most critical to advancing implementation science.

MAIN TEXT

Because current conceptualizations of evidence have been relatively narrow and insufficiently characterized in our opinion, we identify and discuss challenges and debates about the uses, usefulness, and gaps in evidence for implementation science. A set of questions is proposed to assist in determining when evidence is sufficient for dissemination and implementation. Intersecting gaps include the need to (1) reconsider how the evidence base is determined, (2) improve understanding of contextual effects on implementation, (3) sharpen the focus on health equity in how we approach and build the evidence-base, (4) conduct more policy implementation research and evaluation, and (5) learn from audience and stakeholder perspectives. We offer 15 recommendations to assist in filling these gaps and describe a set of tools for enhancing the evidence most needed in implementation science.

CONCLUSIONS

To address our recommendations, we see capacity as a necessary ingredient to shift the field's approach to evidence. Capacity includes the "push" for implementation science where researchers are trained to develop and evaluate evidence which should be useful and feasible for implementers and reflect community or stakeholder priorities. Equally important, there has been inadequate training and too little emphasis on the "pull" for implementation science (e.g., training implementers, practice-based research). We suggest that funders and reviewers of research should adopt and support a more robust definition of evidence. By critically examining the evolving nature of evidence, implementation science can better fulfill its vision of facilitating widespread and equitable adoption, delivery, and sustainment of scientific advances.

Perceptions of nurses and physicians on pay-for-performance in hospital: a systematic review of qualitative studies

AIMS

To systematically examine perceptions of nurses and physicians on pay-for-performance in hospital.

BACKGROUND

Pay-for-performance projects have proliferated over the past two decades, most systematic reviews of which solely focused on its effectiveness in primary healthcare and the physicians' or nurses' attitudes. However, systematic reviews of qualitative approaches for better examining perceptions of both nurses and physicians in hospital are lacking.

EVALUATION

Electronic databases were systematic searched with date from its inception to December 31, 2020. Meta-aggregation synthesis methodology and the conceptual framework of the Theory of Planned Behavior were used to summarize findings.

KEY ISSUES

A total of nine studies were included. Three major synthesized themes were identified: (1) perceptions of the motivation effects and positive outcomes (2) perceptions about the design defects and negative effects (3) perceptions of the obstacles in the implementation process.

CONCLUSION

To maximize the intended positive effects, nurses' and physicians' perceptions should be considered and incorporated into the project design and implementation stage.

IMPLICATIONS FOR NURSING MANAGEMENT AND RESEARCH

The paper gives enlightenment to nurse managers on improving and advancing the cause of nurses when planning for or evaluating their institutions' policies on pay-for-performance in the future research.

Understanding the maternal and child health system response to payment for performance in Tanzania using a causal loop diagram approach

Payment for performance (P4P) has been employed in low and middle-income (LMIC) countries to improve quality and coverage of maternal and child health (MCH) services. However, there is a lack of consensus on how P4P affects health systems. There is a need to evaluate P4P effects on health systems using methods suitable for evaluating complex systems. We developed a causal loop diagram (CLD) to further understand the pathways to impact of P4P on delivery and uptake of MCH services in Tanzania. The CLD was developed and validated using qualitative data from a process evaluation of a P4P scheme in Tanzania, with additional stakeholder dialogue sought to strengthen confidence in the diagram. The CLD maps the interacting mechanisms involved in provider achievement of targets, reporting of health information, and population care seeking, and identifies those mechanisms affected by P4P. For example, the availability of drugs and medical commodities impacts not only provider achievement of P4P targets but also demand of services and is impacted by P4P through the availability of additional facility resources and the incentivisation of district managers to reduce drug stock outs. The CLD also identifies mechanisms key to facility achievement of targets but are not within the scope of the programme; the activities of health facility governing committees and community health workers, for example, are key to demand stimulation and effective resource use at the facility level but both groups were omitted from the incentive system. P4P design considerations generated from this work include appropriately incentivising the availability of drugs and staffing in facilities and those responsible for demand creation in communities. Further research using CLDs to study heath systems in LMIC is urgently needed to further our understanding of how systems respond to interventions and how to strengthen systems to deliver better coverage and quality of care.

•

Holistic analysis key to avoiding suboptimal P4P performance.

•

Availability of drugs is influenced by P4P and affects success of P4P.

•

Need to incentivise groups outside facility that support service coverage.

Can Results-Based Financing improve health outcomes in resource poor settings? Evidence from Zimbabwe

Result Based Financing (RBF) has been implemented in health systems across low and middle-income countries (LMICs), with the objective of improving population health. Most evaluations of RBF schemes have focused on average programme effects for incentivised services. There is limited evidence on the potential effect of RBF on health outcomes, as well as on the heterogeneous effects across socio-economic groups and time periods. This study analyses the effect of Zimbabwe's national RBF scheme on neonatal, infant and under five mortality, using Demographic and Health Survey data from 2005, 2010 and 2015. We use a difference in differences design, which exploits the staggered roll-out of the scheme across 60 districts. We examine average programme effects and perform sub-group analyses to assess differences between socio-economic groups. We find that RBF reduced under-five mortality by two percentage points overall, but that this decrease was only significant for children of mothers with above median wealth (2.7 percentage points) and education (2.1 percentage points). RBF increased institutional delivery by seven percentage points – with a statistically significant effect for poorer socio-economic groups and least educated. We also find that RBF reduced c-section rates by three percentage points. We find no detectable effect of RBF on other incentivised services. When considering programme effects over time, we find that effects were only observed during the second phase of the programme (March 2012) with the exception of c-sections, which only reduced in the longer term. Further research is needed to examine whether these findings can be generalised to other settings.

•

Zimbabwe's national Results Based Financing scheme decreased under-five mortality.

•

Among higher wealth and education groups.

•

RBF increased institutional deliveries in the poorest and least educated groups and reduced the rate of c-section.

•

There were no programme effects on other incentivised services.

•

The RBF programme effects generally appear to be stronger earlier in the implementation process.

Paying for performance to improve the delivery of health interventions in low- and middle-income countries

BACKGROUND

There is growing interest in paying for performance (P4P) as a means to align the incentives of healthcare providers with public health goals. Rigorous evidence on the effectiveness of these strategies in improving health care and health in low- and middle-income countries (LMICs) is lacking; this is an update of the 2012 review on this topic.

OBJECTIVES

To assess the effects of paying for performance on the provision of health care and health outcomes in low- and middle-income countries.

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, and 10 other databases between April and June 2018. We also searched two trial registries, websites, online resources of international agencies, organizations and universities, and contacted experts in the field. Studies identified from rerunning searches in 2020 are under 'Studies awaiting classification.'

SELECTION CRITERIA

We included randomized or non-randomized trials, controlled before-after studies, or interrupted time series studies conducted in LMICs (as defined by the World Bank in 2018). P4P refers to the transfer of money or material goods conditional on taking a measurable action or achieving a predetermined performance target. To be included, a study had to report at least one of the following outcomes: patient health outcomes, changes in targeted measures of provider performance (such as the delivery of healthcare services), unintended effects, or changes in resource use.

DATA COLLECTION AND ANALYSIS

We extracted data as per original review protocol and narratively synthesised findings. We used standard methodological procedures expected by Cochrane. Given diversity and variability in intervention types, patient populations, analyses and outcome reporting, we deemed meta-analysis inappropriate. We noted the range of effects associated with P4P against each outcome of interest. Based on intervention descriptions provided in documents, we classified design schemes and explored variation in effect by scheme design.

MAIN RESULTS

We included 59 studies: controlled before-after studies (19), non-randomized (16) or cluster randomized trials (14);  and interrupted time-series studies (9). One study included both an interrupted time series and a controlled before-after study. Studies focused on a wide range of P4P interventions, including target payments and payment for outputs as modified by quality (or quality and equity assessments). Only one study assessed results-based aid. Many schemes were funded by national governments (23 studies) with the World Bank funding most externally funded schemes (11 studies). Targeted services varied; however, most interventions focused on reproductive, maternal and child health indicators. Participants were predominantly located in public or in a mix of public, non-governmental and faith-based facilities (54 studies). P4P was assessed predominantly at health facility level, though districts and other levels were also involved. Most studies assessed the effects of P4P against a status quo control (49 studies); however, some studies assessed effects against comparator interventions (predominantly enhanced financing intended to match P4P funds (17 studies)). Four studies reported intervention effects against both comparator and status quo. Controlled before-after studies were at higher risk of bias than other study designs. However, some randomised trials were also downgraded due to risk of bias. The interrupted time-series studies provided insufficient information on other concurrent changes in the study context. P4P compared to a status quo control For health services that are specifically targeted, P4P may slightly improve health outcomes (low certainty evidence), but few studies assessed this. P4P may also improve service quality overall (low certainty evidence); and probably increases the availability of health workers, medicines and well-functioning infrastructure and equipment (moderate certainty evidence). P4P may have mixed effects on the delivery and use of services (low certainty evidence) and may have few or no distorting unintended effects on outcomes that were not targeted (low-certainty evidence), but few studies assessed these. For secondary outcomes, P4P may make little or no difference to provider absenteeism, motivation or satisfaction (low certainty evidence); but may improve patient satisfaction and acceptability (low certainty evidence); and may positively affect facility managerial autonomy (low certainty evidence). P4P probably makes little to no difference to management quality or facility governance (low certainty evidence). Impacts on equity were mixed (low certainty evidence). For health services that are untargeted, P4P probably improves some health outcomes (moderate certainty evidence); may improve the delivery, use and quality of some health services but may make little or no difference to others (low certainty evidence); and may have few or no distorting unintended effects (low certainty evidence). The effects of P4P on the availability of medicines and other resources are uncertain (very low certainty evidence). P4P compared to other strategies For health outcomes and services that are specifically targeted, P4P may make little or no difference to health outcomes (low certainty evidence), but few studies assessed this. P4P may improve service quality (low certainty evidence); and may have mixed effects on the delivery and use of health services and on the availability of equipment and medicines (low certainty evidence). For health outcomes and services that are untargeted, P4P may make little or no difference to health outcomes and to the delivery and use of health services (low certainty evidence). The effects of P4P on service quality, resource availability and unintended effects are uncertain (very low certainty evidence). Findings of subgroup analyses Results-based aid, and schemes using payment per output adjusted for service quality, appeared to yield the greatest positive effects on outcomes. However, only one study evaluated results-based aid, so the effects may be spurious. Overall, schemes adjusting both for quality of service and rewarding equitable delivery of services appeared to perform best in relation to service utilization outcomes.

AUTHORS' CONCLUSIONS

The evidence base on the impacts of P4P schemes has grown considerably, with study quality gradually increasing. P4P schemes may have mixed effects on outcomes of interest, and there is high heterogeneity in the types of schemes implemented and evaluations conducted. P4P is not a uniform intervention, but rather a range of approaches. Its effects depend on the interaction of several variables, including the design of the intervention (e.g., who receives payments ), the amount of additional funding,  ancillary components (such as technical support) and contextual factors (including organizational context).

Adapting results-based financing to respond to endogenous and exogenous moderators in Zimbabwe

Purpose

This study examines key adaptations that occurred in the Zimbabwean Results-Based Financing (RBF) programme between 2010 and 2017, locating the endogenous and exogenous factors that required adaptive response and the processes from which changes were made.

Design/methodology/approach

The study is based on a desk review and thematic analysis of 64 policy and academic literatures supplemented with 28 multi-stakeholder interviews.

Findings

The programme experienced substantive adaption between 2010 and 2017, demonstrating a significant level of responsiveness towards increasing efficiency as well as to respond to unforeseen factors that undermined RBF mechanisms. The programme was adaptive due to its phased design, which allowed revision competencies and responsive adaptation, which provide useful insights for other low-and-middle income countries (LMICs) settings where graduated scale-up might better meet contextualised needs. However, exogenous factors were often not systematically examined or reported in RBF evaluations, demonstrating that adaptation could have been better anticipated, planned, reported and communicated, especially if RBF is to be a more effective health system reform tool.

Originality/value

RBF is an increasingly popular health system reform tool in LMICs. However, there are questions about how exogenous factors affect RBF performance and acknowledgement that unforeseen endogenous programme design and implementation factors also greatly affect the performance of RBF. As a result, a better understanding of how RBF operates and adapts to programme level (endogenous) and exogenous (external) factors in LMICs is necessary.

Is the evidence on the effectiveness of pay for performance schemes in healthcare changing? Evidence from a meta-regression analysis

BACKGROUND

This study investigated if the evidence on the success of the Pay for Performance (P4P) schemes in healthcare is changing as the schemes continue to evolve by updating a previous systematic review.

METHODS

A meta-regression analysis using 116 studies evaluating P4P schemes published between January 2010 to February 2018. The effects of the research design, incentive schemes, use of incentives, and the size of the payment to revenue ratio on the proportion of statically significant effects in each study were examined.

RESULTS

There was evidence of an increase in the range of countries adopting P4P schemes and weak evidence that the proportion of studies with statistically significant effects have increased. Factors hypothesized to influence the success of schemes have not changed. Studies evaluating P4P schemes which made payments for improvement over time, were associated with a lower proportion of statistically significant effects. There was weak evidence of a positive association between the incentives' size and the proportion of statistically significant effects.

CONCLUSION

The evidence on the effectiveness of P4P schemes is evolving slowly, with little evidence that lessons are being learned concerning the design and evaluation of P4P schemes.

A realist review to assess for whom, under what conditions and how pay for performance programmes work in low- and middle-income countries

Pay for performance (P4P) programmes are popular health system-focused interventions aiming to improve health outcomes in low-and middle-income countries (LMICs). This realist review aims to understand how, why and under what circumstance P4P works in LMICs.We systematically searched peer-reviewed and grey literature databases, and examined the mechanisms underpinning P4P effects on: utilisation of services, patient satisfaction, provider productivity and broader health system, and contextual factors moderating these. This evidence was then used to construct a causal loop diagram.We included 112 records (19 grey literature; 93 peer-reviewed articles) assessing P4P schemes in 36 countries. Although we found mixed evidence of P4P's effects on identified outcomes, common pathways to improved outcomes include: community outreach; adherence to clinical guidelines, patient-provider interactions, patient trust, facility improvements, access to drugs and equipment, facility autonomy, and lower user fees. Contextual factors shaping the system response to P4P include: degree of facility autonomy, efficiency of banking, role of user charges in financing public services; staffing levels; staff training and motivation, quality of facility infrastructure and community social norms. Programme design features supporting or impeding health system effects of P4P included: scope of incentivised indicators, fairness and reach of incentives, timely payments and a supportive, robust verification system that does not overburden staff. Facility bonuses are a key element of P4P, but rely on provider autonomy for maximum effect. If health system inputs are vastly underperforming pre-P4P, they are unlikely to improve only due to P4P. This is the first realist review describing how and why P4P initiatives work (or fail) in different LMIC contexts by exploring the underlying mechanisms and contextual and programme design moderators. Future studies should systematically examine health system pathways to outcomes for P4P and other health system strengthening initiatives, and offer more understanding of how programme design shapes mechanisms and effects.

Pay-for-performance reduces bypassing of health facilities: Evidence from Tanzania

Many patients and expectant mothers in low-income countries bypass local health facilities in search of better-quality services. This study examines the impact of a payment-for-performance (P4P) scheme on bypassing practices among expectant women in Tanzania. We expect the P4P intervention to reduce incidences of bypassing by improving the quality of services in local health facilities, thereby reducing the incentive to migrate. We used a difference-in-difference regression model to assess the impact of P4P on bypassing after one year and after three years. In addition, we implemented a machine learning approach to identify factors that predict bypassing. Overall, 38% of women bypassed their local health service provider to deliver in another facility. Our analysis shows that the P4P scheme significantly reduced bypassing. On average, P4P reduced bypassing in the study area by 17% (8 percentage points) over three years. We also identified two main predictors of bypassing - facility type and the distance to the closest hospital. Women are more likely to bypass if their local facility is a dispensary instead of a hospital or a health center. Women are less likely to bypass if they live close to a hospital.

Pay for Performance: A Reflection on How a Global Perspective Could Enhance Policy and Research

Pay-for-performance (P4P) is the provision of financial incentives to healthcare providers based on pre-specified performance targets. P4P has been used as a policy tool to improve healthcare provision globally. However, researchers tend to cluster into those working on high or low- and middle-income countries (LMICs), with still limited knowledge exchange, potentially constraining opportunities for learning from across income settings. We reflect here on some commonalities and differences in the design of P4P schemes, research questions, methods and data across income settings. We highlight how a global perspective on knowledge synthesis could lead to innovations and further knowledge advancement.

The development of a new accountability measurement framework and tool for global health initiatives

The Global Strategy for Women’s Children’s and Adolescents’ Health emphasizes accountability as essential to ensure that decision-makers have the information required to meet the health needs of their populations and stresses the importance of tracking resources, results, and rights to see ‘what works, what needs improvement and what requires increased attention’. However, results from accountability initiatives are mixed and there is a lack of broadly applicable, validated tools for planning, monitoring and evaluating accountability interventions. This article documents an effort to transform accountability markers—including political will, leadership and the monitor–review–act cycle—into a measurement tool that can be used prospectively or retrospectively to plan, monitor and evaluate accountability initiatives. It describes the development process behind the tool including the literature review, framework development and subsequent building of the measurement tool itself. It also examines feedback on the tool from a panel of global experts and the results of a pilot test conducted in Bauchi and Gombe states in Nigeria. The results demonstrate that the tool is an effective aid for accountability initiatives to reflect on their own progress and provides a useful structure for future planning, monitoring and evaluation. The tool can be applied and adapted to other accountability mechanisms working in global health.

Implementing performance-based financing in peripheral health centres in Mali: what can we learn from it?

Introduction

Numerous sub-Saharan African countries have experimented with performance-based financing (PBF) with the goal of improving health system performance. To date, few articles have examined the implementation of this type of complex intervention in Francophone West Africa. This qualitative research aims to understand the process of implementing a PBF pilot project in Mali's Koulikoro region.

Method

We conducted a contrasted multiple case study of performance in 12 community health centres in three districts. We collected 161 semi-structured interviews, 69 informal interviews and 96 non-participant observation sessions. Data collection and analysis were guided by the Consolidated Framework for Implementation Research adapted to the research topic and local context.

Results

Our analysis revealed that the internal context of the PBF implementation played a key role in the process. High-performing centres exercised leadership and commitment more strongly than low-performing ones. These two characteristics were associated with taking initiatives to promote PBF implementation and strengthening team spirit. Information regarding the intervention was best appropriated by qualified health professionals. However, the limited duration of the implementation did not allow for the emergence of networks or champions. The enthusiasm initially generated by PBF quickly dissipated, mainly due to delays in the implementation schedule and the payment modalities.

Conclusion

PBF is a complex intervention in which many actors intervene in diverse contexts. The initial level of performance and the internal and external contexts of primary healthcare facilities influence the implementation of PBF. Future work in this area would benefit from an interdisciplinary approach combining public health and anthropology to better understand such an intervention. The deductive–inductive approach must be the stepping-stone of such a methodological approach.

Understanding efficiency and the effect of pay-for-performance across health facilities in Tanzania

BACKGROUND

Ensuring efficient use and allocation of limited resources is crucial to achieving the UHC goal. Performance-based financing that provides financial incentives for health providers reaching predefined targets would be expected to enhance technical efficiency across facilities by promoting an output-oriented payment system. However, there is no study which has systematically assessed efficiency scores across facilities before and after the introduction of pay-for-performance (P4P). This paper seeks to fill this knowledge gap.

METHODS

We used data of P4P evaluation related to healthcare inputs (staff, equipment, medicines) and outputs (outpatient consultations and institutional deliveries) from 75 health facilities implementing P4P in Pwani region, and 75 from comparison districts in Tanzania. We measured technical efficiency using Data Envelopment Analysis and obtained efficiency scores across facilities before and after P4P scheme. We analysed which factors influence technical efficiency by regressing the efficiency scores over a number of contextual factors. We also tested the impact of P4P on efficiency through a difference-in-differences regression analysis.

RESULTS

The overall technical efficiency scores ranged between 0.40 and 0.65 for hospitals and health centres, and around 0.20 for dispensaries. Only 21% of hospitals and health centres were efficient when outpatient consultations and deliveries were considered as output, and <3% out of all facilities were efficient when outpatient consultations only were considered as outputs. Higher efficiency scores were significantly associated with the level of care (hospital and health centre) and wealthier catchment populations. Despite no evidence of P4P effect on efficiency on average, P4P might have improved efficiency marginally among public facilities.

CONCLUSION

Most facilities were not operating at their full capacity indicating potential for improving resource usage. A better understanding of the production process at the facility level and of how different healthcare financing reforms affects efficiency is needed. Effective reforms should improve inputs, outputs but also efficiency.

How are pay-for-performance schemes in healthcare designed in low- and middle-income countries? Typology and systematic literature review

BACKGROUND

Pay for performance (P4P) schemes provide financial incentives to health workers or facilities based on the achievement of pre-specified performance targets and have been widely implemented in health systems across low and middle-income countries (LMICs). The growing evidence base on P4P highlights that (i) there is substantial variation in the effect of P4P schemes on outcomes and (ii) there appears to be heterogeneity in incentive design. Even though scheme design is likely a key determinant of scheme effectiveness, we currently lack systematic evidence on how P4P schemes are designed in LMICs.

METHODS

We develop a typology to classify the design of P4P schemes in LMICs, which highlights different design features that are a priori likely to affect the behaviour of incentivised actors. We then use results from a systematic literature review to classify and describe the design of P4P schemes that have been evaluated in LMICs. To capture academic publications, Medline, Embase, and EconLit databases were searched. To include relevant grey literature, Google Scholar, Emerald Insight, and websites of the World Bank, WHO, Cordaid, Norad, DfID, USAID and PEPFAR were searched.

RESULTS

We identify 41 different P4P schemes implemented in 29 LMICs. We find that there is substantial heterogeneity in the design of P4P schemes in LMICs and pinpoint precisely how scheme design varies across settings. Our results also highlight that incentive design is not adequately being reported on in the literature - with many studies failing to report key design features.

CONCLUSIONS

We encourage authors to make a greater effort to report information on P4P scheme design in the future and suggest using the typology laid out in this paper as a starting point.

Effects in spite of tough constraints - A theory of change based investigation of contextual and implementation factors affecting the results of a performance based financing scheme extended to malnutrition in Burundi

BACKGROUND

From January 2015 to December 2016, the health authorities in Burundi piloted the inclusion of child nutrition services into the pre-existing performance-based financing free health care policy (PBF-FHC). An impact evaluation, focused on health centres, found positive effects both in terms of volume of services and quality of care. To some extent, this result is puzzling given the harshness of the contextual constraints related to the fragile setting.

METHODS

With a multi-methods approach, we explored how contextual and implementation constraints interacted with the pre-identified tracks of effect transmission embodied in the intervention. For our analysis, we used a hypothetical Theory of Change (ToC) that mapped a set of seven tracks through which the intervention might develop positive effects for children suffering from malnutrition. We built our analysis on (1) findings from the facility surveys and (2) extra qualitative data (logbooks, interviews and operational document reviews).

FINDINGS

Our results suggest that six constraints have weighted upon the intervention: (1) initial low skills of health workers; (2) unavailability of resources (including nutritional dietary inputs and equipment); (3) payment delays; (4) suboptimal information; (5) restrictions on autonomy; and (6) low intensity of supervision. Together, they have affected the intensity of the intervention, especially during its first year. From our analysis of the ToC, we noted that the positive effects largely occurred as a result of the incentive and information tracks. Qualitative data suggests that health centres have circumvented the many constraints by relying on a community-based recruitment strategy and a better management of inputs at the level of the facility and the patient himself.

CONCLUSION

Frontline actors have agency: when incentives are right, they take the initiative and find solutions. However, they cannot perform miracles: Burundi needs a holistic societal strategy to resolve the structural problem of child malnutrition.

TRIAL REGISTRATION

Clinical Trials.gov Identifier: NCT02721160; March 2016 (retrospectively registered).

Does payment for performance increase performance inequalities across health providers? A case study of Tanzania

The impact of payment-for-performance (P4P) schemes in the health sector has been documented, but there has been little attention to the distributional effects of P4P across health facilities. We examined the distribution of P4P payouts over time and assessed whether increased service coverage due to P4P differed across facilities in Tanzania. We used two service outcomes that improved due to P4P [facility-based deliveries and provision of antimalarials during antenatal care (ANC)], to also assess whether incentive design matters for performance inequalities. We used data from 150 facilities from intervention and comparison areas in January 2012 and 13 months later. Our primary data were gathered through facility survey and household survey, while data on performance payouts were obtained from the programme administrator. Descriptive inequality measures were used to examine the distribution of payouts across facility subgroups. Difference-in-differences regression analyses were used to identify P4P differential effects on the two service coverage outcomes across facility subgroups. We found that performance payouts were initially higher among higher-level facilities (hospitals and health centres) compared with dispensaries, among facilities with more medical commodities and among facilities serving wealthier populations, but these inequalities declined over time. P4P had greater effects on coverage of institutional deliveries among facilities with low baseline performance, serving middle wealth populations and located in rural areas. P4P effects on antimalarials provision during ANC was similar across facilities. Performance inequalities were influenced by the design of incentives and a range of facility characteristics; however, the nature of the service being targeted is also likely to have affected provider response. Further research is needed to examine in more detail the effects of incentive design on outcomes and researchers should be encouraged to report on design aspects in their evaluations of P4P and systematically monitor and report subgroup effects across providers.

Improvement of the quality payment program by improving data reporting process: an action research

BACKGROUND

Successful implementation of pay-for-quality (P4Q) programs mostly depends upon a valid, timely, and reliable data about quality measures generated by providers, and interpreted by payers. The aim of this study was to establish a data reporting method for P4Q program through an action research.

METHODS

Qualitative method was used to align theory with action through a three-cycle action research. The study was conducted in September 15, 2015 to March 15, 2017, in East-Azerbaijan, Iran. The purposeful sampling was used to select participants. The participants included healthcare providers, staff in district health centers (DHC), experts, and managers in the provincial primary health center (PPHC). Data was collected by interviews, focus group discussions, and expert panels. Content analysis was used to synthesize the data. In each step, decisions about data reporting methods were made through a consensus of expert panel members.

RESULTS

The most important dimensions of data reporting method were data entry and accuracy, data reporting, data analysis and interpretations, the flexibility of method, and training. By establishment of an online data reporting system for the P4Q program, a major improvement was observed in the documentation of performance data, the satisfaction of health care providers and staff (e.g. either in DHCs or PPHC), improvement of the P4Q program and acceptance of the P4Q program by providers. Following the present study, the online system was expanded in Iran's public health system for data collection and estimating the amount of incentive payments in P4Q program. Moreover, more improvements were achieved by linking the system to EMRs and also, providing automated feedback to providers about their own performance.

CONCLUSIONS

A web-based computerized system with the capability of linking medical record and also its ability to provide feedback to healthcare providers was identified as an appropriate method of data reporting in the P4Q program from the viewpoints of participants in this study.

Accountability mechanisms for implementing a health financing option: the case of the basic health care provision fund (BHCPF) in Nigeria

Background

The Nigerian National Health Act proposes a radical shift in health financing in Nigeria through the establishment of a fund – Basic Healthcare Provision Fund, (BHCPF). This Fund is intended to improve the functioning of primary health care in Nigeria. Key stakeholders at national, sub-national and local levels have raised concerns over the management of the BHCPF with respect to the roles of various stakeholders in ensuring accountability for its use, and the readiness of the implementers to manage this fund and achieve its objectives. This study explores the governance and accountability readiness of the different layers of implementation of the Fund; and it contributes to the generation of policy implementation guidelines around governance and accountability for the Fund.

Methods

National, state and LGA level respondents were interviewed using a semi structured tool. Respondents were purposively selected to reflect the different layers of implementation of primary health care and the levels of accountability. Different accountability layers and key stakeholders expected to implement the BHCPF are the Federal government (Federal Ministry of Health, NPHCDA, NHIS, Federal Ministry of Finance); the State government (State Ministry of Health, SPHCB, State Ministry of Finance, Ministry of Local Government); the Local government (Local Government Health Authorities); Health facilities (Health workers, Health facility committees (HFC) and External actors (Development partners and donors, CSOs, Community members).

Results

In general, the strategies for accountability encompass planning mechanisms, strong and transparent monitoring and supervision systems, and systematic reporting at different levels of the healthcare system. Non-state actors, particularly communities, must be empowered and engaged as instruments for ensuring external accountability at lower levels of implementation. New accountability strategies such as result-based or performance-based financing could be very valuable.

Conclusion

The key challenges to accountability identified should be addressed and these included trust, transparency and corruption in the health system, political interference at higher levels of government, poor data management, lack of political commitment from the State in relation to release of funds for health activities, poor motivation, mentorship, monitoring and supervision, weak financial management and accountability systems and weak capacity to implement suggested accountability mechanisms due to political interference with accountability structures.

Understanding for whom, why and in what circumstances payment for performance works in low and middle income countries: protocol for a realist review

Background

Many low and middle income countries (LMIC) are implementing payment for performance (P4P) schemes to strengthen health systems and make progress towards universal health coverage. A number of systematic reviews have considered P4P effectiveness but did not explore how P4P works in different settings to improve outcomes or shed light on pathways or mechanisms of programme effect. This research will undertake a realist review to investigate how, why and in what circumstances P4P leads to intended and unintended outcomes in LMIC.

Methods

Our search was guided by an initial programme theory of mechanisms and involved a systematic search of Medline, Embase, Popline, Business Source Premier, Emerald Insight and EconLit databases for studies on P4P and health in LMIC. Inclusion and exclusion criteria identify literature that is relevant to the initial programme theory and the research questions underpinning the review. Retained evidence will be used to test, revise or refine the programme theory and identify knowledge gaps. The evidence will be interrogated by examining the relationship between context, mechanisms and intended and unintended outcomes to establish what works for who, in which contexts and why.

Discussion

By synthesising current knowledge on how P4P affects health systems to produce outcomes in different contexts and to what extent the programme design affects this, we will inform more effective P4P programmes to strengthen health systems and achieve sustainable service delivery and health impacts.

Mechanisms of implementing public health interventions: a pooled causal mediation analysis of randomised trials

BACKGROUND

The World Health Organization recommends that nations implement evidence-based nutritional guidelines and policies in settings such as schools and childcare services to improve public health nutrition. Understanding the causal mechanism by which implementation strategies exert their effects could enhance guideline implementation. The aim of this study was to assess the mechanisms by which implementation strategies improved schools and childcare services' adherence to nutrition guidelines.

METHODS

We conducted a mechanism evaluation of an aggregated dataset generated from three randomised controlled trials conducted in schools and childcare services in New South Wales, Australia. Each trial examined the impact of implementation strategies that targeted Theoretical Domains Framework constructs including knowledge, skills, professional role and identity, environmental context and resources. We pooled aggregated organisation level data from each trial, including quantitative assessments of the Theoretical Domains Framework constructs, as well as measures of school or childcare nutrition guideline compliance, the primary implementation outcome. We used causal mediation analysis to estimate the average indirect and direct effects of the implementation strategies and assessed the robustness of our findings to varying levels of unmeasured and unknown confounding.

RESULTS

We included 121 schools or childcare services in the pooled analysis: 79 allocated to receive guideline and policy implementation strategies and 42 to usual practice. Overall, the interventions improved compliance (odds ratio = 6.64; 95% CI [2.58 to 19.09]); however, the intervention effect was not mediated by any of the four targeted Theoretical Domains Framework constructs (average causal mediation effects through knowledge = - 0.00 [- 0.05 to 0.04], skills = 0.01 [- 0.02 to 0.07], professional role and identity = 0.00 [- 0.03 to 0.03] and environmental context and resources = 0.00 [- 0.02 to 0.06]). The intervention had no significant effect on the four targeted Theoretical Domains Framework constructs, and the constructs were not associated with school or childcare nutrition guideline compliance. Potentially, this lack of effect could be explained by imprecise measurement of the mediators. Alternatively, it is likely that that the interventions were operating via alternative mechanisms that were not captured by the four Theoretical Domains Framework constructs we explored.

CONCLUSIONS

Even though public health implementation strategies led to meaningful improvements in school or childcare nutrition guideline compliance, these effects were not mediated by key targeted constructs of the Theoretical Domains Framework. Future research should explore the mechanistic role of other Theoretical Domains Framework constructs and evaluate system-level mechanisms informed by an ecological framework.

TRIAL REGISTRATION

All trials were prospectively registered with the Australian New Zealand Clinical Trials Registry ( ACTRN12613000543785 . Registered 15/05/2013; ACTRN12614001148662 . Registered 30/10/2014; ACTRN12615001032549 . Registered 1/10/2015).

Who benefits from increased service utilisation? Examining the distributional effects of payment for performance in Tanzania

BACKGROUND

Payment for performance (P4P) strategies, which provide financial incentives to health workers and/or facilities for reaching pre-defined performance targets, can improve healthcare utilisation and quality. P4P may also reduce inequalities in healthcare use and access by enhancing universal access to care, for example, through reducing the financial barriers to accessing care. However, P4P may also enhance inequalities in healthcare if providers cherry-pick the easier-to-reach patients to meet their performance targets. In this study, we examine the heterogeneity of P4P effects on service utilisation across population subgroups and its implications for inequalities in Tanzania.

METHODS

We used household data from an evaluation of a P4P programme in Tanzania. We surveyed about 3000 households with women who delivered in the last 12 months prior to the interview from seven intervention and four comparison districts in January 2012 and a similar number of households in 13 months later. The household data were used to generate the population subgroups and to measure the incentivised service utilisation outcomes. We focused on two outcomes that improved significantly under the P4P, i.e. institutional delivery rate and the uptake of antimalarials for pregnant women. We used a difference-in-differences linear regression model to estimate the effect of P4P on utilisation outcomes across the different population subgroups.

RESULTS

P4P led to a significant increase in the rate of institutional deliveries among women in poorest and in middle wealth status households, but not among women in least poor households. However, the differential effect was marginally greater among women in the middle wealth households compared to women in the least poor households (p = 0.094). The effect of P4P on institutional deliveries was also significantly higher among women in rural districts compared to women in urban districts (p = 0.028 for differential effect), and among uninsured women than insured women (p = 0.001 for differential effect). The effect of P4P on the uptake of antimalarials was equally distributed across population subgroups.

CONCLUSION

P4P can enhance equitable healthcare access and use especially when the demand-side barriers to access care such as user fees associated with drug purchase due to stock-outs have been reduced.

Twelve months of implementation of health care performance-based financing in Burkina Faso: A qualitative multiple case study

To improve health services' quantity and quality, African countries are increasingly engaging in performance-based financing (PBF) interventions. Studies to understand their implementation in francophone West Africa are rare. This study analysed PBF implementation in Burkina Faso 12 months post-launch in late 2014. The design was a multiple and contrasted case study involving 18 cases (health centres). Empirical data were collected from observations, informal (n = 224) and formal (n = 459) interviews, and documents. Outside the circle of persons trained in PBF, few in the community had knowledge of it. In some health centres, the fact that staff were receiving bonuses was intentionally not announced to populations and community leaders. Most local actors thought PBF was just another project, but the majority appreciated it. There were significant delays in setting up agencies for performance monitoring, auditing, and contracting, as well as in the payment. The first audits led rapidly to coping strategies among health workers and occasionally to some staging beforehand. No community-based audits had yet been done. Distribution of bonuses varied from one centre to another. This study shows the importance of understanding the implementation of public health interventions in Africa and of uncovering coping strategies.

"Scaling-out" evidence-based interventions to new populations or new health care delivery systems

Background

Implementing treatments and interventions with demonstrated effectiveness is critical for improving patient health outcomes at a reduced cost. When an evidence-based intervention (EBI) is implemented with fidelity in a setting that is very similar to the setting wherein it was previously found to be effective, it is reasonable to anticipate similar benefits of that EBI. However, one goal of implementation science is to expand the use of EBIs as broadly as is feasible and appropriate in order to foster the greatest public health impact. When implementing an EBI in a novel setting, or targeting novel populations, one must consider whether there is sufficient justification that the EBI would have similar benefits to those found in earlier trials.

Discussion

In this paper, we introduce a new concept for implementation called “scaling-out” when EBIs are adapted either to new populations or new delivery systems, or both. Using existing external validity theories and multilevel mediation modeling, we provide a logical framework for determining what new empirical evidence is required for an intervention to retain its evidence-based standard in this new context. The motivating questions are whether scale-out can reasonably be expected to produce population-level effectiveness as found in previous studies, and what additional empirical evaluations would be necessary to test for this short of an entirely new effectiveness trial. We present evaluation options for assessing whether scaling-out results in the ultimate health outcome of interest.

Conclusion

In scaling to health or service delivery systems or population/community contexts that are different from the setting where the EBI was originally tested, there are situations where a shorter timeframe of translation is possible. We argue that implementation of an EBI in a moderately different setting or with a different population can sometimes “borrow strength” from evidence of impact in a prior effectiveness trial. The collection of additional empirical data is deemed necessary by the nature and degree of adaptations to the EBI and the context. Our argument in this paper is conceptual, and we propose formal empirical tests of mediational equivalence in a follow-up paper.

Combining Theory-Driven Evaluation and Causal Loop Diagramming for Opening the 'Black Box' of an Intervention in the Health Sector: A Case of Performance-Based Financing in Western Uganda

Increased attention on "complexity" in health systems evaluation has resulted in many different methodological responses. Theory-driven evaluations and systems thinking are two such responses that aim for better understanding of the mechanisms underlying given outcomes. Here, we studied the implementation of a performance-based financing intervention by the Belgian Technical Cooperation in Western Uganda to illustrate a methodological strategy of combining these two approaches. We utilized a systems dynamics tool called causal loop diagramming (CLD) to generate hypotheses feeding into a theory-driven evaluation. Semi-structured interviews were conducted with 30 health workers from two districts (Kasese and Kyenjojo) and with 16 key informants. After CLD, we identified three relevant hypotheses: "success to the successful", "growth and underinvestment", and "supervision conundrum". The first hypothesis leads to increasing improvements in performance, as better performance leads to more incentives, which in turn leads to better performance. The latter two hypotheses point to potential bottlenecks. Thus, the proposed methodological strategy was a useful tool for identifying hypotheses that can inform a theory-driven evaluation. The hypotheses are represented in a comprehensible way while highlighting the underlying assumptions, and are more easily falsifiable than hypotheses identified without using CLD.