Implementation of peer support for people with severe mental health conditions in high-, middle- and low-income-countries: a theory of change approach

BACKGROUND

Stakeholder engagement is essential to the design, implementation and evaluation of complex mental health interventions like peer support. Theory of Change (ToC) is commonly used in global health research to help structure and promote stakeholder engagement throughout the project cycle. Stakeholder insights are especially important in the context of a multi-site trial, in which an intervention may need to be adapted for implementation across very different settings while maintaining fidelity to a core model. This paper describes the development of a ToC for a peer support intervention to be delivered to people with severe mental health conditions in five countries as part of the UPSIDES trial.

METHODS

One hundred thirty-four stakeholders from diverse backgrounds participated in a total of 17 workshops carried out at six UPSIDES implementing sites across high-, middle- and low-income settings (one site each in India, Israel, Uganda and Tanzania; two sites in Germany). The initial ToC maps created by stakeholders at each site were integrated into a cross-site ToC map, which was then revised to incorporate additional insights from the academic literature and updated iteratively through multiple rounds of feedback provided by the implementers.

RESULTS

The final ToC map divides the implementation of the UPSIDES peer support intervention into three main stages: preparation, implementation, and sustainability. The map also identifies three levels of actors involved in peer support: individuals (service users and peer support workers), organisations (and their staff members), and the public. In the UPSIDES trial, the ToC map proved especially helpful in characterising and distinguishing between (a) common features of peer support, (b) shared approaches to implementation and (c) informing adaptations to peer support or implementation to account for contextual differences.

CONCLUSIONS

UPSIDES is the first project to develop a multi-national ToC for a mental health peer support intervention. Stakeholder engagement in the ToC process helped to improve the cultural and contextual appropriateness of a complex intervention and ensure equivalence across sites for the purposes of a multi-site trial. It may serve as a blueprint for implementing similar interventions with a focus on recovery and social inclusion among people with mental ill-health across diverse settings.

TRIAL REGISTRATION

ISRCTN26008944 (Registration Date: 30/10/2019).

Developing a theory of change model for a learning and innovation network: A qualitative study

AIM

The aim of this study is to further develop a preliminary framework into a model that can translate mechanisms into output and impact, based on the views of those working in practice and the relations between the mechanisms: a model that can inform practitioners and organizations on what has to be in place to shape a learning and innovating environment in nursing.

BACKGROUND

A Learning and Innovation Network (LIN) is a network of healthcare professionals, students and education representatives who come together to be part of a nursing community to integrate education, research and practice to contribute to quality of care. In a previous study a preliminary framework was developed through a concept analysis based on publications. The preliminary framework describes input, throughput and output factors in a linear model that does not explain what the components entail in practice and how the components work together.

DESIGN

Focus groups.

METHODS

We designed a Theory of Change (ToC) in four phases. This was based on a focus group interview with lecturer practitioners (Phase 1); a first concept ToC based on thematic analysis of the focus group interview (Phase 2); three paired interviews where the ToC was presented to other lecturer practitioners to complement and verify the ToC model (Phase 3); and adjustment of the model based on the feedback of phase 3 (Phase 4).

RESULTS

The developed ToC model describes important preconditions that have to be in place to start a LIN: a shared vision, a facilitating support system and a diversity of participants who are open to change. It describes the mechanisms by which a wide range of activities can lead to an improvement of the quality of care through collaboration between practice, education and research by working, learning, performing practice based research and implementing new methods together.

CONCLUSION

This study gives a comprehensive overview of the concept of the 'Learning and Innovation Network' (LIN); how the activities in the LIN can lead to impact; and under what conditions. Previously published findings supported elements of the ToC model. The overarching ToC model and the detailed appendix offer a theoretical and practice-based model for practitioners, managers and policy makers.

Equity-related 'knots' in theory of change development: Conceptualization and case illustrations

Theory of change (ToC) is an approach widely used to guide planning, implementing, and evaluating change initiatives. While there is substantial guidance, there has been little attention on equity within ToC research and practice. We propose and illustrate the metaphor of 'knots' to frame practical and ethical challenges that arise when centering equity within ToC processes. Drawing on our experiences using a ToC approach in two case examples, we identify and illustrate five equity-related knots: (a) clarify root causes, pathways, and success; (b) facilitate participation across power and perspective differences; (c) integrate research evidence and practitioner knowledge; (d) represent complex change visually; and (e) creatively navigate constraints. We show why framing these as knots can help practitioners make wise judgments within the circumstances and close with recommendations for including knots in ToC processes, reporting, and guidance.

Using theory of change to co-create a programme theory for a telerehabilitation intervention for pain management in people with haemophilia

BACKGROUND

Improved approaches for chronic pain management are a clinical and research priority for people with haemophilia (PWH). Involving people with lived experience in the design of a complex rehabilitation intervention strengthens the credibility and plausibility of the intervention, particularly in relation to rare disorders. Here we describe using a 'Theory of Change' (ToC) dialogue-based stakeholder process to create a programme theory for a telerehabilitation intervention.

METHODS

An online workshop was convened and stakeholders received a briefing document in advance. Five stakeholders took part (3 PWH and 2 physiotherapists). At the workshop the group first agreed the overall aim of the intervention. Discussions then identified the resources, activities, barriers and enablers needed to achieve this outcome. All discussions were recorded and annotated by the workshop moderator. Behaviour change techniques were mapped for inclusion in the theory.

RESULTS

A programme theory and narrative report were produced. All stakeholders reviewed these for clarity and to ensure a true reflection of the workshop discussions. Agreement was based on how meaningful, well-defined, do-able, plausible, credible, and testable each component was. Stakeholders highlighted the importance of issues unique to PWH. Key components included the need for physiotherapists to be knowledgeable of the condition, a range of exercises that were inclusive of all abilities, and the need for people to feel safe and supported whilst taking part.

CONCLUSIONS

Co-developed theory based approaches to intervention design offer an inclusive and transparent way to develop novel and meaningful interventions for people with complex health conditions. The ToC is wholly transparent in its design and content. Together with the identified behaviour change techniques, the theory informs the protocol for a feasibility study evaluating a telerehabilitation intervention. Importantly, it allows the opportunity to revise, adapt and improve the programme theory for further implementation and evaluation.

Implementation matters: assessing the effectiveness and sustainment of an obstetric triage program at a high-volume facility in Ghana

BACKGROUND

Maternal mortality remains stubbornly high in Ghana. Current national efforts are focused on improving the quality of care offered in health facilities. Obstetric triage is one intervention that has been proposed to improve the timeliness and appropriateness of care, two key elements of quality. In this study, we describe and evaluate a theory-based implementation approach to introduce obstetric triage into Tema General Hospital, a high-volume maternity hospital in Greater Accra, that blends concepts from implementation science and quality improvement. This implementation project was a first attempt to scale this intervention into a new facility, following initial development in the Greater Accra Regional Hospital (formerly Ridge Hospital) in Accra.

METHODS

This was a retrospective mixed-methods evaluation of two stages of implementation: active implementation and sustainment. We triangulated monitoring data captured during active implementation with clinical outcome data (timeliness of first assessment, accuracy of diagnosis, and appropriateness of care plan) from direct observation or patient obstetric triage assessment forms at baseline, at the completion of the active implementation stage, and following a 12-month "washout" period with no contact between hospital staff and the purveyor organization. Finally, we assessed embeddedness of the new triage procedures using the NoMad, a quantitative assessment of constructs from normalization process theory (NPT).

RESULTS

Patient waiting time decreased substantially during the study. At baseline, the median arrival-to-assessment waiting time was 70.5 min (IQR: 30.0-443.0 min). Waiting time decreased to 6.0 min (IQR: 3.0-15.0 min) following active implementation and to 5.0 min (IQR: 2.0-10.0 min) during the sustainment period. Accuracy of diagnosis was high at the end of active implementation (75.7% correct) and improved during the sustainment period (to 77.9%). The appropriateness of care plans also improved during the sustainment period (from 66.0 to 78.9%). Per NoMad data, hospital staff generally perceive obstetric triage to be well integrated into the facility.

CONCLUSIONS

This theory-based implementation approach proved to be successful in introducing a novel obstetric triage concept to a busy high-volume hospital, despite resource constraints and a short implementation window. Results proved long-lasting, suggesting this approach has high potential for engendering sustainability in other facilities as well. Our approach will be useful to other initiatives that aim to utilize program data to create and test implementation theories.

Outcome assessment of a complex mental health intervention in the workplace. Results from the MENTUPP pilot study

OBJECTIVE

Multicomponent interventions are recommendable to achieve the greatest mental health benefits, but are difficult to evaluate due to their complexity. Defining long-term outcomes, arising from a Theory of Change (ToC) and testing them in a pilot phase, is a useful approach to plan a comprehensive and meaningful evaluation later on. This article reports on the pilot results of an outcome evaluation of a complex mental health intervention and examines whether appropriate evaluation measures and indicators have been selected ahead of a clustered randomised control trial (cRCT).

METHODS

The MENTUPP pilot is an evidence-based intervention for Small and Medium Enterprises (SMEs) active in three work sectors and nine countries. Based on our ToC, we selected the MENTUPP long-term outcomes, which are reported in this article, are measured with seven validated scales assessing mental wellbeing, burnout, depression, anxiety, stigma towards depression and anxiety, absenteeism and presenteeism. The pilot MENTUPP intervention assessment took place at baseline and at 6 months follow-up.

RESULTS

In total, 25 SMEs were recruited in the MENTUPP pilot and 346 participants completed the validated scales at baseline and 96 at follow-up. Three long-term outcomes significantly improved at follow-up (p < 0.05): mental wellbeing, symptoms of anxiety, and personal stigmatising attitudes towards depression and anxiety.

CONCLUSIONS

The results of this outcome evaluation suggest that MENTUPP has the potential to strengthen employees' wellbeing and decrease anxiety symptoms and stigmatising attitudes. Additionally, this study demonstrates the utility of conducting pilot workplace interventions to assess whether appropriate measures and indicators have been selected. Based on the results, the intervention and the evaluation strategy have been optimised.

Improving paramedic responses for patients dying at home: a theory of change-based approach

BACKGROUND

Paramedics are increasingly being called to attend patients dying from advanced incurable conditions. However, confidence to deal with such calls varies, with many feeling relatively unskilled in this aspect of their role. A number of interventions have been piloted to improve their skills in end-of-life care (EoLC) but without a fully specified theoretical model. Theory of Change models can provide theoretical and testable links from intervention activities to proposed long-term outcomes and indicate the areas for assessment of effectiveness. This study aimed to develop an intervention for improving paramedic EoLC for patients in the community.

METHODS

A Theory of Change approach was used as the overarching theoretical framework for developing an intervention to improve paramedic end-of-life skills. Nine stakeholders - including specialist community paramedics, ambulance call handlers and palliative care specialists - were recruited to five consecutive online workshops, ranging between 60 and 90 min. Each workshop had 2-3 facilitators. Over multiple workshops, stakeholders decided on the desired impact, short- and long-term outcomes, and possible interventions. During and between these workshops a Theory of Change model was created, with the components shared with stakeholders.

RESULTS

The stakeholders agreed the desired impact was to provide consistent, holistic, patient-centred, and effective EoLC. Four potential long-term outcomes were suggested: (1) increased use of anticipatory and regular end-of-life medications; (2) reduced end-of-life clinical and medication errors; (3) reduced unnecessary hospitalisations; (4) increased concordance between patient preferred and actual place of death. Key interventions focused on providing immediate information on what to do in such situations including: appraising the situation, developing an algorithm for a treatment plan (including whether or not to convey to hospital) and how to identify ongoing support in the community.

CONCLUSIONS

A Theory of Change approach was effective at identifying impact, outcomes, and the important features of an end-of-life intervention for paramedics. This study identified the need for paramedics to have immediate access to information and resources to support EoLC, which the workshop stakeholders are now seeking to develop as an intervention.

Implementing Interventions to Address Gender and Power Inequalities in Early Adolescence: Utilizing a Theory of Change to Assess Conditions for Success

PURPOSE

To create a set of criteria to assess facilitators and barriers to implementation among gender transformative interventions that target very young adolescents (VYAs) across different cultural settings.

METHODS

Interventionists and researchers involved in the Global Early Adolescent Study created a Theory of Change (ToC) based on summarizing intervention components from five different gender transformative intervention curricula. Embedded within the ToC is a set of criteria labeled, 'Conditions of Success' which were developed to illustrate that change cannot happen unless interventions are implemented successfully. To test the feasibility of these criteria, implementation data collected across the five interventions in Global Early Adolescent Study were mapped onto the 'Conditions for Success' criteria and used to identify common facilitators and barriers to implementation.

RESULTS

Using the 'Conditions for Success' criteria, we found that gender transformative interventions targeting VYAs were most challenged in meeting program delivery and facilitation conditions and needed to build more multisectoral support to shift rigid gender norms. Parents and caregivers also needed to be engaged in the program either as a separate target population or as codesigners and implementers for the interventions.

DISCUSSION

The Conditions for Success criteria provide a useful framework for assessing facilitators and barriers to implementation among gender transformative interventions for VYAs. Additional research is underway to examine whether interventions that meet more conditions of success result in greater program impact, which will be used to further refine the overall ToC.

Enhancing the quality of psychological interventions delivered by telephone in mental health services: increasing the likelihood of successful implementation using a theory of change

BACKGROUND

The implementation of new and complex interventions in mental health settings can be challenging. This paper explores the use of a Theory of Change (ToC) for intervention design and evaluation to increase the likelihood of complex interventions being effective, sustainable, and scalable. Our intervention was developed to enhance the quality of psychological interventions delivered by telephone in primary care mental health services.

METHODS

A ToC represents how our designed quality improvement intervention targeting changes at service, practitioner, and patient levels was expected to improve engagement in, and the quality of, telephone-delivered psychological therapies. The intervention was evaluated following implementation in a feasibility study within three NHS Talking Therapies services through a qualitative research design incorporating semi-structured interviews and a focus group with key stakeholders (patients, practitioners, and service leads) (N = 15). Data were analysed using the Consolidated Framework for Implementation Research (CFIR) and the ToC was examined and modified accordingly following the findings.

RESULTS

CFIR analysis highlighted a set of challenges encountered during the implementation of our service quality improvement telephone intervention that appeared to have weakened the contribution to the change mechanisms set out by the initial ToC. Findings informed changes to the intervention and refinement of the ToC and are expected to increase the likelihood of successful future implementation in a randomised controlled trial.

CONCLUSIONS

Four key recommendations that could help to optimise implementation of a complex intervention involving different key stakeholder groups in any setting were identified. These include: 1-developing a good understanding of the intervention and its value among those receiving the intervention; 2-maximising engagement from key stakeholders; 3-ensuring clear planning and communication of implementation goals; and 4-encouraging the use of strategies to monitor implementation progress.

Assumptions in theories of change

This paper argues that assumptions in a theory of change are the causal connections, events, and conditions that need to be realized for the intervention to work. Using an example of an intervention aimed at improving educational outcomes for girls in a conservative region, two kinds of assumptions are discussed: cause-effect assumptions and causal-link assumptions. Implications for the use of theories of change, including their use in setting causality and the utility of evidence in argument for learning about and testing assumptions are also discussed. The need for an explicit description of what is meant by the term 'assumptions' in association with a theory of change is also highlighted.

Testing theory of change assumptions of health behavior change interventions: A blended approach exploring local contexts

This paper used a blended approach that involves multiple techniques to, first, test a set of assumptions around a health behavior change communication intervention theory of change (ToC) and, second, surface some unidentified assumptions involving the local context. The intervention was integrated with women's self-help groups (SHGs) in Uttar Pradesh, India. The key assumption tested in this paper was the linkage between SHG membership, program exposure, and maternal, newborn, and child health practices. Learnings were substantiated through empirical investigations, including structural equation modeling and mediation analysis, as well as 'co-learning' workshops within the community. The workshops aimed to capture and interpret the heterogeneity of local contexts through deep dialogs with the community and program implementers at various levels. Statistical analyses indicated a significant association between the amount of women's program exposure and their health practices. SHG membership was shown to affect maternal health practices; however, it did not have a direct effect on neonatal or child health practices. The 'co-learning' workshops revealed crucial aspects, such as prevailing socio-cultural norms, which prevented pregnant or recently delivered women from participating in SHG meetings. This paper encourages evaluators to work with the community to interpret and co-construct meaning in unpacking the contextual forces that seldom appear in the program ToC.

The theory of change and realistic evaluation applied to the evaluation of a transport intervention: The case of the Nottingham Workplace Parking Levy

This paper focuses on the application of evaluation based on a Theory of Change Approach, including elements of Realistic Evaluation to a transport intervention; the UK's first Workplace Parking Levy (WPL) introduced in 2012 in the City of Nottingham. A WPL places a charge off-street parking provided by employers. The scheme acts as a transport demand management measure with the revenue hypothecated for funding transport improvements. The WPL and the measures that it funds thus form an integrated package aimed at achieving social, economic and environmental benefits. The approach afforded a robust evaluation of the outcomes and impacts of a WPL package of measures. Based on this case study it is able to conclude that this evaluation approach is an appropriate framework for evaluating public sector interventions in general and transport interventions specifically and recommend as to how the methodology may be refined for future transport evaluations.

Using grounded theory to develop a theory of change for a violence intervention program

Hospital-based violence intervention programs (HVIP) are critical to interrupting the cycle of violence. These interventions are considered "complex" in that they have many mechanisms of change and related outcomes. Few HVIPs clearly identify the underlying mechanisms of intervention and explicitly link those with key outcomes however, limiting the field's ability to know what works best and for whom. To develop a program theory of change for these "complex interventions," a non-linear, robust methodology that is grounded in the lived experience of those delivering and receiving services is needed. To aid researchers, evaluators, students, and program developers, we describe the use of Grounded Theory as a methodology to enhance the development of complex interventions, illuminating a non-linear approach that engages key stakeholders. To illustrate application, we describe a case example of The Antifragility Initiative, a HVIP in Cleveland, Ohio. The development of the program theory of change was conducted in four phases: (1) review of existing program documents, (2) semi-structured interviews with program developers (n = 6), (3) a focus group with program stakeholders (n = 8), and (4) interviews with caregivers and youth (n = 8). Each phase informed the next and culminated in a theoretical narrative and visual model of the Antifragility Initiative. Together, the theoretical narrative and visual model identify the underlying mechanisms that can promote change by the program.

Learning from experiences of evaluators implementing theory-driven evaluations in diverse settings: Building on the contributions of John Mayne

The papers in this volume grapple with various issues related to the use of theories of change in program evaluation. This introductory paper reviews some of the critical challenges that arise in developing and learning from theory-driven evaluations. These challenges include the relationship between theories of change and ecologies of evidence, the need for epistemic fluency in learning, and coming to terms with the initial incompleteness of knowledge in program mechanisms. The nine papers that follow, which represent a geographically diverse set of evaluations and evaluators (including Scotland, India, Canada, USA), help develop these and other themes. This volume of papers also serves as a celebration of the work of John Mayne, one of the foremost theory-driven evaluators of the last few decades. John passed away in December of 2020. This volume is intended to honor his legacy while also identifying challenging issues that call for further development.

Interrogating assumptions about the relationship between service providers and recipients: Learning from a new service for survivors of In Care Abuse

Understanding the dynamic relationship between service providers and the people who use their services is key to effective evaluation. This paper presents a practical approach to embedded evaluation that can be used by services to interrogate assumptions about relationships. The approach includes a simple framework for developing theories of change that centres relational aspects of the change process. This framework is complemented by a structured approach to surfacing risks and assumptions. Using the example of the evaluation of Future Pathways, a new and ground breaking service provided to people who experienced abuse or neglect as children In Care in Scotland, the paper describes how this approach was used in practice. Focusing in on assumptions identified by the service around trust, the paper describes the process undertaken to interrogate these assumptions through data collection with people using the service and staff. This process led to rich learning to support the development of the service as well as the identification of new elliptical assumptions. The paper concludes by sharing reflections on the learning from this work for the wider evaluation community highlighting the need for evaluators to take a relational approach to interrogating assumptions about relationships.

Towards an evolutionary approach to learning from assumptions: Lessons from the evaluation of Dancing with Parkinson's

This paper highlights how learnings from exploring assumptions can be strengthened by taking an evolutionary approach to theory building and analysis. We discuss theory-driven evaluation applied to a community-based intervention implemented by Dancing With Parkinson's in Toronto, Canada, targeting Parkinson's disease (PD), a neurodegenerative condition affecting movement. A major gap in the literature is understanding the mechanisms by which dance might make a difference in the daily lives of people living with PD. This study was an early exploratory evaluation to better understand mechanisms and short-term outcomes. Conventional thinking generally favors "permanent" over "transitory" changes, and "long-term" over "short-term" effects. Yet, for people living with degenerative conditions (and also people experiencing chronic pain and other chronic symptoms), transitory and short-term changes may be highly valued and welcomed relief. In order to study and link multiple longitudinal events to explore key linkages in the theory of change, we piloted the use of diaries, with brief entries filled out daily by participants. The aim was to better understand the short-term experiences of participants using their daily routines as a means of learning about potential mechanisms, what matters to participants, and to see if small effects could be observed on days when participants danced versus days when they did not dance and also longitudinally over several months. Our initial theoretical stance began with a view of dance as exercise and the well-established benefits of exercise; yet, we explored through the diary data collected, as well as client interviews and literature review, potential other mechanisms of dancing (such as group interaction, touch, stimulation by the music, and esthetics including "feeling lovely"). This paper does not develop a full, comprehensive theory of dance but moves towards a more comprehensive view that locates dance within the routine activities of participants' daily lives. We argue that given the challenges of evaluating complex interventions comprising multiple, interacting components, there is a need for an evolutionary learning process to understand heterogeneities in mechanisms -- what works for whom -- when faced with knowledge incompleteness in the theory of change.

Surfacing, as well as testing, "elliptical assumptions" in a theory of change: Principled discovery

While the testing of known assumptions in a theory of change is important, so too is the discovery or surfacing of previously unrecognized assumptions. This paper describes and illustrates the surfacing of "elliptical assumptions," which involve the unknown ingredients that are necessary for a program to be effective. Identifying the ingredients of program success is important for several reasons, which include (a) guiding the development of an improved theory of change, which in turn can guide program improvement, and (b) informing efforts to transfer the program to other settings and populations. However, when an observed pattern, such as differential program effects, points to the existence of a previously unidentified but important ingredient, this might be a just-so story, a seemingly compelling but inaccurate account. Accordingly, the testing of previously unidentified elliptical assumptions is recommended and illustrated.

Back to the future: Are we trapped in our past?

This paper is a reflection on how evaluators could approach theories of change through a foresight lens. It explores the role of assumptions and especially anticipatory assumptions in how we design our theories of change. It argues for a more open approach, a transdisciplinary approach to the multiple knowledges we bring to bear. It goes on to argue that if we do not build and use our imaginations to think differently about the future than the past, as evaluators we risk being trapped in findings and recommendations that assume continuity in a highly discontinuous world.

Theory-based evaluation of the impact of Master's programmes in development studies: Insights from a mixed-methods and multicultural alumni action research project

To understand the impact of the international Master's programmes offered at the Institute of Development Policy (University of Antwerp), a theory-based evaluation was undertaken. In the first phase, a Theory of Change (ToC) was elaborated, distinguishing between three levels of impact (individual, organisational and societal), four learning dimensions (knowledge, skills, attitudes and networks) and five implicit pathways (change agent, social network, widening access, academic diversity, international understanding). Given the multifaceted and vague nature of the 'impact' concept under study, we selected an international, gender-balanced, multi-sectoral team of alumni researchers who fostered inclusiveness of different perspectives, at the same time capitalising on their depth of understanding, having gone through the study experience themselves. A mixed-methods approach was adopted to validate the ToC, combining a Most Significant Change approach, categorisation and text analysis of 101 alumni impact stories. Our findings demonstrate the importance of the three levels of impact and four learning dimensions in capturing graduate impact. While the impact stories confirmed the dominant 'change agent' pathway, they also hinted at the importance of hybrid complementary configurations of pathways to fully grasp how impact materialises.

Stakeholder engagement as a strategy to enhance palliative care involvement in intensive care units: A theory of change approach

BACKGROUND

Adult patients admitted to intensive care units in the terminal phase experience high symptom burden, increased costs, and diminished quality of dying. There is limited literature on palliative care engagement in ICU, especially in lower-middle-income countries. This study explores a strategy to enhance palliative care engagement in ICU through a stakeholder participatory approach.

METHODS

Theory of Change approach was used to develop a hypothetical causal pathway for palliative care integration into ICUs in India. Four facilitated workshops and fifteen research team meetings were conducted virtually over three months. Thirteen stakeholders were purposively chosen, and three facilitators conducted the workshops. Data included workshop discussion transcripts, online chat box comments, and team meeting minutes. These were collected, analysed and represented as theory of change map.

RESULTS

The desired impact of palliative care integration was good death. Potential long-term outcomes identified were fewer deaths in ICUs, discharge against medical advice, and inappropriate admissions; increased referrals to palliative care; and improved patient and family satisfaction. Twelve preconditions were identified, and eleven key interventions were developed. Five overarching assumptions related to contextual factors influencing the outcomes of interventions.

CONCLUSION

Theory of change framework facilitated the identification of proposed mechanisms and interventions underpinning palliative care integration in ICUs.

From here to Utopia: Theories of Change in Nonideal Animal Ethics

Animal ethics has often been criticized for an overreliance on "ideal" or even "utopian" theorizing. In this article, I recognize this problem, but argue that the "nonideal theory" which critics have offered in response is still insufficient to make animal ethics action-guiding. I argue that in order for animal ethics to be action-guiding, it must consider agent-centered theories of change detailing how an ideally just human-animal coexistence can and should be brought about. I lay out desiderata that such a theory of change should suffice so as to be helpful in guiding action. Specifically, a theory of change should determine (1) who needs to do what in order for ideal justice to be achieved in the long run, (2) who should be expected to refuse compliance and how they should be moved to comply, and (3) why specific intermediate steps are necessary. I show how previous "nonideal" contributions, though helpful in other ways, are insufficiently determinate on these points and I sketch a (still somewhat utopian) theory of change for one specific context. This brings animal ethics a crucial step closer to being action-guiding in the real world.

Synthesizing frameworks and tools to develop a plan for evaluating an online data utilization curriculum for teachers

This paper presents an overview of how a plan was developed for evaluating Develop Your Data Mindset: Essentials of Educational Data Use, an online curriculum intended to improve in-service and pre-service educators' knowledge and skills relevant to data utilization. Five categories of assessment, a program evaluation framework, logic model, theory of change, project plan, metrics spreadsheet, and the A+ Inquiry model are synthesized to demonstrate how they supported the development of an evaluation plan. Types of evidence are described to provide an overview of the need for improved instructional supports relevant to educator data utilization, how the activities to develop and implement an online curriculum are appropriate to meet the need, how well the activities are implemented as planned, and the extent to which intended outcomes of the curriculum are achieved. Applying the methods outlined in this paper yielded a plan to help ensure the curriculum would be assessed through a comprehensive evaluation lens. Limitations are acknowledged. Methods described in this paper could be adapted to support program evaluation planning and implementation beyond the scope of this project.

Using a theory of change in monitoring, evaluating and steering scale-up of a district-level health management strengthening intervention in Ghana, Malawi, and Uganda - lessons from the PERFORM2Scale consortium

Background

Since 2017, PERFORM2Scale, a research consortium with partners from seven countries in Africa and Europe, has steered the implementation and scale-up of a district-level health management strengthening intervention in Ghana, Malawi and Uganda. This article presents PERFORM2Scale’s theory of change (ToC) and reflections upon and adaptations of the ToC over time. The article aims to contribute to understanding the benefits and challenges of using a ToC-based approach for monitoring and evaluating the scale-up of health system strengthening interventions, because there is limited documentation of this in the literature.

Methods

The consortium held annual ToC reflections that entailed multiple participatory methods, including individual scoring exercises, country and consortium-wide group discussions and visualizations. The reflections were captured in detailed annual reports, on which this article is based.

Results

The PERFORM2Scale ToC describes how the management strengthening intervention, which targets district health management teams, was expected to improve health workforce performance and service delivery at scale, and which assumptions were instrumental to track over time. The annual ToC reflections proved valuable in gaining a nuanced understanding of how change did (and did not) happen. This helped in strategizing on actions to further steer the scale-up the intervention. It also led to adaptations of the ToC over time. Based on the annual reflections, these actions and adaptations related to: assessing the scalability of the intervention, documentation and dissemination of evidence about the effects of the intervention, understanding power relationships between key stakeholders, the importance of developing and monitoring a scale-up strategy and identification of opportunities to integrate (parts of) the intervention into existing structures and strategies.

Conclusions

PERFORM2Scale’s experience provides lessons for using ToCs to monitor and evaluate the scale-up of health system strengthening interventions. ToCs can help in establishing a common vision on intervention scale-up. ToC-based approaches should include a variety of stakeholders and require their continued commitment to reflection and learning on intervention implementation and scale-up. ToC-based approaches can help in adapting interventions as well as scale-up processes to be in tune with contextual changes and stakeholders involved, to potentially increase chances for successful scale-up.

Using participatory action research to pilot a model of service user and caregiver involvement in mental health system strengthening in Ethiopian primary healthcare: a case study

Background

Little is known about actual involvement or how to achieve service user and caregiver in mental health systems strengthening in low-and middle-income countries. This study describes the processes and explores involvement experiences of participants in a pilot study of a new model of service user involvement in mental health system strengthening in a rural district in southern Ethiopia.

Methods

We applied a case study design using participatory action research (PAR). The PAR process comprised of three stages, each with iterative activities of plan, act, observe and reflect. Two stakeholder groups, a Research Advisory Group (RAG) and Research Participant Group (RPG), were established and collaborated in the PAR process. Data collection involved process documentation of meetings and activities: attendances, workshop minutes, discussion outputs, reflective notes, participatory observation of sessions, and in-depth interviews with 12 RPG members. We analyzed the process data descriptively. Thematic analysis was used for qualitative data. Triangulation and synthesis of findings was carried out to develop the case study.

Results

The stakeholder groups identified their top research priorities, developed an intervention and action plan and made a public presentation of preliminary findings. Key mechanisms used for inclusive participation included capacity building and bringing together diverse stakeholders, anchoring the study in established strong community involvement structures, and making use of participatory strategies and activities during the PAR process. Four themes were developed about experiences of involvement in PAR: (i) expectations and motivation, (ii) experiences of the dynamics of the PAR process, (iii) perceived impacts of involvement in the PAR process, and (iv) implementation challenges and future directions.

Conclusions

This case study demonstrated the feasibility and acceptability of implementing a complex model of service-user involvement in mental health system strengthening in a resource constrained setting. More needs to be done to embed service-user involvement into routines of the primary healthcare system, alongside sustained support and strengthening multi-stakeholder collaboration at multiple levels.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13033-022-00545-8.

Health and social outcomes in the Housing First model: Testing the theory of change

BACKGROUND

Homelessness continues to grow globally. The Housing First (HF) model offers immediate access to housing and support services without preconditions and has a growing body of evidence documenting its effectiveness at ending homelessness. HF has a robust theory of change that hypothesizes how unique program components (i.e., immediate access to housing, separation of services from housing, client choice, etc.) drive positive social and health changes over time. We advance the understanding of how HF causes client improvement by empirically testing this program's theory of change.

METHODS

Using a unique longitudinal quantitative data from the large Canadian At Home/Chez Soi Housing First trial we used path analysis to test the theory of change for Quality of Life, Crisis related events or service utilization, and Recovery.  Program pathways and health and social outcomes were measured at enrolment, 6-, 12- and 24-months post-enrolment.

FINDINGS

Most hypothesized pathways were confirmed with path analysis.  Confirmed pathways for two outcomes- Quality of Life (QOL) and Recovery - were similar. Health and social consultations at enrolment, health status at 6- and 12-months post enrolment, and social connectedness at 12-months were important predictors of the 24-month outcomes of Quality of Life and Recovery, but not for Crisis related events or service utilization.

INTERPRETATION

This analysis directly responds to recent calls for more empirical evidence about intervention mechanisms. Ensuring linkages to health and social service consultations for clients, supporting clients' engagement with family and community, and enabling clients to improve or maintain good health will drive better longer term client outcomes within Housing First.

FUNDING

Funding Mental Health Commission of Canada.

Identifying key risks to the performance of privately protected areas (PPAs) through theory of change (ToC)

Privately protected areas (PPAs) are internationally considered to be important policy implementation instruments to augment and strengthen protected area networks. However, there has been limited reflection on the performance of PPAs over time. This paper aims to identify key risks to the performance of PPAs as policy implementation instruments through the application of Theory of Change (ToC). Identifying and understanding these risks are important to allow for the evaluation and monitoring of PPA performance. The ToC method was applied to a specific PPA policy instrument namely, private nature reserves (PNRs) in the South African context. The research results produced 29 key assumptions translated into 29 key risks. These risk are critically discussed against existing South African and international literature. To test and refine the risks further it is recommended that they be applied to PPA case studies in different contexts.

"We are not stray leaves blowing about in the wind": exploring the impact of Family Wellbeing empowerment research, 1998-2021

BACKGROUND

An Aboriginal-developed empowerment and social and emotional wellbeing program, known as Family Wellbeing (FWB), has been found to strengthen the protective factors that help Indigenous Australians to deal with the legacy of colonisation and intergenerational trauma. This article reviews the research that has accompanied the implementation of the program, over a 23 year period. The aim is to assess the long-term impact of FWB research and identify the key enablers of research impact and the limitations of the impact assessment exercise. This will inform more comprehensive monitoring of research impact into the future.

METHODS

To assess impact, the study took an implementation science approach, incorporating theory of change and service utilisation frameworks, to create a logic model underpinned by Indigenous research principles. A research impact narrative was developed based on mixed methods analysis of publicly available data on: 1) FWB program participation; 2) research program funding; 3) program outcome evaluation (nine studies); and 4) accounts of research utilisation (seven studies).

RESULTS

Starting from a need for research on empowerment identified by research users, an investment of $2.3 million in research activities over 23 years produced a range of research outputs that evidenced social and emotional wellbeing benefits arising from participation in the FWB program. Accounts of research utilisation confirmed the role of research outputs in educating participants about the program, and thus, facilitating more demand (and funding acquisition) for FWB. Overall research contributed to 5,405 recorded participants accessing the intervention. The key enablers of research impact were; 1) the research was user- and community-driven; 2) a long-term mutually beneficial partnership between research users and researchers; 3) the creation of a body of knowledge that demonstrated the impact of the FWB intervention via different research methods; 4) the universality of the FWB approach which led to widespread application.

CONCLUSIONS

The FWB research impact exercise reinforced the view that assessing research impact is best approached as a "wicked problem" for which there are no easy fixes. It requires flexible, open-ended, collaborative learning-by-doing approaches to build the evidence base over time. Steps and approaches that research groups might take to build the research impact knowledge base within their disciplines are discussed.

Expanding theory-based evaluation: Incorporating value creation in a theory of change

Evaluations of policies and programs often use a theory of change to articulate how the intervention is intended to function and the mechanisms by which it is supposed to generate outcomes. When an evaluation includes cost and efficiency considerations, economic and other concepts can be added to a theory of change to articulate a theory of value creation that articulates the mechanisms by which the intervention should use resources efficiently, effectively and create sufficient value to justify the resource use. This paper introduces some theories of value creation that are often implicit in program designs. Making these theories explicit can support clearer evaluative thinking about value for money - including specification of criteria and standards that are aligned with the theory, methods of inquiry that test the theory, and well-reasoned judgements that answer evaluative questions about value for money. Implications for evaluation practice will be discussed.

Rigour versus the need for evidential diversity

This paper defends the need for evidential diversity and the mix of methods that that can in train require. The focus is on causal claims, especially 'singular' claims about the effects of causes in a specific setting-either what will happen or what has happened. I do so by offering a template that categorises kinds of evidence that can support these claims. The catalogue is generated by considering what needs to happen for a causal process to carry through from putative cause at the start to the targeted effect at the end. The usual call for mixed methods focusses on a single overall claim and argues that we increase certainty by the use of different methods with compensating strengths and weaknesses. My proposals instead focus on the evidence that supports the great many subsidiary claims that must hold if the overall one is to be true. As is typical for singular causal claims, the mix of methods that will generally be required to collect the kinds of evidence I urge will usually have little claim to the kind of rigour that is now widely demanded in evidencing causal claims, especially those for policy/treatment effectiveness. So I begin with an exploration of what seems to be intended by 'rigour' in such discussions, since it is seldom made clear just what makes the favoured methods especially rigorous. I then argue that the emphasis on rigour can be counterproductive. Rigour is often the enemy of evidential diversity, and evidential diversity-lots of it-can make for big improvements in the reliability of singular causal predictions and post hoc evaluations. I illustrate with the paragon of rigour for causal claims, randomised controlled trials (RCTs), rehearsing at some length what they can and cannot do to make it easier to assess the importance of rigour in warranting singular causal claims.

Theory of change models deepen online learning evaluation

Massive Open Online Courses (MOOCs) have gained traction as resources for professional development. This article presents the method that we used to evaluate a professional development MOOC for postdoctoral trainees that was created by a university consortium in the US. Most approaches to evaluating MOOCs focus on analysis of participation, outcomes from course assignments, self-reported learning outcomes, course completion and user pathways through the online content or clickstream data. Few published evaluations describe in detail how learning happens within online courses and the anticipated medium and longer term cognitive and behavioral outcomes on participants. This work aims to guide those who are designing, implementing and evaluating MOOCs through applying theories of change to focus evaluation on the process of learning. This approach can be used as a complement to traditional approaches for evaluating MOOCs. We described how we worked with the MOOC team building the content to develop a theory of change for each module (or lesson) within the MOOC and used the theory of change to guide evaluation of short and medium term participant cognitive, affective and attitudinal, and behavioral outcomes. Finally, we share lessons learned and suggestions for implementing theories of change in both the design and evaluation phases of MOOC development.

Psychosocial interventions for self-harm in low-income and middle-income countries: systematic review and theory of change

PURPOSE

To synthesise the evidence on effectiveness, acceptability and the delivery mechanisms of psychosocial interventions for self-harm in low and middle income countries and to develop a pathway of change specific for self-harm interventions.

METHOD

Studies reporting one or more patient or implementation outcomes of a psychosocial intervention targeting self-harm and conducted in low- and middle-income countries were included. Taxonomy of treatment components and a theory of change map was created using information from the studies.

RESULTS

We identified thirteen studies including nine randomised controlled trials (RCT), three non-RCTs, and a single experimental case design study. A single study using postcard contact and another using cognitive behaviour therapy (CBT) reported a reduction in self-harm attempts. Suicidal ideations were significantly reduced with CBT, volitional help sheets and postcard contact in different studies. Suicide risk assessment, problem solving and self-validation were the most frequently used elements in interventions. Goal-setting was the technique used most commonly. Cultural adaptations of psychotherapies were used in two studies. High attrition rates in psychotherapy trials, limited benefit of the delivery of treatment by non-specialist providers, and variable benefit observed using phone contact as a means to deliver intervention were other important findings.

CONCLUSION

There were no strong positive findings to draw definitive conclusions. Limited availability and evidence for culturally adapted interventions in self-harm, lack of evaluation of task sharing using evidence based interventions as well as a dearth in evaluation and reporting of various intervention delivery models in low- and middle-income countries were major literature gaps.

A commentary: Using a theory-based approach to guide a global programme of FGM/C research: What have we learned about creating actionable research findings?

The 'Evidence to End FGM/C: Research to Help Girls and Women Thrive' programme created a research consortium that brought together African institutions and the world's leading and most experienced researchers in Female genital mutilation/cutting (FGM/C) abandonment. The priority countries for research were Burkina Faso, Egypt, Ethiopia, Kenya, Nigeria, Senegal, Somalia, and Sudan, given their stages of abandonment, the scale of the practice, and socio-cultural variations. Given the ambitious scope of the research component, the consortium outlined a theory of change (ToC). The ToC was designed to facilitate an exploration of the attendant complexities of researching a social practice underpinned with political and religious sensitivities and strong links to individual, family and community identity. Overall, we believe that efforts to encourage FGM/C abandonment would be more effective, more efficient and with greater value for money if a theory of change were used routinely to inform research, programming and evaluation design. This commentary contributes to this goal by documenting our experiences with using a theory of change in designing and implementing research to inform the uptake of research for the design and evaluation of FGM/C interventions. A substantial portfolio of robust evidence is now online. This evidence provides new knowledge, insights, methods, and tools that are valuable to diverse end-users such as researchers, programme implementers, the health, legal and education systems, governments, donors, and the media. A vibrant South-South and South-North collaboration has been built fostering strong working relationships with government ministries and programme/policymakers in the priority countries.

Development of an on-job mentorship programme to improve nursing experience for enhanced patient experience of compassionate care

Background

Evidence suggests improvement in nursing staff satisfaction, competence, and retention after implementation of evidence-based mentorship programmes. When guided by a framework of compassion, mentoring as a caring action can not only build healthy, transformative relationships but a similar behavior is reciprocated to patients which subsequently can drive patient experience of care. However, examples of on-job mentorship programs for nurses in low- and middle-income countries (LMIC) are limited.

Objective

The objective of the study was to develop an on-job nursing mentorship programme using a compassionate framework aimed at improving nurses’ experience and thus enhancing patient experience in a tertiary care hospital in Pakistan.

Methods

Designed as an intervention development study, it was completed between January 2018–December 2019. The programme was developed by a team composed of service and nursing leadership, director patient experience of care and a compassion specialist using a theory of change model. The package followed a series of steps, a) identification of a framework, b) creation of working group c) needs assessment and d) multiple meetings to frame the model followed by implementing the preconditions for roll-out of the programme with the frontline staff.

Results

The eventual outcome was improving the patient’s experience of compassion while the intermediate outcome was to have nurses demonstrate compassionate care. The pre-conditions were identified as: recruitment of staff with appropriate skills for pediatric care, provision of compassionate experience to the frontline nurses by addressing their specific pain points, development of competent head nurses as supervisors and creation of a compassionate culture. To ensure the pre-conditions, various interventions were planned with some implemented through the course of the study while others are in the process of being rolled out. These involved, inclusion of pediatric compassion specific module during orientation of new hires, creation of space to talk about compassionate skills with staff, provision of trainings and mentorship to create competent head nurses, and creating a culture that promoted and recognized compassionate care values.

Conclusion

The approach helped to delineate feasible pathways for an on-job compassionate mentorship programme enhancing routine supervisors' role as facilitators of compassionate care.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-021-00682-4.

Using path analysis to test theory of change: a quantitative process evaluation of the MapSan trial

BACKGROUND

Although theory-driven evaluations should have empirical components, few evaluations of public health interventions quantitatively test the causal model made explicit in the theory of change (ToC). In the context of a shared sanitation trial (MapSan) in Maputo, Mozambique, we report findings of a quantitative process evaluation assessing intervention implementation, participant response and impacts on hypothesised intermediary outcomes on the pathway to trial health outcomes. We examine the utility of path analysis in testing intervention theory using process indicators from the intervention's ToC.

METHODS

Process data were collected through a cross-sectional survey of intervention and control compounds of the MapSan trial > 24-months post-intervention, sampling adult residents and compound leaders. Indicators of implementation fidelity (dose received, reach) and participant response (participant behaviours, intermediary outcomes) were compared between trial arms. The intervention's ToC (formalised post-intervention) was converted to an initial structural model with multiple alternative pathways. Path analysis was conducted through linear structural equation modelling (SEM) and generalised SEM (probit model), using a model trimming process and grouped analysis to identify parsimonious models that explained variation in outcomes, incorporating demographics of respondents and compounds.

RESULTS

Among study compounds, the MapSan intervention was implemented with high fidelity, with a strong participant response in intervention compounds: improvements were made to intermediary outcomes related to sanitation 'quality' - latrine cleanliness, maintenance and privacy - but not to handwashing (presence of soap / soap residue). These outcomes varied by intervention type: single-cabin latrines or multiple-cabin blocks (designed for > 20 users). Path analysis suggested that changes in intermediary outcomes were likely driven by direct effects of intervention facilities, with little contribution from hygiene promotion activities nor core elements expected to mediate change: a compound sanitation committee and maintenance fund. A distinct structural model for two compound size subgroups (≤ 20 members vs. > 20 members) explained differences by intervention type, and other contextual factors influenced specific model parameters.

CONCLUSIONS

While process evaluation found that the MapSan intervention achieved sufficient fidelity and participant response, the path analysis approach applied to test the ToC added to understanding of possible 'mechanisms of change', and has value in disentangling complex intervention pathways.

TRIAL REGISTRATION

MapSan trial registration: NCT02362932 Feb-13-2015.

Nothing so practical as theory: a rapid review of the use of behaviour change theory in family planning interventions involving men and boys

BACKGROUND

There is growing recognition of the need for interventions that effectively involve men and boys to promote family planning behaviours. Evidence suggests that the most effective behavioural interventions in this field are founded on theoretical principles of behaviour change and gender equality. However, there are few evidence syntheses on how theoretical approaches are applied in this context that might guide best practice in intervention development. This review addresses this gap by examining the application and reporting of theories of behaviour change used by family planning interventions involving men and boys.

METHODS

We adopted a systematic rapid review approach, scoping findings of a previously reported evidence and gap map of intervention reviews (covering 2007-2018) and supplementing this with searches of academic databases and grey literature for reviews and additional studies published between 2007 and 2020. Studies were eligible for inclusion if their title, abstract or keywords referred to a psychosocial or behavioural intervention targeting family planning behaviours, involved males in delivery, and detailed their use of an intervention theory of change.

RESULTS

From 941 non-duplicate records identified, 63 were eligible for inclusion. Most records referenced interventions taking place in low- and middle-income countries (65%). There was a range of intervention theories of change reported, typically targeting individual-level behaviours and sometimes comprising several behaviour change theories and strategies. The most commonly identified theories were Social Cognitive Theory, Social Learning Theory, the Theory of Planned Behaviour, and the Information-Motivation-Behaviour Skills (IMB) Model. A minority of records explicitly detailed gender-informed elements within their theory of change.

CONCLUSION

Our findings highlight the range of prevailing theories of change used for family planning interventions involving men and boys, and the considerable variability in their reporting. Programmers and policy makers would be best served by unified reporting and testing of intervention theories of change. There remains a need for consistent reporting of these to better understand how complex interventions that seek to involve men and boys in family planning may lead to behaviour change.

Exploring the mechanism through which a child-friendly storybook addresses barriers to child-participation during HIV care in primary healthcare settings in KwaZulu-Natal, South Africa

Background

Healthcare workers (HCWs) in South Africa widely use job-aids as practical tools to enhance the provision of HIV services, thereby improving patient-provider interactions during the care process. Job-aids are visual support materials that provide appropriate information using graphics and words in a simple and yet effective manner. We explored the mechanism through the KidzAlive Talk tool storybook (Talk tool), a child-centred job-aid for HCWs that facilitates child-participation during HIV consultations in primary healthcare (PHC) clinics implementing the KidzAlive model.

Methods

The study was conducted in PHC clinics across four districts; namely: uMkhanyakude, Zululand, uMgungundlovu, and eThekwini in KwaZulu-Natal (KZN), South Africa. We conducted in-depth interviews with children (n = 30), their primary caregivers (PCGs) (n = 30), and KidzAlive trained and mentored HCWs (n = 20). Data were collected in both English and isiZulu languages through user-specific, structured in-depth interviews. All the interviews were audio-recorded (with participants’ assent and consent, respectively). Data were transcribed verbatim, prior to translating the isiZulu transcripts to English. Translations were done by a member of the research team competent in both languages. Electronic data were imported to NVivo 10 for analysis and subsequently analysed using a thematic analysis method followed by a constant comparative and modified grounded theory analysis method.

Results

The findings identified the following barriers to child-participation: Primary caregiver limiting the child’s involvement due to fear of traumatising them; HCWs’ limited knowledge and skills to deliver child-centred HIV care; childhood developmental stage-related limitations and healthcare institutional paternalism. The Talk tool addresses the above barriers by using simple language and terminology to cater for children at various stages of development; alleviating HCWs’ and PCGs’ fear of possible psychological harm to the child; using storytelling and colourful cartoon illustrations for child edutainment; Being versatile by allowing for multiple utility and tackling institutional paternalism that limit child-involvement in the process of care.

Conclusions

This study provided evidence on how the Talk tool storybook addresses barriers to child-participation in the HIV care process. The evidence generated from this study is compelling enough to recommend the scale-up of this innovation in low-resource settings.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-021-10483-8.

Lessons from a theory of change-driven evaluation of a global mental health funding portfolio

BACKGROUND

Given the underinvestment in global mental health to-date, it is important to consider how best to maximize the impact of existing investments. Theory of Change (ToC) is increasingly attracting the interest of funders seeking to evaluate their own impact. This is one of four papers investigating Grand Challenges Canada's (GCC's) first global mental health research funding portfolio (2012-2016) using a ToC-driven approach.

METHODS

A portfolio-level ToC map was developed through a collaborative process involving GCC grantees and other key stakeholders. Proposed ToC indicators were harmonised with GCC's pre-existing Results-based Management and Accountability Framework to produce a "Core Metrics Framework" of 23 indicators linked to 17 outcomes of the ToC map. For each indicator relevant to their project, the grantee was asked to set a target prior to the start of implementation, then report results at six-month intervals. We used the latest available dataset from all 56 projects in GCC's global mental health funding portfolio to produce a descriptive analysis of projects' characteristics and outcomes related to delivery.

RESULTS

12,999 people were trained to provide services, the majority of whom were lay or other non-specialist health workers. Most projects exceeded their training targets for capacity-building, except for those training lay health workers. Of the 321,933 people screened by GCC-funded projects, 162,915 received treatment. Most projects focused on more than one disorder and exceeded all their targets for screening, diagnosis and treatment. Fewer people than intended were screened for common mental disorders and epilepsy (60% and 54%, respectively), but many more were diagnosed and treated than originally proposed (148% and 174%, respectively). In contrast, the three projects that focused on perinatal depression exceeded screening and diagnosis targets, but only treated 43% of their intended target.

CONCLUSIONS

Under- or over-achievement of targets may reflect operational challenges such as high staff turnover, or challenges in setting appropriate targets, for example due to insufficient epidemiological evidence. Differences in delivery outcomes when disaggregated by disorder suggest that these challenges are not universal. We caution implementers, funders and evaluators from taking a one-size-fits all approach and make several recommendations for how to facilitate more in-depth, multi-method evaluation of impact using portfolio-level ToC.

Designing for Scale and taking scale to account: lessons from a community score card project in Uganda

BACKGROUND

Planning for the implementation of community scorecards (CSC) is an important, though seldom documented process. Makerere University School of Public Health (MakSPH) and Future Health Systems Consortium set out to develop and test a sustainable and scalable CSC model. This paper documents the process of planning and adapting the design of the CSC, incorporating key domains of the scalable model such as embeddedness, legitimacy, feasibility and ownership, challenges encountered in this process and how they were mitigated.

METHODS

The CSC intervention comprised of five rounds of scoring in five sub counties and one town council of Kibuku district. Data was drawn from ten focus group discussions, seven key informant interviews with local and sub national leaders, and one reflection meeting with the project team from MakSPH. More data was abstracted from notes of six quarterly stakeholder meetings and six quarterly project meetings. Data was analyzed using a thematic approach, drawing constructs outlined in the project's theory of change.

RESULTS

Embeddedness, legitimacy and ownership were promoted through aligning the model with existing processes and systems as well as the meaningful and strategic involvement of stakeholders and leaders at local and sub national level. The challenges encountered included limited technical capacity of stakeholders facilitating the CSC, poor functionality of existing community engagement platforms, and difficulty in promoting community participation without financial incentives. However, these challenges were mitigated through adjustments to the intervention design based on the feedback received.

CONCLUSION

Governments seeking to scale up CSCs and to take scale to account should keenly adapt existing models to the local implementation context with strategic and meaningful involvement of key legitimate local and sub national leaders in decision making during the design and implementation process. However, they should watch out for elite capture and develop mitigating strategies. Social accountability practitioners should document their planning and adaptive design efforts to share good practices and lessons learned. Enhancing local capacity to implement CSCs should be ensured through use of existing local structures and provision of technical support by external or local partners familiar with the skill until the local partners are competent.

Theories of change for e-health interventions targeting HIV/STIs and sexual risk, substance use and mental ill health amongst men who have sex with men: systematic review and synthesis

BACKGROUND

Sexual risk, substance use, and mental ill health constitute a syndemic of co-occurring, mutually reinforcing epidemics amongst men who have sex with men (MSM). Developed since 1995, e-health interventions offer accessible, anonymous support and can be effective in addressing these outcomes, suggesting the potential value of developing e-health interventions that address these simultaneously amongst MSM. We conducted a systematic review of e-health interventions addressing one or more of these outcomes amongst MSM and in this paper describe the theories of change underpinning relevant interventions, what these offer and how they might complement each other.

METHODS

We identified eligible reports via expert requests, reference-checking and database and Google searches. Results were screened for reports published in 1995 or later; focused on MSM; reporting on e-health interventions providing ongoing support to prevent HIV/STIs, sexual risk behaviour, substance use, anxiety or depression; and describing intervention theories of change. Reviewers assessed report quality, extracted intervention and theory of change data, and developed a novel method of synthesis using diagrammatic representations of theories of change.

RESULTS

Thirty-three reports on 22 intervention theories of change were included, largely of low/medium-quality. Inductively grouping these theories according to their core constructs, we identified three distinct groupings of theorised pathways. In the largest, the 'cognitive/skills' grouping, interventions provide information and activities which are theorised to influence behaviour via motivation/intention and self-efficacy/perceived control. In the 'self-monitoring' grouping, interventions are theorised to trigger reflection, self-reward/critique and self-regulation. In the 'cognitive therapy' grouping, the theory of change is rooted in cognitive therapy techniques, aiming to reframe negative emotions to improve mental health.

CONCLUSIONS

The synthesised theories of change provide a framework for developing e-health interventions that might holistically address syndemic health problems amongst MSM. Improving reporting on theories of change in primary studies of e-health interventions would enable a better understanding of how they are intended to work and the evidence supporting this. The novel diagrammatic method of theory of change synthesis used here could be used for future reviews where interventions are driven by existing well-defined behaviour and behaviour change theories.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42018110317.

Designing and using digital platforms for 21st century learning

This paper is written in response to the original article "Designing for 21 ^st century learning online: A heuristic method to enable educator learning support roles" (Nacu et al. 2018). The article presents a guide for teacher interactions with students in networked technologies. It also serves as a guide for network designers who are concerned about the quality of online learning, and want more equitable access to systems that inspire young people "to pursue their interests and take ownership of their learning" (Nacu et al. 2018, p. 1029). During the Covid-19 pandemic, network platforms assumed a central role in the educational process. The article is a timely reminder of the importance of designing and implementing platforms that will address twenty-first century learning goals. This response to the Nacu et al. article reviews their research from a "theory of change" perspective. The authors in the original article present a heuristic that expands the online roles that educators currently use to support young people in building their knowledge. To realize the full potential of the heuristic, a more holistic approach is required, one that repositions online learning as a knowledge-building environment. A theory of change could identify the critical variables needed to help teachers and designers transition to a more comprehensive understanding of online learning. Unfortunately, heuristics by themselves do not prepare network designers or teachers to do this work. Future research can embed the heuristic in a broader, deeper effort to prepare teachers and designers to enact student-centered online learning environments.

Combining the theory of change and realist evaluation approaches to elicit an initial program theory of the MomConnect program in South Africa

Background

One of the Sustainable Development Goals is to reduce the global maternal mortality ratio to less than 70 per 100,000 live births by 2030. In South Africa, the flagship National Department of Health MomConnect program was launched in 2014 to strengthen the quality of maternal and child health (MCH) services and improve mortality outcomes. MomConnect was rapidly rolled out with a limited understanding of how and why the program was expected to work even though studies had shown the effectiveness of the MomConnect program in improving the uptake of MCH services. This study aimed to unearth the initial program theory of the MomConnect program based on explicit and implicit assumptions of how the program was organized and expected to work.

Methods

We conducted a document analysis using design- and implementation-related documents of the MomConnect program guided by the principles of Theory of Change (ToC) and Realist Evaluation (RE). Content and thematic analysis approaches were deductively applied to analyze the documents toward constructing ToC and RE-informed models. Abductive thinking and retroduction were further applied to the realist-informed approach to link program context, mechanisms, and outcomes to construct the initial program theory.

Results

ToC and RE-informed models illustrated how the MomConnect program was organized and expected to work. The process of constructing the ToC provided the platform for the development of the initial program theory, which identified three critical elements: (1) the central modalities of the MomConnect program; (2) the intended outcomes; and (3) the tentative causal links indicating, in a stepwise manner of, how the outcomes were intended to be achieved. The RE approach ‘enhanced’ the causal links by identifying relevant programmatic contexts and linking the postulated mechanisms of action (empowerment, encouragement, motivation, and knowledge acquisition) to program outcomes.

Conclusion

The application of ToC and RE provided an explicitly cumulative approach to knowledge generation in unveiling the initial program theory of MomConnect rather than delivering answers to questions of program effectiveness.

Tacit working models of human behavioural change I: Implementation of conservation projects

The "human dimension" of conservation is increasingly recognised as critical for success. Most conservation research involving people is based not on explicit "theories of change", but tacit local knowledge or folk theories guiding programme design.In this study, I propose a schematization of the local socioecological knowledge and folk theories about the "human dimension" of conservation into tacit working models, comprised of individual factors and systemic factors influencing human behaviour in conservation contexts. These are called the Persuasion, Normative, Involvement and Uniformity tacit working models. I review a set of conservation interventions and programmes, in order to assess which of the implicit working models inform their design. I argue that in order to better understand how a project may arrive at different outcomes, the underlying assumptions about human behaviour and the implicit "theory of change" that went into programme design need to be made explicit. This schema does not evaluate different approaches to conservation, but it can help point out the underlying assumptions that structure interventions and that may be more or less suited to particular situations. This can allow researchers to recognise their own assumptions and test them explicitly, leading to the formulation of more reflective and explicit theories, and improving the quality of both discourse and practice in conservation.

A review of animal health and drug use practices in India, and their possible link to antimicrobial resistance

BACKGROUND

Livestock production, particularly the dairy sector, is important for food and nutritional wellbeing of communities in India, it supports livelihoods of many farmers, and contributes to the economy of the country. India is a high consumer of antibiotics and antimicrobial resistant (AMR) bacteria are a major public health concern.

OBJECTIVES

Our objectives were to identify animal health and drug use practices that may contribute to emergence and spread of AMR in the country, review previous AMR- mitigation strategies, and discuss "theory of change" as an approach to informing the choice of interventions.

METHODS

We undertook a desk review of literature to identify practices with potential to contribute to emergence and spread of antimicrobial resistance in India. Searches were done in PubMed, Google scholar, and Google. Data were synthesized and discussed by themes.

RESULTS

Animal disease surveillance is less developed and infrastructure to support delivery of services is inadequate. Several groups are known to offer animal health services. The untrained "animal health workers" and para-veterinarians are more popular with farmers as they charge less for consultations (compared to veterinarians who are few and charge more). Over-the-counter access of antibiotics, without prescription, and direct marketing of drugs to farmers are common. Because of this, farmers are able to treat their animals and only consult when cases become non- responsive to treatment. Antibiotics are mostly used in management of mastitis cases. Drug withdrawal periods are rarely observed and occurrence of antibiotic- contaminated milk has been reported. Awareness on AMR is low and antimicrobial stewardship in livestock is yet to be developed. Initiatives such as the National programme for containment of AMR, National Action Plan on AMR, and the National Health policy point to government's commitment in addressing the problem of AMR in the country.

CONCLUSION

Several animal health and drug use practices, with potential to cause AMR, have been described, and their contribution can be discussed further by engaging stakeholders in a "theory of change" exercise. Interventions that address AMR from the animal health perspective should be promoted, and incentives to increase their adoption explored.

Building evaluative culture in community services: Caring for evidence

An organization with a strong evaluative culture engages in self-reflection, evidence-based learning and experimentation. It sees evidence as essential for managing well, but building such a culture is challenging. Community service organizations seek to provide effective services for their clients. To build an evaluative culture, they need to acquire basic monitoring and evaluation capabilities, be provided with opportunities for using these capabilities and be adequately motivated to care about evidence as a means to improve services to their clients. Leadership along with a phased in approach are key in bringing about these behaviour changes.

Pathways towards scaling up Problem Management Plus in Turkey: a theory of change workshop

Background

A considerable evidence base has been produced in recent years highlighting the effectiveness of brief scalable psychological interventions for people living in communities exposed to adversity. However, practical guidance on how to scale up these interventions to wider populations does not exist. In this paper we report on the use of Theory of Change (ToC) to plan the scale up of the World Health Organization’s flagship low intensity psychological intervention “Problem Management Plus” (PM+) for Syrian refugees in Turkey.

Methods

We conducted a one-day ToC workshop in Istanbul. ToC is a participatory planning process used in the development, implementation and evaluation of projects. It is similar to driver diagrams or logic models in that it offers a tool to visually present the components needed to reach a desired long-term outcome or impact. The overall aim of ToC is to understand the change process of a complex intervention and to map out causal pathways through which an intervention or strategy has an effect.

Results

Twenty-four stakeholders (including governmental officials, mental health providers, officials from international/national non-governmental organisations, conflict and health researchers) participated in the ToC workshop. A ToC map was produced identifying three key elements of scaling up (the resource team; the innovation and the health system; and the user organisation) which are represented in three distinct causal pathways. Context-specific barriers related to the health system and the political environment were identified, and possible strategies for overcoming these challenges were suggested.

Conclusion

ToC is a valuable methodology to develop an integrated framework for scaling up. The results highlight that the scaling up of PM+ for Syrian refugees in Turkey needs careful planning and investment from different stakeholders at the national level. Our paper provides a theoretical foundation of the scaling up of PM+, and exemplifies for the first time the use of ToC in planning the scaling up of an evidence-based psychological intervention in global mental health.

A refined method for theory-based evaluation of the societal impacts of research

With high and increasing expectations for research to have social and environmental impact, there is a corresponding need for appropriate methods to demonstrate (for accountability) and analyze (for learning) whether and how research projects contribute to change processes. Evaluation is especially challenging for problem-oriented research that employs inter- and transdisciplinary approaches and intervenes in complex systems, where experimental and statistical approaches to causal inference are inappropriate. Instead, theory-based evaluation can be applied to identify and test causal processes. This paper presents a detailed explanation of the Outcome Evaluation approach applied in Belcher et al. (2019b). It draws on concepts and approaches used in theory-based program evaluation and the more limited experience of theory-based research evaluation, providing a brief overview of conceptual strengths and limitations of other methods. The paper offers step-by-step guidance on application of the Outcome Evaluation approach, detailing how to: document a theory of change; determine data needs and sources; collect data; manage and analyze data; and present findings. This approach provides a clear conceptual and analytical framework in addition to actor-specific and impact pathway analyses for more precision in the assessment of outcomes. Specifically, the Outcome Evaluation approach: •Conceptualizes research within a complex system and explicitly recognizes the role of other actors, context, and external processes;•Utilizes a detailed actor-centred theory of change (ToC) as the analytical framework; and•Explicitly tests a set of hypotheses about the relationship between the research process/outputs and outcomes.

Analysing facilitating and hindering factors for implementing gender equality interventions in R&I: Structures and processes

This article analyses the facilitating and hindering factors that have affected the implementation of gender equality interventions in research and innovation in Europe. It applies the evaluation framework developed in the EFFORTI project that recognizes the complexity of evaluating gender equality interventions in R&I, the importance of factoring in context to any sound evaluation as well as the need to distinguish between the design and implementation of interventions in evaluations. It is based on the analysis of 19 empirical case studies carried out throughout Europe and focuses on those structural and procedural factors that have either facilitated or hindered the implementation process of these interventions. Findings include how the governance framework; top-management commitment; bottom-up participation; framing synergies with other initiatives, strategies for tackling resistance; resources; sustainability of actions; gender competence, experience and knowledge and transparency, targets, standards and monitoring; and accessible data and information all contributed to the successful implementation of the interventions.

Finding "What Works": Theory of Change, Contingent Universals, and Virtuous Failure in Global Mental Health

Global Mental Health has developed interventions that strive to work across great difference-variously conceptualized as cultural, socio-economic, geographic, or pertaining to the characteristics of health systems. This article discusses how the evaluation framework Theory of Change (ToC) facilitates the production of 'global' knowledge across such differences. Drawing on 14 months of multi-sited fieldwork among Global Mental Health actors in Europe, North America and South Africa, it traces the differential use of ToC in GMH interventions. While much critical scholarship of Global Health metrics holds that techniques of quantification rely on universals that necessarily betray the "real world", ToC unsettles these critiques. It comes into view as an epistemic and relational device that produces 'contingent universals'-concepts that are true and measurable until they stop working in the field, or until the parameters of 'what works' shift to a new iteration. As such, Theory of Change produces actionable-rather than true-knowledge attuned to open-ended change, both desirable (impact) and unforeseen (adaptation). Its effects, however, are ambiguous. ToC presents us with a horizoning technique that enables what I call "virtuous failure" within the evidence-based paradigm. It may equally harbor the potential to disrupt distinctions such as bricolage (tinkering) and design (planning) and their respective politics, as it may tie neatly into audit cultures, depending on its use. The article analyzes the novel stakes of reflexive evaluation techniques and calls on anthropology and critical Global Health for renewed empirical engagement.

Using theory of change as a framework for examining community context and philanthropic impact

As part of the process of examining their theory of change (TOC) and reflecting on grant making activity, one Midwestern foundation employed a multipronged strategy to assess 209 community-based mental health grants across seven years of funding. This article details the evaluation approach, which comprised these areas of the TOC: grantees' use of evidence-based interventions, cultural competency, quality improvement, community collaboration, and use of integrated care. Inductive analyses identified grantees' use of innovative practices, trauma-informed care, and use of validated instruments. In an iterative exercise spanning multiple years, the foundation and researchers found ways to gather information about community capacity and impact while simultaneously minimizing grantee data collection/reporting burden. Findings revealed that community agencies were improving in their uptake of evidence-based interventions and use of validated instruments. Community grantee use of trauma informed care also improved over the study period. Grantee desire to collaborate and coordinate services within the community was also strong. The research also revealed that in multiple domains (such as technology and non-profit operating costs) grantees needed support in building agency capacity. These findings shed light on which aspects of the foundation's TOC were being addressed through funding, and which areas of the community needed additional support.

Participatory learning and action to address type 2 diabetes in rural Bangladesh: a qualitative process evaluation

BACKGROUND

Diabetes is 7th largest cause of death worldwide, and prevalence is increasing rapidly in low-and middle-income countries. There is an urgent need to develop and test interventions to prevent and control diabetes and develop the theory about how such interventions can be effective. We conducted a participatory learning and action (PLA) intervention with community groups in rural Bangladesh which was evaluated through a cluster randomised controlled trial. There was a large reduction in the combined prevalence of type 2 diabetes and intermediate hyperglycaemia in the PLA group compared with the control group. We present findings from qualitative process evaluation research to explore how this intervention was effective.

METHODS

We conducted group interviews and focus group discussions using photovoice with purposively sampled group attenders and non-attenders, and intervention implementers. Data were collected before the trial analysis. We used inductive content analysis to generate theory from the data.

RESULTS

The intervention increased the health literacy of individuals and communities - developing their knowledge, capacity and self-confidence to enact healthy behaviours. Community, household and individual capacity increased through social support and social networks, which then created an enabling community context, further strengthening agency and enabling community action. This increased opportunities for healthy behaviour. Community actions addressed lack of awareness about diabetes, gendered barriers to physical activity and lack of access to blood glucose testing. The interaction between the individual, household, and community contexts amplified change, and yet there was limited engagement with macro level, or 'state', barriers to healthy behaviour.

CONCLUSIONS

The participatory approach enabled groups to analyse how context affected their ability to have healthy behaviours and participants engaged with issues as a community in the ways that they felt comfortable. We suggest measuring health literacy and social networks in future interventions and recommend specific capacity strengthening to develop public accountability mechanisms and health systems strengthening to complement community-based interventions.

TRIAL REGISTRATION

Registered at ISRCTN on 30th March 2016 (Retrospectively Registered) Registration number: ISRCTN41083256 .