Theory-based evaluation: Past, present, and future

Beyond the average effect of the innovation subsidies: Using case selection via matching to break impasse in delivering useful advice to policy makers

Experts and stakeholders promote the combined use of counterfactual and theory-based approaches in program evaluation. We illustrated combined application of these two approaches in a single evaluation study of innovation subsidies, using "case selection via matching" and follow-up in-depth interviews. We conducted interviews in contrasting pairs of companies-one successful and one unsuccessful-which were otherwise similar on a defined set of covariates. Our procedure helped to reveal factors, which might be overlooked or simply not available in data at hand and hence not accounted for in analyses of the intervention effects. As such it extends beyond the average effect estimate to highlight causes of an intervention success or failure.

How do oilcloth sessions work? A realist evaluation approach to exploring ripple effects in an implementation strategy

PURPOSE

To explore the mechanisms of the implementation strategy, "oilcloth sessions" and understand and explain the ripple effects of oilcloth sessions as a strategy to implement a new emergency department.

DESIGN/METHODOLOGY/APPROACH

A qualitative design was used whereby data were collected using field notes from an ethnographic study of the oilcloth sessions and follow-up semi-structured interviews with staff, managers and key employees who participated in the oilcloth sessions. The data analysis was inspired by the realist evaluation approach of generative causality proposed by Pawson and Tilley.

FINDINGS

The primary ripple effect was that the oilcloth sessions were used for different purposes than the proposed program theory, including being used as: (1) a stage, (2) a battlefield, (3) a space for imagination and (4) a strategic management tool influencing the implementation outcomes. The results bring essential knowledge that may help to explain why and how a well-defined implementation strategy has unplanned outcomes.

ORIGINALITY/VALUE

Unintended outcomes of implementation strategies are an underexplored issue. This study may help implementation researchers rethink the activities required to reduce unintended negative outcomes or explore potential unplanned outcomes and, in this way, hinder or enhance outcomes, effectiveness and sustainability. Future studies within implementation research should incorporate attention to unintended outcomes to fully understand the impact of implementation strategies.

Equity-related 'knots' in theory of change development: Conceptualization and case illustrations

Theory of change (ToC) is an approach widely used to guide planning, implementing, and evaluating change initiatives. While there is substantial guidance, there has been little attention on equity within ToC research and practice. We propose and illustrate the metaphor of 'knots' to frame practical and ethical challenges that arise when centering equity within ToC processes. Drawing on our experiences using a ToC approach in two case examples, we identify and illustrate five equity-related knots: (a) clarify root causes, pathways, and success; (b) facilitate participation across power and perspective differences; (c) integrate research evidence and practitioner knowledge; (d) represent complex change visually; and (e) creatively navigate constraints. We show why framing these as knots can help practitioners make wise judgments within the circumstances and close with recommendations for including knots in ToC processes, reporting, and guidance.

Academic Leadership Academy Summer Program: Clerkship Transition Preparation for Underrepresented in Medicine Medical Students

PROBLEM

Enhancing workforce diversity by increasing the recruitment of students who have been historically excluded/underrepresented in medicine (UIM) is critical to addressing healthcare inequities. However, these efforts are inadequate when undertaken without also supporting students' success. The transition to clerkships is an important and often difficult to navigate inflection point in medical training where attention to the specific needs of UIM students is critical.

INTERVENTION

We describe the design, delivery, and three-year evaluation outcomes of a strengths-based program for UIM second year medical students. The program emphasizes three content areas: clinical presentations/clinical reasoning, community building, and surfacing the hidden curriculum. Students are taught and mentored by faculty, residents, and senior students from UIM backgrounds, creating a supportive space for learning.

CONTEXT

The program is offered to all UIM medical students; the centerpiece of the program is an intensive four-day curriculum just before the start of students' second year. Program evaluation with participant focus groups utilized an anti-deficit approach by looking to students as experts in their own learning. During focus groups mid-way through clerkships, students reflected on the program and identified which elements were most helpful to their clerkship transition as well as areas for programmatic improvement.

IMPACT

Students valued key clinical skills learning prior to clerkships, anticipatory guidance on the professional landscape, solidarity and learning with other UIM students and faculty, and the creation of a community of peers. Students noted increased confidence, self-efficacy and comfort when starting clerkships.

LESSONS LEARNED

There is power in learning in a community connected by shared identities and grounded in the strengths of UIM learners, particularly when discussing aspects of the hidden curriculum in clerkships and sharing specific challenges and strategies for success relevant to UIM learners. We learned that while students found unique benefits to preparing for clerkships in a community of UIM students, near peers, and faculty, future programs could be enhanced by pairing this formal intensive curriculum with more longitudinal opportunities for community building, mentoring, and career guidance.

Understanding the convergence of social enterprise, digital health, and citizen engagement for co-producing integrated Person-Centred health services: A critical review and theoretical framework

INTRODUCTION

To achieve Universal Health Coverage and the United Nations' (UN) Sustainable Development Goals (SDGs) agenda for 2030, the World Health Organisation (WHO) recommended the use of social enterprise, digital technology, and citizen engagement in the delivery of Integrated People-Centred Health Services (IPCHS) as part of its strategic vision for 21st century primary care.

METHODS

We conducted a hermeneutic review of frameworks, models and theories on social enterprise, digital health, citizen engagement and IPCHS. This involved multiple iterative cycles of (i) searching and acquisition, followed by (ii) critical analysis and interpretation of literature to assemble arguments and evidence for conceptual relationships until information saturation was reached. This process identified a set of constructs which we synthesised into a testable framework.

RESULTS

Several interdisciplinary frameworks, models and theories explain how social enterprises could use digital technology, and citizen engagement to enable the technical and social integration required to facilitate people-centred primary care. Innovative approaches can be used to maintain financial sustainability while delivering IPCHS at lower cost to vulnerable and marginalised populations in both developed and developing countries.

CONCLUSION

This framework provides a theoretical grounding to guide empirical inquiry into how social enterprises use digital technology to engage citizens in co-producing IPCHS.

Aligning intuition and theory: a novel approach to identifying the determinants of behaviours necessary to support implementation of evidence into practice

BACKGROUND

Disentangling the interplay between experience-based intuition and theory-informed implementation is crucial for identifying the direct contribution theory can make for generating behaviour changes needed for successful evidence translation. In the context of 'clinicogenomics', a complex and rapidly evolving field demanding swift practice change, we aimed to (a) describe a combined clinician intuition- and theory-driven method for identifying determinants of and strategies for implementing clinicogenomics, and (b) articulate a structured approach to standardise hypothesised behavioural pathways and make potential underlying theory explicit.

METHODS

Interview data from 16 non-genetic medical specialists using genomics in practice identified three target behaviour areas across the testing process: (1) identifying patients, (2) test ordering and reporting, (3) communicating results. The Theoretical Domains Framework (TDF) was used to group barriers and facilitators to performing these actions. Barriers were grouped by distinct TDF domains, with 'overarching' TDF themes identified for overlapping barriers. Clinician intuitively-derived implementation strategies were matched with corresponding barriers, and retrospectively coded against behaviour change techniques (BCTs). Where no intuitive strategies were provided, theory-driven strategies were generated. An algorithm was developed and applied to articulate how implementation strategies address barriers to influence behaviour change.

RESULTS

Across all target behaviour areas, 32 identified barriers were coded across seven distinct TDF domains and eight overarching TDF themes. Within the 29 intuitive strategies, 21 BCTs were represented and used on 49 occasions to address 23 barriers. On 10 (20%) of these occasions, existing empirical links were found between BCTs and corresponding distinct TDF-coded barriers. Twenty additional theory-driven implementation strategies (using 19 BCTs on 31 occasions) were developed to address nine remaining barriers.

CONCLUSION

Clinicians naturally generate their own solutions when implementing clinical interventions, and in this clinicogenomics example these intuitive strategies aligned with theoretical recommendations 20% of the time. We have matched intuitive strategies with theory-driven BCTs to make potential underlying theory explicit through proposed structured hypothesised causal pathways. Transparency and efficiency are enhanced, providing a novel method to identify determinants of implementation. Operationalising this approach to support the design of implementation strategies may optimise practice change in response to rapidly evolving scientific advances requiring swift translation into healthcare.

From the Brazilian Income Transfer Program to Brazil Assistance: challenges and achievements according to a theory-driven evaluation research on the program

The recent COVID-19 pandemic has led many countries to adopt emergency cash transfer policies as a way to contain the economic and humanitarian crisis. Such initiatives were developed in a context of exacerbated gender, race, ethnicity and class inequalities resulting from physical and social distancing measures. The article analyzes, by means of an exploratory study and the so-called theory-driven evaluation of the program, the theoretical premises of the Brazilian Income Transfer Program (2003), Brazilian Emergency Assistance (2020) and Brazil Assistance (2021) programs, and their corresponding implementation dynamics. As cash transfer programs are given centrality in the contemporary public agenda, the conclusion is that evaluating their limits and advances - as to theoretical conception and mechanisms triggered in each context - contributes to trace evidence about their effectiveness in addressing long-term inequalities and those inequalities that arise in health emergency contexts.

Evaluating Research for Development: Innovation to Navigate Complexity

Large publicly funded programmes of research continue to receive increased investment as interventions aiming to produce impact for the world's poorest and most marginalized populations. At this intersection of research and development, research is expected to contribute to complex processes of societal change. Embracing a co-produced view of impact as emerging along uncertain causal pathways often without predefined outcomes calls for innovation in the use of complexity-aware approaches to evaluation. The papers in this special issue present rich experiences of authors working across sectors and geographies, employing methodological innovation and navigating power as they reconcile tensions. They illustrate the challenges with (i) evaluating performance to meet accountability demands while fostering learning for adaptation; (ii) evaluating prospective theories of change while capturing emergent change; (iii) evaluating internal relational dimensions while measuring external development outcomes; (iv) evaluating across scales: from measuring local level end impact to understanding contributions to systems level change. Taken as a whole, the issue illustrates how the research for development evaluation field is maturing through the experiences of a growing and diverse group of researchers and evaluators as they shift from using narrow accountability instruments to appreciating emergent causal pathways within research for development.

Community health workers' involvement in mother-child care during the 1st year after birth, in Kaya health district, Burkina Faso: A contribution analysis

Introduction

Maternal and infant morbidity and mortality remain high in sub-Saharan Africa. However, actions to strengthen postpartum care are still weak and mainly limited to health facilities (HFs). In Kaya health district, Burkina Faso, community health workers (CHWs) were involved in mother and child care during the 1st year postpartum through home visits, outreach sessions and accompanying mothers to health facilities. The aim of this study was to assess the contribution of CHWs to postpartum women's attendance at the health facilities.

Methods

We conducted an effect assessment using Mayne and Lemire's contribution analysis framework. Qualitative and quantitative data were collected through project documents review and individual semi-structured interviews with key-informants.

Results

All the participants interviewed acknowledged that the number of women, who came to postpartum care, had increased since the implementation of the project activities. Postpartum consultation rates within the 1st week postpartum increased from 29% in 2011 to 80% in 2015 and from 19 to 50% within 6 weeks. Others interventions such as Performance based financing, Save The Children nutritional project and the health services component of Missed Opportunities in Mother and Infant Health (MOMI) were the alternative explanations.

Conclusions

CHWs involvement in women care contributed to improve their adherence to postpartum consultations in Kaya health district.

Improving child health service interventions through a Theory of Change: A scoping review

Background

The objective of this scoping review was to map how child health service interventions develop, utilise, and refine theories of change. A Theory of Change (ToC) is a tool for designing, implementing, and evaluating interventions that is being increasingly used by child health practitioners who are aiming to enact change in health services.

Methods

A published protocol guided this scoping review. Relevant publications were identified through selected electronic databases and grey literature via a search strategy. The main inclusion criteria were any child health service intervention globally that described their ToC or ToC development process. These were applied by two independent reviewers. Data relevant to the research sub-questions were extracted, charted and discussed.

Findings

38 studies were included in the analysis. This scoping review highlights the disparate and inconsistent use, and reporting of ToCs in the child health service intervention literature.

Conclusion

A ToC may be a helpful tool to enact change in a child health service but careful consideration must be undertaken by the child health service regarding how to maximise the benefits of doing a ToC, and how to accurately report it.

Improving HIV pre-exposure prophylaxis persistence among adolescent girls and young women: Insights from a mixed-methods evaluation of community, hybrid, and facility service delivery models in Namibia

Introduction

Despite the potential for community-based approaches to increase access to pre-exposure prophylaxis (PrEP) for adolescent girls and young women (AGYW), there is limited evidence of whether and how they improve PrEP persistence. We compared PrEP persistence among AGYW receiving services through community and hybrid models in Namibia to facility-based services. We subsequently identify potential mechanisms to explain how and why community and hybrid models achieved (or not) improved persistence to inform further service delivery innovation.

Methods

Data were collected from PrEP service delivery to AGYW over two-years in Namibia's Khomas Region. We used Kaplan-Meier analysis to estimate survival curves for PrEP persistence beyond three-months after initiation and report the cumulative probability of persistence at one- and three-months. Persistence was defined as any PrEP use within three months after initiation followed by a PrEP refill or previously prescribed supply of at least 30 days at the three-month visit. Interviews were conducted with 28 AGYW and 19 providers and analyzed using a deductive-inductive thematic approach.

Results

From October 2017 through September 2019, 372 (18.7%) AGYW received services through a facility model, 302 (15.1%) through a community model, and 1,320 (66.2%) through a hybrid model. PrEP persistence at one- and three-months was 41.2% and 34.9% in the community model and 6.2% and 4.8% in the hybrid model compared to 36.8% and 26.7% in the facility model. Within the community and hybrid models, we identified three potential mechanisms related to PrEP persistence. Individualized service delivery offered convenience and simplicity which enabled AGYW to overcome barriers to obtaining refills but did not work as well for highly mobile AGYW. Consistent interactions and shared experiences fostered social connectedness with providers and with peers, building social networks and support systems for PrEP use. PrEP and HIV-related stigma, however, was widely experienced outside of these networks. Community-to-facility referral for PrEP refill triggered apprehension towards unfamiliar PrEP services and providers in AGYW, which discouraged persistence.

Conclusion

Service delivery approaches that offer convenience and simplicity and foster social connectedness may reduce access barriers and increase social support enabling AGYW to self-manage their PrEP use and achieve improved PrEP persistence.

Theory-based evaluation of three research-practice partnerships designed to deliver novel, sustainable collaborations between adult social care research and practice in the UK: a research protocol for a 'layered' contributions analysis and realist evaluation

INTRODUCTION

Research-practice partnerships (RPPs) are long-term collaborations between research and practice that aim to conduct research that can be used to make practice-based improvements. They intentionally bring together diverse experience in decision making and seek to shift power dynamics so that all partners have a say. The Creating Care Partnerships project aims to explore whether the RPP approach developed within the US educational context can be successfully applied to the English care home context. The project involves a programme of codesign, implementation and evaluation within three case study sites. This protocol set outs the aims, research design and governance of the evaluation.

METHODS AND ANALYSIS

The evaluation takes a theory-based approach to explore how, why and in what circumstances RPPs in the care home context contribute to enhancing research and research use in local care homes and informing wider improvement efforts. A mixed-methods design will be used for each case study, including semistructured interviews, observations of RPP events and meetings, an online survey, activity diary and review of local data and documents. Data collection will proceed in waves, with the theory of change (ToC) being continually refined and used to guide further data collection and analysis. Insights will be drawn using Contribution Analysis, Realist Evaluation and systems perspectives to assess the contribution made by the case study sites to achieving outcomes and the influence of contextual factors. Economic consequences will be identified through the ToC, using a narrative economic analysis to assess costs, consequences and value for money.

ETHICS AND DISSEMINATION

The study has undergone ethics review by HRA Research Ethics Committee. It does not pose major ethical issues. A final report will be published and articles will be submitted to international journals.

Improving team effectiveness using a program evaluation logic model: case study of the largest provincial simulation program in Canada

Historically simulation-based education (SBE) has primarily focused on program development and delivery as a means for improving the effectiveness of team behaviours; however, these programs rarely embed formal evaluations of the programs themselves. Logic models can provide simulation programs with a systematic framework by which organizations and their evaluators can begin to understand complex interprofessional teams and their programs to determine inputs, activities, outputs and outcomes. By leveraging their use, organizational leaders of simulation programs can contribute to both

Exploring the design of a sport for employability program: A case study

Rates of young people who are neither in employment, education, or training (NEET) are fairly high in the European Union. Correspondingly, there has been a growing tendency to regard sport as a suitable tool to develop soft skills and raise NEETs' level of employability. However, if and how such sport for employability (SfE) programs are able to realize these major claims has been called into question. The purpose of the present study was, therefore, to explore how an actual SfE initiative constructs and delivers its program. In addition, the article assessed whether the investigated program operates in line with researchers' recent calls for theory-based approaches. Guided by a case study approach set up within an initiative located in Flanders, data were gathered through 12 semi-structured interviews with 8 program providers. Results, analyzed using thematic analysis, indicated that the program was characterized by an absence of well-defined desired outcomes, imprecision as to how the program should contribute to these outcomes, and consequently minimal attention to the follow-up of participants' progress. As such, these findings and the accompanying challenges point to the absence of a theory-based approach. Several possible sources for the lack of a systematic approach are discussed.

Key learnings from an outcome and embedded process evaluation of a direct to beneficiary mobile health intervention among marginalised women in rural Bihar, India

INTRODUCTION

Mobile Vaani was implemented as a pilot programme across six blocks of Nalanda district in Bihar state, India to increase knowledge of rural women who were members of self-help groups on proper nutrition for pregnant or lactating mothers and infants, family planning and diarrhoea management. Conveners of self-help group meetings, community mobilisers, introduced women to the intervention by giving them access to interactive voice response informational and motivational content. A mixed methods outcome and embedded process evaluation was commissioned to assess the reach and impact of Mobile Vaani.

METHODS

The outcome evaluation, conducted from January 2017 to November 2018, used a quasi-experimental pre-post design with a sample of 4800 married women aged 15-49 from self-help group households, who had a live birth in the past 24 months. Surveys with community mobilisers followed by meeting observations (n=116), in-depth interviews (n=180) with self-help group members and secondary analyses of system generated data were conducted to assess exposure and perceptions of the intervention.

RESULTS

From the outcome evaluation, 23% of women interviewed had heard about Mobile Vaani. Women in the intervention arm had significantly higher knowledge than women in the comparison arm for two of seven focus outcomes: knowledge of how to make child's food nutrient and energy dense (treatment-on-treated: 18.8% (95% CI 0.4% to 37.2%, p<0.045)) and awareness of at least two modern spacing family planning methods (treatment-on-treated: 17.6% (95% CI 4.7% to 30.5%, p<0.008)). Women with any awareness of Mobile Vaani were happy with the programme and appreciated the ability to call in and listen to the content.

CONCLUSION

Low population awareness and programme exposure are underpinned by broader population level barriers to mobile phone access and use among women and missed opportunities by the programme to improve targeting and programme promotion. Further research is needed to assess programmatic linkages with changes in health practices.

Realist evaluation of three programmes aimed at reducing harm and risks associated with alcohol consumption in the Nouvelle Aquitaine region of France: the ECIAE study protocol

INTRODUCTION

In Europe, alcohol consumption is responsible for many diseases, disabilities, injuries and premature deaths. In France, alcohol consumption represents an important health burden, due to its frequency, scale and the serious damage it causes. One of the keys to addressing the problem would appear to be the adoption of harm and risk reduction approaches. In order to operationalise this strategy, the Nouvelle Aquitaine Regional Health Agency is funding three different programmes to reduce the harm and risks associated with alcohol consumption: Alcochoix, Iaca and ETP (Education Thérapeutique du Patient) Conso-repère. We are interested in understanding how, under what circumstances, through which mechanisms and for which population the different programmes work.

METHODS AND ANALYSIS

The ECIAE study (a cross-evaluation of the 3 programs Iaca/Alcochoix/ ETP (Patient Therapeutic Education) is a theory-based evaluation where the realist evaluation method is used to explore effects, intervention mechanisms and the influence of context on outcomes. This realist evaluation is based on multiple case studies in two nested levels. At the first level, each centre implementing the programme will represent a case. At the second level, each programme will represent a case in which a set of activities is conducted to achieve risk reduction objectives.

ETHICS AND DISSEMINATION

The project will be carried out in full compliance with existing legislation and international conventions. It was subject to analysis, including a privacy impact assessment conducted by the Data Protection Officer of the University of Bordeaux. The University of Bordeaux has ensured that all the regulatory procedures related to the ECIAE study have been carried out. The dissemination plan includes scientific papers, seminars, a report and recommendation and a public restitution. The study will provide evidence-based results to help health authorities roll out strategies to reduce risks and harm associated with alcohol use.

Conducting focus groups in realist evaluation

Focus groups are valuable tools for evaluators to help stakeholders to clarify programme theories. In 1987, R.K. Merton, often attributed with the birth of focus groups, wrote about how these were 'being mercilessly misused'. In the 1940s, his team had conceived focus groups as tools for developing middle-range theory, but through their astonishing success focus groups have metamorphosed and are often an 'unchallenged' choice in many evaluation approaches, while their practice seems to provide a philosophically diverse picture. This article examines what knowledge focus group data generate, and how they support theory development. It starts with an overview of the history of focus groups, establishing a relationship between their emergence as a data collection method and the evaluation profession. Practical lessons for conducting groups in realist evaluation are suggested, while exploring how qualitative data can support programme and middle-range theory development using the example of realist evaluation.

Realist evaluation of the impact, viability and transferability of an alcohol harm reduction support programme based on mental health recovery: the Vitae study protocol

INTRODUCTION

Addiction is considered a chronic disease associated with a high rate of relapse as a consequence of the addictive condition. Most of the current therapeutic work focuses on the notion of relapse prevention or avoidance and the control of its determinants. Since only a small portion of patients can access alcohol addiction treatment, it is crucial to find a way to offer new support towards safe consumptions, reductions or cessations. The harm reduction (HR) approach and mental health recovery perspective offers another way to support the patient with alcohol addiction. Vitae is a realist evaluation of the impact, viability and transferability of the IACA! programme, an HR programme based on the principle of psychosocial recovery for people with alcohol use disorders.

METHODS AND ANALYSIS

The Vitae study adheres to the theory-driven evaluation framework where the realist evaluation method and contribution analysis are used to explore the effects, mechanisms and influence of context on the outcomes and to develop and adjust an intervention theory. This study is a 12-month, multi-case, longitudinal descriptive pilot study using mixed methods. It is multi-centred, and carried out in 10 addiction treatment or prevention centres. In this study, outcomes are related to the evolution of alcohol use and the beneficiaries trajectory in terms of psychosocial recovery during these 12 months after the start of IACA!. The target number of participants are 100 beneficiaries and 23 professionals.

ETHICS AND DISSEMINATION

This research was approved by the Committee for the Protection of Persons Ouest V n°: 21/008-3HPS and was reported to the French National Agency for the Safety of Health Products. All participants will provide consent prior to participation. The results will be reported in international peer-reviewed journals and presented at scientific and public conferences.

TRIAL REGISTRATION NUMBERS

NCT04927455; ID-RCB2020-A03371-38.

Towards health equity: core components of an extended home visiting intervention in disadvantaged areas of Sweden

BACKGROUND

Understanding the mechanisms of implementation of public health interventions in community settings is a key aspect of programme assessments. To determine core components and establish a programme theory are important tools to improve functioning and support dissemination of programme models to new locations. An extended early childhood home visiting intervention has been developed on-site in a socioeconomically disadvantaged area of Sweden since 2013 with the aim of reducing persisting health inequities in the population. This study aimed at investigating the core programme components and how the intervention was perceived to contribute towards health equity from early childhood.

METHODS

Qualitative framework method was applied in a document analysis and subsequent semi-structured interviews with 15 key actors involved in the programme.

RESULTS

The intervention was found to be constituted of five core components centred around the situation-based, parental strengthening work method delivered by a qualified team of child health care nurse and social worker. The programme theory foresaw positive effects on child and parental health, responsive parenting practices, families' use of welfare services according to need and increased integration and participation in society. The principles of Proportionate Universalism were recognised in the programme theory and the intervention was perceived as an important contribution to creating conditions for improved health equity for the families. Still, barriers to health equity were identified on the structural level which limit the potential impact of the programme.

CONCLUSIONS

The core components of the Extended home visiting programme in Rinkeby correspond well to those of similar evidence-based home visiting interventions. Combining focus on early childhood development and responsive parenting with promoting access to the universal welfare services and integration into society are considered important steppingstones towards health equity. However, a favourable macro-political environment is required in the endeavour to balance the structural determinants' influence on health inequities. Improved availability and accessibility to welfare services that respond to the needs of the families regarding housing, education and employment are priorities.

TRIAL REGISTRATION

The study was retrospectively registered on 11/08/2016 in the ISRCTN registry ( ISRCTN11832097 ).

Evaluation proposal of a national community-based obesity prevention programme: a novel approach considering the complexity perspective

Community-based obesity prevention programmes are considered an important strategy to curb the obesity epidemic. The JOGG (Youth At a Healthy Weight) approach is a large-scale community-based programme for childhood obesity prevention in the Netherlands that has been implemented over the past ten years. Practice-based development of the programme, both at the national and local level, increasingly poses challenges for its evaluation. One considerable challenge is the increasing acknowledgement of the complexity in the JOGG-approach, characterized by (a) objectives that vary locally, (b) adaptions to the programme over time in response to a community's shifting needs, challenges and opportunities, and (c) emergent outcomes and non-linear causality.We propose an evaluation framework that highlights elements of the complex local practice, including the local programme theory, implementation, adaption, the influence of context and feedback loops and intended as well as emergent and unintended outcomes. By studying each of these elements in practice, we hope to learn about principles that guide effective obesity prevention across contexts. The results of the proposed evaluation will inform both practice and research.Considering complexity in evaluation is a relatively new challenge in public health and therefore an emergent research area. The proposed framework for complex evaluations allows to retrospectively evaluate a programme that was implemented and developed in practice, and enables us to learn from practice-based experiences. Following the ISBNPA Dare2Share initiative, we kindly invite other researchers in the field to share their ideas and experiences regarding integration of complexity in evaluation.

Health promotion innovations scale up: combining insights from framing and actor-network to foster reflexivity

There are numerous hurdles down the road for successfully scaling up health promotion innovations into formal programmes. The challenges of the scaling-up process have mainly been conceived in terms of available resources and technical or management problems. However, aiming for greater impact and sustainability involves addressing new contexts and often adding actors whose perspectives may challenge established orientations. The social dimension of the scaling-up process is thus critical. Building on existing conceptualizations of interventions as dynamic networks and of evolving framing of health issues, this paper elaborates a social view of scaling up that accounts for the transformations of innovations, using framing analysis and the notion of 'expanding scaling-up networks'. First, we discuss interventions as dynamic networks. Second, we conceptualize scaling-up processes as networks in expansion within which social learning and change occur. Third, we propose combining a 'representational approach' to frame analysis and an 'interactional approach' that illustrates framing processes related to the micro-practices of leading public health actors within expanding networks. Using an example concerning equity in early childhood development, we show that this latter approach allows documenting how frames evolve in the process. Considering the process in continuity with existing conceptualizations of interventions as actor-networks and transformation of meanings enriches our conceptualization of scaling up, improves our capacity to anticipate its outcomes, and promotes reflexivity about health promotion goals and means.

Using theory of change in child health service interventions: a scoping review protocol

Background: The objective of this scoping review is to map the evidence of how child health service interventions use their theory of change. A theory of change is a hypothesis of how and why an intervention is intended to bring about change. It can be used as a program design, implementation, and evaluation tool. This scoping review will provide an overview of the evidence base for, and identify the way in which, theories of change in child health service interventions are defined, rationalised, developed, presented, and refined. Methods: The inclusion criteria for this scoping review is any child health service intervention globally, that describes their theory of change or theory of change development process. Relevant exclusions include: logic models or logic frameworks that do not meet this review's definition of theory of change, systematic reviews, behavioural change interventions that target patient's behaviour, school-based interventions, and maternal health interventions not related to child health outcomes. This scoping review will follow the Joanna Briggs Institute Reviewer's manual. Relevant publications will be first searched on selected electronic databases and grey literature. A search strategy will be developed. The search will be limited to articles written in the English language. Results of the search will be curated using Endnote and duplicates removed. Results will be imported to Rayyan. The inclusion criteria will be applied during the process of title and abstract screening, by two independent reviewers and disagreements resolved by a third independent reviewer. Full-texts will have the inclusion criteria applied via the same reviewer process. Data relevant to the research sub-questions will be extracted, analysed, charted and discussed. Ethics and dissemination: Ethical approval is not required for this review as we will make use of already published data. We aim to publish the findings of our review in a peer-reviewed journal.

Mechanisms for the prevention of adolescent intimate partner violence: A realist review of interventions in low- and middle-income countries

Adolescent girls are among those at the greatest risk of experiencing intimate partner violence (IPV). Despite adolescence being widely regarded as a window of opportunity to influence attitudes and behaviours related to gender equality, evidence on what works to prevent IPV at this critical stage is limited outside of high-income, school-based settings. Even less is understood about the mechanisms of change in these interventions. We conducted a realist review of primary prevention interventions for adolescent IPV in low- and middle-income countries (LMICs) to synthesise evidence on how they work, for whom, and under which circumstances. The review took place in four iterative stages: 1) exploratory scoping, 2) developing initial programme theory, 3) systematic database search, screening and extraction, and 4) purposive searching and refinement of programme theory. We identified eleven adolescent IPV prevention interventions in LMICs, most of which demonstrated a positive impact on IPV experience and/or perpetration (n = 10). Most interventions (n = 9) implemented school- or community-based interactive peer-group education to transform attitudes and norms around gender and relationships for behaviour change. The central mechanism of change related to gender transformative content prompting adolescents to critically reflect on their attitudes and relationships, leading to a reconceptualisation of their values and beliefs. This central mechanism was supported by two secondary implementation mechanisms: 1) the design and delivery of interventions: interactive, age-appropriate education delivered in peer-groups provided adolescents a safe space to engage with content and build communication skills, and 2) the target group: social norms interventions targeting the wider community created enabling environments supportive of individual change. This review highlights the immense potential of gender transformative interventions during the critical period of adolescence for IPV prevention. Future interventions should consider the broader drivers of adolescent IPV and ensure intersectionality informed approaches to maximise their potential to capitalise on this window of opportunity.

A quality of life supports model: Six research-focused steps to evaluate the model and enhance research practices in the field of IDD

In the field of intellectual and developmental disabilities (IDD) the constructs of quality of life and supports have been combined into a Quality of Life Supports Model (QOLSM) that is currently used internationally for supports provision, organization transformation, and systems change. With the model's increasing and widespan use in research and practice, there is a need to evaluate the model's impact at the individual, organization, and systems levels. The purpose of this article is to outline six specific research-focused evaluation steps that allow researchers to evaluate the model and thereby enhance research practices is the field of IDD. These steps involve: (1) operationalizing components of the QOLSM; (2) relating QOLSM components to the type of research planned and intended outcome indicators; (3) gathering evidence and establishing its credibility; (4) interpreting and communicating the results; (5) implementing research outcomes at the level of the microsystem, mesosystem and macrosystem; and (6) judging the impact of the model. These steps delineate a research framework that is based on a systematic approach to evidence-based practices and enhanced research practices in the field of IDD.

A cluster RCT and process evaluation of an implementation optimisation intervention to promote parental engagement enrolment and attendance in a childhood obesity prevention programme: results of the Optimising Family Engagement in HENRY (OFTEN) trial

Background

Poor and variable implementation of childhood obesity prevention programmes reduces their population impact and sustainability. We drew upon ethnographic work to develop a multi-level, theory-based implementation optimisation intervention. This intervention aimed to promote parental enrolment and attendance at HENRY (Health Exercise Nutrition for the Really Young), a UK community obesity prevention programme, by changing behaviours of children’s centre and local authority stakeholders.

Methods

We evaluated the effectiveness of the implementation optimisation intervention on HENRY programme enrolment and attendance over a 12-month implementation period in a cluster randomised controlled trial. We randomised 20 local government authorities (with 126 children’s centres) to HENRY plus the implementation optimisation intervention or to HENRY alone. Primary outcomes were (1) the proportion of centres enrolling at least eight parents per programme and (2) the proportion of centres with a minimum of 75% of parents attending at least five of eight sessions per programme. Trial analyses adjusted for stratification factors (pre-randomisation implementation of HENRY, local authority size, deprivation) and allowed for cluster design. A parallel mixed-methods process evaluation used qualitative interviews and routine monitoring to explain trial results.

Results

Neither primary outcome differed significantly between groups; 17.8% of intervention centres and 18.0% of control centres achieved the parent enrolment target (adjusted difference − 1.2%; 95% CI − 19.5%, 17.1%); 17.1% of intervention centres and 13.9% of control centres achieved the attendance target (adjusted difference 1.2%; 95% CI − 15.7%, 18.1%). Unexpectedly, the trial coincided with substantial national service restructuring, including centre closures and reduced funds. Some commissioning and management teams stopped or reduced delivery of both HENRY and the implementation optimisation intervention due to competing demands. Thus, at follow-up, HENRY programmes were delivered to approximately half the number of parents compared to baseline (n = 433 vs. 881).

Conclusions

During a period in which services were reduced by external policies, this first definitive trial found no evidence of effectiveness for an implementation optimisation intervention promoting parent enrolment to and attendance at an obesity prevention programme.

Trial registration

ClinicalTrials.govNCT02675699. Registered on 4 February 2016

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05757-w.

Developmental Evaluation: six ways to get a grip on the potential of education scholarship to serve innovation

In March 2020, COVID-19 challenged health and educational systems across the country. The rapid reallocation of resources to ensure public safety had taken priority over educational obligations. Healthcare students were removed from clinical environments as their learning came to a grinding halt. While academic institutions were pivoting and transforming teaching and learning experiences, students responded to the pandemic with innovation, attending to gaps in patient care. As educators, we must understand how we can further support students and faculty to unleash innovative thinking during a crisis. To begin to address this educational need, academic institutions now have an opportunity to broaden the practice of education scholarship in accordance with best practices to nurture innovation and innovative thinking. What framework can aid us in this endeavor? In times of instability, Developmental Evaluation is an approach that can support the implementation of innovations within medical education. Using an example of an innovation in medical education, we offer six practical tips to begin to use Developmental Evaluation to support and enable learners and faculty in the creation of innovations and contribute to a broader definition of education scholarship.

Enhancing research practices in intellectual and developmental disabilities through person-centered outcome evaluation

BACKGROUND

Research practices in IDD need to align current values towards people with IDD, the current understanding of IDD, and best practices regarding change strategies and valued outcomes.

AIMS

To describe the components to-and application of-a person-centered outcome evaluation model that meets the above criteria.

METHODS AND PROCEDURES

A person-centered evaluation logic model is used to identify and describe a conceptual framework (input), a change strategy (throughput), personal outcomes (outcome), and meaningful impacts (output).

OUTCOMES AND RESULTS

Specific person-centered outcomes and exemplary outcome indicators are presented for two outcome evaluation frameworks: human functioning dimensions, and the four theoretical perspectives on IDD: biomedical, psychoeducational, sociocultural, and justice.

CONCLUSIONS AND IMPLICATIONS

A person-centered approach to outcome evaluation enhances research practices in IDD by identifying and assessing valued personal outcomes that align current values, understanding, and best practices; increases transparency; facilitates accountability; and expands understanding.

What works for vulnerable families? Interpretations of effective health promotion

Under the umbrella of the Healthy Futures Nearby programme, 46 small-scale projects were funded to promote changes in health-related behaviours (smoking, alcohol, diet and exercise) and to improve perceived health among vulnerable families in the Netherlands. The evaluation of these health-related multiple project programmes is often based on funder-defined outcomes and strategies. However, within the funded projects, assumptions about improving the health of vulnerable families based on local knowledge and experiences will also shape the project outcomes and strategies. These additional outcomes and strategies are project-specific interpretations of effective health promotion. Knowing these interpretations is crucial for the policy related and scientific relevance of the evaluation. Therefore, we aimed to determine the interpretations of each project and how they translate into relevant inputs for the overall evaluation of the programme. Based on 46 semi-structured group interviews with local project stakeholders, we produced a list of assumptions about what health promotion for vulnerable families should look like and then identified five main clusters: (i) strategies of offering pre-defined, health (behaviour)-related activities to families, (ii) actively involving vulnerable families in the initiative, (iii) assumptions about how health promotion should start with or include non-health-related topics, (iv) assumptions on how one should build on what already exists in the local context of the families and (v) assumptions on the role of the (health) professional in health promotion among vulnerable families. These project interpretations of effective health promotion provide inputs and priorities for the HFN programme’s overall evaluation.

Developing a programme theory for a transdisciplinary research collaboration: Complex Urban Systems for Sustainability and Health

Background: Environmental improvement is a priority for urban sustainability and health and achieving it requires transformative change in cities. An approach to achieving such change is to bring together researchers, decision-makers, and public groups in the creation of research and use of scientific evidence. Methods: This article describes the development of a programme theory for Complex Urban Systems for Sustainability and Health (CUSSH), a four-year Wellcome-funded research collaboration which aims to improve capacity to guide transformational health and environmental changes in cities. Results: Drawing on ideas about complex systems, programme evaluation, and transdisciplinary learning, we describe how the programme is understood to "work" in terms of its anticipated processes and resulting changes. The programme theory describes a chain of outputs that ultimately leads to improvement in city sustainability and health (described in an 'action model'), and the kinds of changes that we expect CUSSH should lead to in people, processes, policies, practices, and research (described in a 'change model'). Conclusions: Our paper adds to a growing body of research on the process of developing a comprehensive understanding of a transdisciplinary, multiagency, multi-context programme. The programme theory was developed collaboratively over two years. It involved a participatory process to ensure that a broad range of perspectives were included, to contribute to shared understanding across a multidisciplinary team. Examining our approach allowed an appreciation of the benefits and challenges of developing a programme theory for a complex, transdisciplinary research collaboration. Benefits included the development of teamworking and shared understanding and the use of programme theory in guiding evaluation. Challenges included changing membership within a large group, reaching agreement on what the theory would be 'about', and the inherent unpredictability of complex initiatives.

Developing a programme theory for a transdisciplinary research collaboration: Complex Urban Systems for Sustainability and Health

Background: Environmental improvement is a priority for urban sustainability and health and achieving it requires transformative change in cities. An approach to achieving such change is to bring together researchers, decision-makers, and public groups in the creation of research and use of scientific evidence.

Methods: This article describes the development of a programme theory for Complex Urban Systems for Sustainability and Health (CUSSH), a four-year Wellcome-funded research collaboration which aims to improve capacity to guide transformational health and environmental changes in cities.

Results: Drawing on ideas about complex systems, programme evaluation, and transdisciplinary learning, we describe how the programme is understood to “work” in terms of its anticipated processes and resulting changes. The programme theory describes a chain of outputs that ultimately leads to improvement in city sustainability and health (described in an ‘action model’), and the kinds of changes that we expect CUSSH should lead to in people, processes, policies, practices, and research (described in a ‘change model’).

Conclusions: Our paper adds to a growing body of research on the process of developing a comprehensive understanding of a transdisciplinary, multiagency, multi-context programme. The programme theory was developed collaboratively over two years. It involved a participatory process to ensure that a broad range of perspectives were included, to contribute to shared understanding across a multidisciplinary team. Examining our approach allowed an appreciation of the benefits and challenges of developing a programme theory for a complex, transdisciplinary research collaboration. Benefits included the development of teamworking and shared understanding and the use of programme theory in guiding evaluation. Challenges included changing membership within a large group, reaching agreement on what the theory would be ‘about’, and the inherent unpredictability of complex initiatives.

Understanding the complexity of population health interventions: assessing intervention system theory (ISyT)

Given their inherent complexity, we need a better understanding of what is happening inside the “black box” of population health interventions. The theory-driven intervention/evaluation paradigm is one approach to addressing this question. However, barriers related to semantic or practical issues stand in the way of its complete integration into evaluation designs. In this paper, we attempt to clarify how various theories, models and frameworks can contribute to developing a context-dependent theory, helping us to understand the black box of population health interventions and to acknowledge their complexity. To achieve this goal, we clarify what could be referred to as “theory” in the theory-driven evaluation of the interventional system, distinguishing it from other models, frameworks and classical theories. In order to evaluate the interventional system with a theory-driven paradigm, we put forward the concept of interventional system theory (ISyT), which combines a causal theory and an action model. We suggest that an ISyT could guide evaluation processes, whatever evaluation design is applied, and illustrate this alternative method through different examples of studies. We believe that such a clarification can help to promote the use of theories in complex intervention evaluations, and to identify ways of considering the transferability and scalability of interventions.

Mechanisms of impact and experiences of a person-centred transition programme for adolescents with CHD: the Stepstones project

Background

During the past decade there has been some evaluation of transition programmes for adolescents with chronic conditions. However, this has rarely involved process evaluations focusing on mechanisms leading to outcomes, thus hampering implementation of these complex interventions. Our aim was to (I) describe adolescents’ and parents’ experiences of participating in a person-centred transition programme aiming to empower them in transition to adulthood and (II) explore the mechanisms of impact.

Methods

A qualitative process evaluation was performed, embedded in a randomized controlled trial evaluating the effectiveness of a transition programme for adolescents with congenital heart disease in Sweden. A purposive sample of 14 adolescents and 12 parents randomized to the intervention group were interviewed after participation in the programme. Data were analysed deductively and inductively in NVivo v12.

Results

Experiences of participation in the transition programme were generally positive. Meeting a transition coordinator trained in person-centred care and adolescent health and embarking on an educational process based on the adolescents’ prerequisites in combination with peer support were considered key change mechanisms. However, support to parents were not sufficient for some participants, resulting in ambivalence about changing roles and the unmet needs of parents who required additional support.

Conclusions

Participants experienced increased empowerment in several dimensions of this construct, thus demonstrating that the transition programme was largely implemented as intended and the evidence-based behaviour-change techniques used proved effective in reaching the outcome. These findings can inform future implementation of transition programmes and illuminate challenges associated with delivering a complex intervention for adolescents with chronic conditions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06567-1.

A Theory of Change for Web-Based Therapy and Support Services for Children and Young People: Collaborative Qualitative Exploration

BACKGROUND

Web-based counseling and support has become increasingly commonplace for children and young people (CYP). Currently, there is limited research that focuses on the mechanisms of change within complex telepsychology platforms, a factor that makes designing and implementing outcome measures challenging.

OBJECTIVE

This project aims to articulate a theory of change (ToC) for Kooth, a web-based therapy and support platform for CYP.

METHODS

A collaborative qualitative research design involving professional staff, academic partners, and young people was used to develop the ToC. The following three major reflective phases were engaged: a scoping workshop involving professional staff and academic partners, a series of explorative projects were completed to inform the development of the ToC, and the draft ToC was reviewed for coherence by key stakeholders (young people, online professionals, and service managers).

RESULTS

A collaboratively developed ToC was presented. This was divided into the conditions that lead to individuals wanting to access web-based therapy and support (eg, individuals wanting support there and then or quickly), the mode of service delivery (eg, skilled and experienced professionals able to build empathetic relationships with CYP), and the observed and reported changes that occur as a consequence of using the service (eg, individuals being better able to manage current and future situations).

CONCLUSIONS

Developing the ToC helps to shed light on how web-based therapy and support services aid the mental health and well-being of CYP. Furthermore, it helps to understand the development of positive virtual ecosystems and can be used to devise evaluative tools for CYP telepsychology providers.

Generating Evidence From Contextual Clinical Research in Low- to Middle Income Countries: A Roadmap Based on Theory of Change

Along with inadequate access to high-quality care, competing health priorities, fragile health systems, and conflicts, there is an associated delay in evidence generation and research from LMICs. Lack of basic epidemiologic understanding of the disease burden in these regions poses a significant knowledge gap as solutions can only be developed and sustained if the scope of the problem is accurately defined. Congenital heart disease (CHD), for example, is the most common birth defect in children. The prevalence of CHD from 1990 to 2017 has progressively increased by 18.7% and more than 90% of children with CHD are born in Low and Middle-Income Countries (LMICs). If diagnosed and managed in a timely manner, as in high-income countries (HICs), most children lead a healthy life and achieve adulthood. However, children with CHD in LMICs have limited care available with subsequent impact on survival. The large disparity in global health research focus on this complex disease makes it a solid paradigm to shape the debate. Despite many challenges, an essential aspect of improving research in LMICs is the realization and ownership of the problem around paucity of local evidence by patients, health care providers, academic centers, and governments in these countries. We have created a theory of change model to address these challenges at a micro- (individual patient or physician or institutions delivering health care) and a macro- (government and health ministries) level, presenting suggested solutions for these complex problems. All stakeholders in the society, from government bodies, health ministries, and systems, to frontline healthcare workers and patients, need to be invested in addressing the local health problems and significantly increase data to define and improve the gaps in care in LMICs. Moreover, interventions can be designed for a more collaborative and effective HIC-LMIC and LMIC-LMIC partnership to increase resources, capacity building, and representation for long-term productivity.

Developing a Theory of Change model of service user and caregiver involvement in mental health system strengthening in primary health care in rural Ethiopia

BACKGROUND

The involvement of service users and caregivers is recommended as a strategy to strengthen health systems and scale up quality mental healthcare equitably, particularly in low-and-middle-income countries. Service user and caregiver involvement is complex, and its meaningful implementation seems to be a worldwide challenge. Theory of Change (ToC) has been recommended to guide the development, implementation and evaluation of such complex interventions. This paper aims to describe a ToC model for service user and caregiver involvement in a primary mental health care in rural Ethiopia.

METHODS

The ToC was developed in two workshops conducted in (i) Addis Ababa with purposively selected psychiatrists (n = 4) and multidisciplinary researchers (n = 3), and (ii) a rural district in south-central Ethiopia (Sodo), with community stakeholders (n = 24). Information from the workshops (provisional ToC maps, minutes, audio recordings), and inputs from a previous qualitative study were triangulated to develop the detailed ToC map. This ToC map was further refined with written feedback and further consultative meetings with the research team (n = 6) and community stakeholders (n = 35).

RESULTS

The experiential knowledge and professional expertise of ToC participants combined to produce a ToC map that incorporated key components (community, health organisation, service user and caregiver), necessary interventions, preconditions, assumptions and indicators towards the long-term outcomes. The participatory nature of ToC by itself raised awareness of the possibilities for servicer user and caregiver involvement, promoted co-working and stimulated immediate commitments to mobilise support for a grass roots service user organization.

CONCLUSIONS

The ToC workshops provided an opportunity to co-produce a ToC for service user and caregiver involvement in mental health system strengthening linked to the planned model for scale-up of mental health care in Ethiopia. The next steps will be to pilot a multi-faceted intervention based on the ToC and link locally generated evidence to published evidence and theories to refine the ToC for broader transferability to other mental health settings.

Realist evaluation of a theory-based life skills programme aiming to prevent addictive behaviours in adolescents: the ERIEAS study protocol

INTRODUCTION

Adolescence is a sensitive life stage during which tobacco, alcohol and cannabis are used as ways to learn and adopt roles. There is a great deal of interest in substance use (SU) prevention programmes for young people that work to change representations of these products and help with mobilisation of life skills. Unfortunately, few existing programmes are evidence-based.In France, a programme called Expériences Animées (EA, Animated Experiences) has been developed, inspired by life skills development programmes that have been proven to be successful. The EA programme uses animated short movies and talks with high school and secondary school pupils about the use of psychoactive substances and addictions. By allowing life skills mobilisation and modifying representations and beliefs about SU, it is aimed at delaying initiation of use of psychoactive substances, preventing adolescents from becoming regular consumers, reducing the risks and harms related to the use of these substances and opening the way for adapted support measures.We are interested in understanding how, under what circumstances, through which mechanisms and among which adolescents the EA programme works. Therefore, we have developed the ERIEAS study ('Evaluation Réaliste de l'Intervention Expériences Animées en milieu Scolaire'; Realist Evaluation of the EA Intervention in Schools).

METHODS AND ANALYSIS

EA will be conducted in 10 schools. A multi-case approach will be adopted with the aim of developing and adjusting an intervention theory. The study comes under the theory-driven evaluation framework. The investigation methodology will include four stages: (i) elaboration of a middle-range theory; (ii) data collection for validating/adjusting the theory; (iii) data analysis; and (iv) refinement and adjustment of the middle-range theory and definition of the programme's key functions.

ETHICS AND DISSEMINATION

The study will provide evidence-based results to health authorities to help in the rollout of health promotion strategies in schools. It will provide knowledge about the strategic configurations most suitable for leading to life skills mobilisation and change young people's representations about SU. The project will be carried out with full respect of current relevant legislation (eg, the Charter of Fundamental Rights of the European Union) and international conventions (eg, Helsinki Declaration). It follows the relevant French legislation of the research category on interventional research protocol involving the human person. The protocol was approved by the Comité et Protection des Personnes (CPP), that is, Committee for the Protection of Persons CPP SUD-EST VI n°: AU 1525 and was reported to the Agence Française de Sécurité Sanitaire des Produits de Santé (ANSM) that is, the French National Agency for the Safety of Health Products. It is in conformity with reference methodology MR003 of Bordeaux University Hospital (CNIL n° 2 026 779 v0).Trial registration detailsThis research has been registered on ClinicalTrials.gov (No. NCT04110626).The research project is registered in the European database ID-RCB (No. 2019-A01003-54).

Using implementation science theories and frameworks in global health

In global health, researchers and decision makers, many of whom have medical, epidemiology or biostatistics background, are increasingly interested in evaluating the implementation of health interventions. Implementation science, particularly for the study of public policies, has existed since at least the 1930s. This science makes compelling use of explicit theories and analytic frameworks that ensure research quality and rigour. Our objective is to inform researchers and decision makers who are not familiar with this research branch about these theories and analytic frameworks. We define four models of causation used in implementation science: intervention theory, frameworks, middle-range theory and grand theory. We then explain how scientists apply these models for three main implementation studies: fidelity assessment, process evaluation and complex evaluation. For each study, we provide concrete examples from research in Cuba and Africa to better understand the implementation of health interventions in global health context. Global health researchers and decision makers with a quantitative background will not become implementation scientists after reading this article. However, we believe they will be more aware of the need for rigorous implementation evaluations of global health interventions, alongside impact evaluations, and in collaboration with social scientists.

A participatory method to assess the contribution of agricultural research to societal changes in developing countries

Over the last decade, societal pressure has increased on public research agencies to provide evidence that their research contributes to achieve development goals. To address this challenge, the French Agricultural Research Center for International Development (Cirad), developed an ex-post impact assessment method based on the impact pathway approach, called Impact of Research in the South (ImpresS). The reconstruction of the impact pathway identifies and assesses research outputs, outcomes, and impacts on development over long-time spans, taking into account the contribution of other stakeholders, projects, and contextual factors. By applying mixed methods and participatory approaches, ImpresS involves key actors in assessing the contribution of research to impacts in innovation processes. Such a participatory approach raises, however, questions about the advantages and disadvantages of participation in impact assessment. This article examines whether and how participation affects the results of an evaluation and the methodological challenges this poses. The analysis is based on 13 case studies covering different innovation processes, countries, and time spans. The main results show that participation, combined with triangulation of information including quantitative and qualitative data, strengthens the explanation of the causal relationships among outputs, outcomes, and impacts. It helps reveal a large and diversified list of impacts based on the perception of actors, especially uncovering positive and negative impacts unexpected by researchers. However, participation may render other impacts less visible, especially those related to environmental and political issues. Furthermore, participatory evaluation entangles challenges linked to expectations, divergent perceptions, power dynamics, and social inequality, which must be carefully addressed to provide robust and transparent evaluation results.

Participatory action research to pilot a model of mental health service user involvement in an Ethiopian rural primary healthcare setting: study protocol

BACKGROUND

Involvement of service-users at all levels of the mental health system is a policy imperative in many countries internationally. However, putting policy into practice seems complex; little is known about how best to involve service users and efforts are often criticized for being tokenistic. In low-and-middle income countries, less attention has been given to the roles of service users within mental health systems. The proposed study is part of a larger project intended to develop service-user involvement in mental health system strengthening in Ethiopia. A Theory of Change (ToC) model has already been developed through a participatory approach. This study protocol aims to describe the theoretical background and methods to pilot this model using participatory action research (PAR) and explore participants' experience of involvement.

METHODS

The proposed study will apply a PAR approach situated in critical social theory and conduct a phenomenological case study to find out participants' experience of involvement. This will be conducted in three stages. The focus of Stage 1 will be to(i) establish a Research Advisory Group (RAG), and Research Participant Group (RPG) at district and primary healthcare facility levels, respectively, and (ii) identify and prioritize potential areas of concern for involvement in the domains of advocacy, service planning and development, monitoring and improving service quality. In Stage 2, we will work with the RPG to develop a plan of action for the selected area. Stage 3 will aim to assist the RPG to implement and evaluate the plan of action. Process indicators and observation will be combined with in-depth interviews with participants to elicit their experiences of involvement. Thematic content analysis will be used.

DISCUSSION

The participatory approach to mental health service user involvement in health system strengthening employed by this study will support the implementation of solutions through locally relevant and contextualized actions. Findings from this study will contribute to the body of knowledge towards understanding the complexity of implementation of service user involvement and refine the ToC model for transferability to similar settings.

Evaluation of an English language peer tutoring intervention

Purpose

Desertion rates in Colombian universities remain unacceptably high. In the field of foreign languages, academic failure is particularly concerning since English language instruction is compulsory in most universities. To address the issue of poor student performance and high dropout rates, the University of Colombia has set up a peer tutoring scheme (PTS) for English as a foreign language (EFL) students in order to inform programme development. The paper aims to discuss this issue.

Design/methodology/approach

The study was informed by realist evaluation principles. Qualitative in-depth interviews were conducted with eight PTS stakeholders supplemented by documentary analysis of the programme’s publicity material on the PTS website. The data were analysed using thematic analysis.

Findings

Findings reveal discrepancies between the “espoused theory” about how the programme operates and the “theory-in-use”. In particular, according to stakeholders, the programme does not appear to be used by many of those EFL students who would benefit from it, which suggests that the programme is not as effective as it could be. Student and teacher contextual factors and mechanisms may explain the reasons for issues with programme effectiveness.

Research limitations/implications

Formative evaluations such as the current study can provide rich contextual information, but cannot be generalised to other settings. Also, this study does not explore the perspective of peer tutors and tutees, which means key variables may have been overlooked. Further research into the perspectives of tutors and tutees would therefore be needed to firm up these conclusions.

Practical implications

Due to the scarcity of literature into EFL peer tutoring interventions in higher education (HE), it is hoped that these findings will have relevance for similar contexts. The current evaluation highlights the influence of contextual factors such as willingness to ask for help, student motivation, student priorities, tutor credibility, teacher workload, timetabling and scheduling issues and involvement from teachers on the success of open-access peer tutoring programmes for EFL students in higher educational settings.

Originality/value

As far the researcher is aware, this is the first evaluation of an EFL peer tutoring programme in a private HE context in Colombia, and one of only a handful of studies into EFL peer tutoring programmes. The findings therefore have implications for those working in similar contexts.

Using theory of change to develop an intervention theory for designing and evaluating behavior change SDApps for healthy eating and physical exercise: the OCAPREV theory

Background

Connected health devices and applications (referred to hereafter as “SDApps” - Smart devices and applications) are being portrayed as a new way for prevention, with the promise of accessibility, effectiveness and personalization. Many effectiveness evaluations (experimental designs) with strong internal validity exist. While effectiveness does appear to vary, the mechanisms used by these devices have not yet been thoroughly investigated. This article seeks to unpack this black box, and describes the process of elaboration of an intervention theory for healthy eating and physical activity SDApps. It includes a set of requirements relative to their impact on social health inequalities.

Methods

To build this theory, we drew on theory-driven approaches and in particular on the theory of change (ToC) method. To this end, we developed a cumulative and iterative process combining scientific data from the literature with knowledge from experts (researchers and practitioners) and from patients or users. It was a 3-step process, as follows: 1 - identifying the evidence base; 2 - developing the theory through design intervention and creating realistic expectations, including in our case specific work on social health inequalities (SHIs); 3 - modeling process and outcome.

Results

We produced an evidence-based theory according to the ToC model, based on scientific evidence and knowledge from experts and users. It sets out a causal pathway leveraging 11 key mechanisms - theoretical domains - with which 50 behavior change techniques can be used towards 3 ultimate goals: Capacity, Opportunity, Motivation – Behavior (COM-B). Furthermore, the theory specifically integrates requirements relative to the impact on SHIs.

Conclusions

This theory is an aid to SDAapp design and evaluation and it can be used to consider the question of the possible impact of SDApps on the increase in inequalities. Firstly, it enables developers to adopt a more overarching and thorough approach to supporting behavior change, and secondly it encourages comprehensive and contributive evaluations of existing SDApps. Lastly, it allows health inequalities to be fully considered.

Conceptualizing the societal impact of research in terms of elements of logic models: a survey of researchers in sub-Saharan Africa

This study addressed the conceptualization of the societal impact of research from the perspective of programme evaluation, by focusing on the three ‘result’ elements of logic models: outputs, outcomes, and impact. In research evaluation, the distinction could resemble a difference between product, use, and benefit. The study established whether researchers in sub-Saharan Africa (SSA), South Africa excluded, view societal impact as extending across all three elements or as confined to the last element only. A web survey of 485 SSA researchers was conducted, as researchers from this region are not yet bounded by policy definitions of impact. The survey imposed the three elements of logic models onto five hypothetical descriptions of ‘impactful’ research initiatives. Respondents rated each element in terms of how much it reflects the societal impact of research. For any initiative, use was more likely to be considered a strong example of societal impact compared to a product, but less likely so compared to benefit. Between 23% and 43% of respondents rated all three elements as strong examples of the societal impact of research. Responses were analyzed by SSA region and the research domain and years of research experience of survey participants. An open-ended question about own understandings of societal impact was included as well in the survey. The responses portrayed impact as a (generally) positive effect that contributes to change in the daily life of human kind. The expectation that research should have impact at an almost general level of aggregation could be unique to the SSA context.

Evaluation of health promotion capacity gains in a state-wide rural food literacy intervention

Building capacity to enhance health promotion intervention effectiveness is a desirable, difficult to achieve and rarely evaluated aspect of practice. This study aims to describe an approach for evaluating capacity building embedded in a state-wide health promotion intervention that had a primary objective of enhancing food literacy and secondary objective of building health promotion capacity. The multi-strategy rural food literacy intervention centred on a group-based, hands-on learning workshop series. Logic modelling, theory of change and clarification of the explicit assumption of the intervention relating to food literacy and capacity building objectives were documented. The evaluation approach acknowledged the complexity of the intervention that utilised a food literacy program as a vessel to build relationships, enabling capacity-building strategies over time, and used multiple forms of data to measure organisational, community and individual capacity. The development of a shared mixed method program evaluation plan was achieved through co-design. One-hundred and twenty-two peer facilitators were trained across 81 communities and reported increases in self-efficacy for health promotion action. There was broad awareness and support for the program within the organisation. The majority (75%) of communities involved in the intervention implemented at least one health promotion activity, demonstrating some health promotion capacity gain.

Community-Based Watershed Change: A Case Study in Eastern Congo

Conflict and environmental degradation in the Democratic Republic of the Congo are interrelated and complex. The authors conducted a case study of a community-based environmental restoration project in Eastern Congo and provide early results which suggest a link between community environmental action and multidimensional outcomes such as peace and reconciliation. The project examined in this study is based on a framework (Theory of Change) which networks communities through autonomous savings groups, churches, mosques, schools, and a community leadership network with the goal of catalyzing sustainable farming, reforestation, and community forest management. The primary project input was training, and the resulting voluntary community action included tree planting and the management of common forest areas. A mixed-methods approach was used to evaluate project results comparing two watersheds, and included a difference in differences analysis, participatory workshops, remote sensing analysis, and community activity reports. Positive change was observed in the treatment watershed in terms of ecosystem health and household economic condition. Results suggest a possible influence on peace conditions which, while fragile, offers hope for continued restorative action by communities. This study provides evidence that a community-based approach to environmental restoration may have a positive influence on multidimensional issues such as forests, watershed health, economic well-being, and peace.

The feasibility of early pulmonary rehabilitation and activity after COPD exacerbations: external pilot randomised controlled trial, qualitative case study and exploratory economic evaluation

BACKGROUND

Chronic obstructive pulmonary disease (COPD) affects > 3 million people in the UK. Acute exacerbations of COPD (AECOPD) are the second most common reason for emergency hospital admission in the UK. Pulmonary rehabilitation is usual care for stable COPD but there is little evidence for early pulmonary rehabilitation (EPR) following AECOPD, either in hospital or immediately post discharge.

OBJECTIVE

To assess the feasibility of recruiting patients, collecting data and delivering EPR to patients with AECOPD to evaluate EPR compared with usual care.

DESIGN

Parallel-group, pilot 2 × 2 factorial randomised trial with nested qualitative research and an economic analysis.

SETTING

Two acute hospital NHS trusts. Recruitment was carried out from September 2015 to April 2016 and follow-up was completed in July 2016.

PARTICIPANTS

Eligible patients were those aged ≥ 35 years who were admitted with AECOPD, who were non-acidotic and who maintained their blood oxygen saturation level (SpO₂) within a prescribed range. Exclusions included the presence of comorbidities that affected the ability to undertake the interventions.

INTERVENTIONS

(1) Hospital EPR: muscle training delivered at the patient's hospital bed using a cycle ergometer and (2) home EPR: a pulmonary rehabilitation programme delivered in the patient's home. Both interventions were delivered by trained physiotherapists. Participants were allocated on a 1 : 1 : 1 : 1 ratio to (1) hospital EPR (n = 14), (2) home EPR (n = 15), (3) hospital EPR and home EPR (n = 14) and (4) control (n = 15). Outcome assessors were blind to treatment allocation; it was not possible to blind patients.

MAIN OUTCOME MEASURES

Feasibility of recruiting 76 participants in 7 months at two centres; intervention delivery; views on intervention/research acceptability; clinical outcomes including the 6-minute walk distance (6WMD); and costs. Semistructured interviews with participants (n = 27) and research health professionals (n = 11), optimisation assessments and an economic analysis were also undertaken.

RESULTS

Over 7 months 449 patients were screened, of whom most were not eligible for the trial or felt too ill/declined entry. In total, 58 participants (76%) of the target 76 participants were recruited to the trial. The primary clinical outcome (6MWD) was difficult to collect (hospital EPR, n = 5; home EPR, n = 6; hospital EPR and home EPR, n = 5; control, n = 5). Hospital EPR was difficult to deliver over 5 days because of patient discharge/staff availability, with 34.1% of the scheduled sessions delivered compared with 78.3% of the home EPR sessions. Serious adverse events were experienced by 26 participants (45%), none of which was related to the interventions. Interviewed participants generally found both interventions to be acceptable. Home EPR had a higher rate of acceptability, mainly because patients felt too unwell when in hospital to undergo hospital EPR. Physiotherapists generally found the interventions to be acceptable and valued them but found delivery difficult because of staffing issues. The health economic analysis results suggest that there would be value in conducting a larger trial to assess the cost-effectiveness of the hospital EPR and hospital EPR plus home EPR trial arms and collect more information to inform the hospital cost and quality-adjusted life-year parameters, which were shown to be key drivers of the model.

CONCLUSIONS

A full-scale randomised controlled trial using this protocol would not be feasible. Recruitment and delivery of the hospital EPR intervention was difficult. The data obtained can be used to design a full-scale trial of home EPR. Because of the small sample and large confidence intervals, this study should not be used to inform clinical practice.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN18634494.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 22, No. 11. See the NIHR Journals Library website for further project information.

Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria

Logic models (LMs) have been used in programme evaluation for over four decades. Current debate questions the ability of logic modelling techniques to incorporate contextual factors into logic models. We share experience of developing a logic model within an ongoing realist evaluation which assesses the extent to which, and under what circumstances a community health workers (CHW) programme promotes access to maternity services in Nigeria. The article contributes to logic modelling debate by: i) reflecting on how other scholars captured context during LM development in theory-driven evaluations; and ii) explaining how we explored context during logic model development for realist evaluation of the CHW programme in Nigeria. Data collection methods that informed our logic model development included documents review, email discussions and teleconferences with programme stakeholders and a technical workshop with researchers to clarify programme goals and untangle relationships among programme elements. One of the most important findings is that, rather than being an end in itself, logic model development is an essential step for identifying initial hypotheses for tentative relevant contexts, mechanisms and outcomes (CMOs) and CMO configurations of how programmes produce change. The logic model also informed development of a methodology handbook that is guiding verification and consolidation of underlying programme theories.

Protocol for a mixed methods realist evaluation of regional District Health Board groupings in New Zealand

INTRODUCTION

Achieving effective integration of healthcare across primary, secondary and tertiary care is a key goal of the New Zealand (NZ) Health Strategy. NZ's regional District Health Board (DHB) groupings are fundamental to delivering integration, bringing the country's 20 DHBs together into four groups to collaboratively plan, fund and deliver health services within their defined geographical regions. This research aims to examine how, for whom and in what circumstances the regional DHB groupings work to improve health service integration, healthcare quality, health outcomes and health equity, particularly for Māori and Pacific peoples.

METHODS AND ANALYSIS

This research uses a mixed methods realist evaluation design. It comprises three linked studies: (1) formulating initial programme theory (IPT) through developing programme logic models to describe regional DHB working; (2) empirically testing IPT through both a qualitative process evaluation of regional DHB working using a case study design; and (3) a quantitative analysis of the impact that DHB regional groupings may have on service integration, health outcomes, health equity and costs. The findings of these three studies will allow refinement of the IPT and should lead to a programme theory which will explain how, for whom and in what circumstances regional DHB groupings improve service integration, health outcomes and health equity in NZ.

ETHICS AND DISSEMINATION

The University of Otago Human Ethics Committee has approved this study. The embedding of a clinician researcher within a participating regional DHB grouping has facilitated research coproduction, the research has been jointly conceived and designed and will be jointly evaluated and disseminated by researchers and practitioners. Uptake of the research findings by other key groups including policymakers, Māori providers and communities and Pacific providers and communities will be supported through key strategic relationships and dissemination activities. Academic dissemination will occur through publication and conference presentations.

Communities for healthy living (CHL) - A family-centered childhood obesity prevention program integrated into Head Start services: Study protocol for a pragmatic cluster randomized trial

BACKGROUND

Childhood obesity is highly prevalent and carries substantial health consequences. Childhood obesity interventions have had mixed results, which may be partially explained by the absence of theory that incorporates broader family context and methods that address implementation challenges in low-resource settings. Communities for Healthy Living (CHL) is an obesity prevention program for Head Start preschools designed with careful focus on theory and implementation. This protocol paper outlines the design, content, implementation, and evaluation of CHL.

METHODS/DESIGN

CHL integrates a parenting program co-led by Head Start staff and parents, enhanced nutrition support, and a media campaign. CHL content and implementation are informed by the Family Ecological Model, Psychological Empowerment Theory, and Organizational Empowerment Theory. The intervention is directed by community-based participatory research and implementation science principles, such as co‑leadership with parents and staff, and implementation in a real world context. CHL is evaluated in a three-year pragmatic cluster-randomized trial with a stepped wedge design. The primary outcome is change in child Body Mass Index z-score. Secondary outcomes include children's weight-related behaviors (i.e., diet, physical activity, screen use, and sleep), parenting practices targeted at these behaviors (e.g., food parenting), and parent empowerment. The evaluation capitalizes on routine health data collected by Head Start (e.g., child height and weight, diet) coupled with parent surveys completed by subsamples of families.

DISCUSSION

CHL is an innovative childhood obesity prevention program grounded in theory and implementation science principles. If successful, CHL is positioned for sustained implementation and nationwide Head Start scale-up.