Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

Health and wellbeing of survivors of sexual violence and abuse attending sexual assault referral centres in England: the MESARCH mixed-methods evaluation

Background

One million people in England and Wales experience sexual violence and abuse each year, with nearly half experiencing serious sexual offences; around 30,000 survivors access sexual assault referral centres.

Objectives

This research was commissioned by National Institute for Health and Care Research to evaluate access, interventions and care pathways for survivors, especially those provided through sexual assault referral centres.

Design, setting, participants

The sexual assault referral centres care pathway was investigated through six sub-studies. There were two Cochrane Reviews (4274 participants). Seventy-two providers and 5 survivors were interviewed at eight sites; the children and young people study involved 12 participants from two sexual assault referral centres. A cohort study involving three-wave data collection over 1 year (21 sites; 2602 service users screened, 337 recruited) used a multilevel modelling framework to explore risk factors for burden of post-traumatic stress disorder symptoms at baseline and change at 1 year. We analysed costs and outcomes and conducted a narrative analysis (41 survivors). We worked closely with survivors and prioritised the safety/welfare of participants and researchers.

Results

Cochrane Reviews identified large effects from psychosocial interventions for post-traumatic stress disorder and depression. Sexual assault referral centres delivered a high-quality frontline service for survivors but groups experiencing domestic abuse and some ethnic and cultural minorities were under-represented. The qualitative research emphasised inter-agency collaboration for survivor benefit. The cohort study identified a risk 'triad' of adverse childhood experiences, poor mental health and economic deprivation, which was associated with baseline trauma burden. There were important improvements in trauma symptoms a year later. These improvements were unrelated to different sexual assault referral centre models. Costs and other outcomes were also similar across models. Harmful policing and justice practices/procedures were identified by 25% of participants. In this context, trauma-competent interviewing techniques, regular/timely updates and conveying case decisions with care signalled good practice.

Limitations

The cohort study lacked a comparison group, reducing confidence in the finding that access to sexual assault referral centres explained the reduction observed in post-traumatic stress disorder.

Conclusions and future work

Barriers to access call for concerted efforts to implement trauma-informed universal health services. The risk 'triad' underscores the value of holistic approaches to care at sexual assault referral centres and timely follow-on care. Poor mental health was the main barrier to service access beyond sexual assault referral centres. The persistence of trauma symptoms a year after accessing sexual assault referral centres signals urgent need for tackling counselling wait-lists, expanding support options and commitment to lifelong care. Multidisciplinary evaluation of sexual assault referral centres for better health provides a foundation for advancing trauma-informed practices in the context of sexual violence and abuse.

Study registration

This study is registered as ISRCTN30846825 https://doi.org/10.1186/ISRCTN30846825.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: 16/117/04) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 35. See the NIHR Funding and Awards website for further award information.

Implementation of health-promoting retail initiatives in the Healthier Choices in Supermarkets Study-qualitative perspectives from a feasibility study

BACKGROUND

Improving food environments like supermarkets has the potential to affect customers' health positively. Scholars suggest researchers and retailers collaborate closely on implementing and testing such health-promoting interventions, but knowledge of the implementation of such interventions is limited. We explore the implementation of four health-promoting food retail initiatives selected and developed by a partnership between a research institution, a large retail group, and a non-governmental organisation.

METHODS

The four initiatives included downsizing of bags for pick'n' mix sweets and soda bottles at the check-out registers, shelf tags promoting healthier breakfast cereal options, and replacing a complimentary bun with a banana offered to children. The initiatives were implemented for 6 weeks (or longer if the store manager allowed it) in one store in Copenhagen, Denmark. Data were collected through observations, informal interviews with customers, and semi-structured interviews with retailers. We conducted a thematic analysis of transcripts and field notes inspired by process evaluation concepts and included quantitative summaries of selected data.

RESULTS

Two out of four initiatives were not implemented as intended. The implementation was delayed due to delivery issues, which also resulted in soda bottles not being downsized as intended. The maintenance of the shelf tags decreased over time. Retailers expressed different levels of acceptability towards the initiatives, with a preference for the complimentary banana for children. This was also the only initiative noticed by customers with both positive and negative responses. Barriers and facilitators of implementation fell into three themes: Health is not the number one priority, general capacity of retailers, and influence of customers and other stakeholders on store operation.

CONCLUSIONS

The retailers' interests, priorities, and general capacity influenced the initiative implementation. Retailers' acceptability of the initiatives was mixed despite their involvement in the pre-intervention phase. Our study also suggests that customer responses towards health-promoting initiatives, as well as cooperation with suppliers and manufacturers in the development phase, may be determining to successful implementation. Future studies should explore strategies to facilitate implementation, which can be applied prior to and during the intervention.

Healthcare quality improvement: It's time to update the Donabedian approach with a complex systems perspective

There is broad consensus that healthcare systems are complex systems, which as a result, face complex problems. From this perspective, quality of care can be conceptualised as an emergent outcome of the healthcare system, that is more than the sum of individual components of care (in terms of inputs and processes), and quality improvement as a complex systems problem. However, traditional approaches, such as Donabedian's structure/process/outcome framework, are rooted in a linear, reductionist perspective, that fails to recognise that quality of care is created in the context of complex healthcare systems, and the many interactions and uncertainties at play that shape quality of care and health outcomes. A paradigm shift is needed from a reductionist to a systems thinking approach if we are to better understand and improve quality of care. Such a shift begins with asking different research questions, situated within the system context, that focus on identifying how interventions may contribute to system improvement, as opposed to seeking to directly link interventions with quality of care outcomes. In contrast to traditional healthcare quality measures focusing on single components of the system in isolation, research needs to explicitly consider quality of care as an emergent system outcome and identify new indicators and methods of assessment that provide insight into how the healthcare system functions as an interconnected whole. It is an opportune moment to harness the energy of the international healthcare quality movement to drive the innovation needed in research and practice to adopt a systems thinking approach.

Mapping the ripple effects of a compassionate university for serious illness, death, and bereavement

Background

Compassionate communities have been put forward as a promising model for community-based support for people facing serious illness, caregiving, dying, and loss. In particular, educational institutions are increasingly acknowledged as potential settings to function as compassionate schools and compassionate workplaces, cultivating acceptance and validation of these experiences beyond the university setting.

Objectives

This paper investigates the activities and outcomes of a compassionate community initiative-the Compassionate University program at the Vrije Universiteit Brussel in Belgium.

Design

Ripple Effects Mapping was used to guide the focus group and individual interviews conducted with core team members responsible for the development and implementation of the Compassionate University program.

Methods

During the focus group and individual interviews, the core team members reflected on the program contributions, with their narratives visually depicted via a hand-drawn mind map. Qualitative data derived from this mind map were entered into XMIND mapping software and fine-tuned based on the focus group and individual interview transcripts and additional project records.

Results

Thematic analysis identified four outcome areas that encapsulate the key contributions of the Compassionate University program: (i) increased acceptance and integration of topics such as serious illness, death, and bereavement into existing practices; (ii) broader support for and formalization of compassionate procedures and policies; (iii) emergence of informal networks and internal collaboration on the topics; and (iv) diffusion of compassionate ideas beyond the university.

Conclusion

The Compassionate University program facilitates a cultural shift within the university environment, fostering greater acceptance of integrating topics such as serious illness, death, and bereavement into existing practices. Additionally, compassionate procedures and policies for students and staff have been formalized, and core team members are increasingly called upon to provide support on these matters. Notably, Compassionate University stands out as one of the pioneering initiatives in Europe, attracting different educational institutions seeking guidance on cultivating a more compassionate environment.

How can health systems approach reducing health inequalities? An in-depth qualitative case study in the UK

BACKGROUND

Addressing socioeconomic inequalities in health and healthcare, and reducing avoidable hospital admissions requires integrated strategy and complex intervention across health systems. However, the understanding of how to create effective systems to reduce socio-economic inequalities in health and healthcare is limited. The aim was to explore and develop a system's level understanding of how local areas address health inequalities with a focus on avoidable emergency admissions.

METHODS

In-depth case study using qualitative investigation (documentary analysis and key informant interviews) in an urban UK local authority. Interviewees were identified using snowball sampling. Documents were retrieved via key informants and web searches of relevant organisations. Interviews and documents were analysed independently based on a thematic analysis approach.

RESULTS

Interviews (n = 14) with wide representation from local authority (n = 8), NHS (n = 5) and voluntary, community and social enterprise (VCSE) sector (n = 1) with 75 documents (including from NHS, local authority, VCSE) were included. Cross-referenced themes were understanding the local context, facilitators of how to tackle health inequalities: the assets, and emerging risks and concerns. Addressing health inequalities in avoidable admissions per se was not often explicitly linked by either the interviews or documents and is not yet embedded into practice. However, a strong coherent strategic integrated population health management plan with a system's approach to reducing health inequalities was evident as was collective action and involving people, with links to a "strong third sector". Challenges reported include structural barriers and threats, the analysis and accessibility of data as well as ongoing pressures on the health and care system.

CONCLUSION

We provide an in-depth exploration of how a local area is working to address health and care inequalities. Key elements of this system's working include fostering strategic coherence, cross-agency working, and community-asset based approaches. Areas requiring action included data sharing challenges across organisations and analytical capacity to assist endeavours to reduce health and care inequalities. Other areas were around the resilience of the system including the recruitment and retention of the workforce. More action is required to embed reducing health inequalities in avoidable admissions explicitly in local areas with inaction risking widening the health gap.

A decade of change towards Value-Based Health Care at a Dutch University Hospital: a complexity-informed process study

BACKGROUND

While healthcare organizations in several countries are embracing Value-Based Health Care (VBHC), there are limited insights into how to achieve this paradigm shift. This study examines the decade-long (2012-2023) change towards VBHC in a pioneering Dutch university hospital.

METHOD

Through retrospective, complexity-informed process research, we study how a Dutch university hospital's strategy to implement VBHC evolved, how implementation outcomes unfolded, and the underlying logic behind these developments. Data include the hospital's internal documents (n = 10,536), implementation outcome indicators (n = 4), a survey among clinicians (n = 47), and interviews with individuals contributing to VBHC at the hospital level (n = 20).

RESULTS

The change towards VBHC is characterized by three sequential strategies. Initially, the focus was on deep change through local, tailored implementation of multiple VBHC elements. The strategy then transitioned to a hospital-wide program aimed at evolutionary change on a large scale, emphasizing the integration of VBHC into mainstream IT and policies. Recognizing the advantages and limitations of both strategies, the hospital currently adopts a "hybrid" strategy. This strategy delicately combines deep and broad change efforts. The strategy evolved based on accumulated insights, contextual developments and shifts in decision-makers. The complexity of change was downplayed in plans and stakeholder communication. By the end of 2023, 68 (sub)departments engaged in VBHC, enabled to discuss patients' responses to Patient Reported Outcomes Measures (PROMs) during outpatient care. However, clinicians' use of PROMs data showed limitations. While pioneers delved deeper into VBHC, laggards have yet to initiate it.

CONCLUSIONS

VBHC does not lend itself to linear planning and is not easily scalable. While there appears to be no golden standard for implementation, blending local and larger-scale actions appears advantageous. Local, deep yet harmonized and system-integrated changes culminate in large scale transformation. Embracing complexity and focusing on the ultimate aims of (re)institutionalization and (re)professionalization are crucial.

Applying dynamic systems theory and complexity theory methods in psychotherapy research: A systematic literature review

OBJECTIVE

Dynamic systems theory and complexity theory (DST/CT) is a framework explaining how complex systems change and adapt over time. In psychotherapy, DST/CT can be used to understand how a person's mental and emotional state changes during therapy incorporating higher levels of complexity. This study aimed to systematically review the variability of DST/CT methods applied in psychotherapy research.

METHODS

A primary studies search was conducted in the EBSCO and Web of Knowledge databases, extracting information about the analyzed DST/CT phenomena, employed mathematical methods to investigate these phenomena, descriptions of specified dynamic models, psychotherapy phenomena, and other information regarding studies with empirical data (e.g., measurement granularity).

RESULTS

After screening 38,216 abstracts and 4,194 full texts, N = 41 studies published from 1990 to 2021 were identified. The employed methods typically included measures of dynamic complexity or chaoticity. Computational and simulation studies most often employed first-order ordinary differential equations and typically focused on describing the time evolution of client-therapist dyadic influences. Eligible studies with empirical data were usually based on case studies and focused on data with high time intensity of within-session dynamics.

CONCLUSION

This review provides a descriptive synthesis of the current state of the proliferation of DST/CT methods in the psychotherapy research field.

A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance

The UK Medical Research Council's widely used guidance for developing and evaluating complex interventions has been replaced by a new framework, commissioned jointly by the Medical Research Council and the National Institute for Health Research, which takes account of recent developments in theory and methods and the need to maximise the efficiency, use, and impact of research.

Navigating One Health in research-for-development: Reflections on the design and implementation of the CGIAR Initiative on One Health

Adopting One Health approaches is key for addressing interconnected health challenges. Yet, how to best put One Health into practice in research-for-development initiatives aiming to 'deliver impacts' remains unclear. Drawing on the CGIAR Initiative on One Health - a global initiative to address zoonotic diseases, antimicrobial resistance, and food and water safety - we reflect on challenges during program conception and implementation, prompting us to suggest improvements in multisectoral collaboration, coordination, and communication. Our approach involves conducting a researcher-centered process evaluation, comprising individual interviews that are subsequently thematically analyzed and synthesized. The key takeaway is that limited time for planning processes and short program timelines compared to envisioned development impacts may impede research-for-development efforts. Yet, collaborative work can be successful when adequate time and resources are allocated for planning with minimal disruption throughout implementation. Additionally, due to the multifaceted nature of One Health initiatives, it is important to pay attention to co-benefits and trade-offs, where taking action in one aspect may yield advantages and disadvantages in another, aiding to identify sustainable One Health development pathways. Forming close partnerships with national governments and local stakeholders is essential not only to promote sustainability but also to ensure local relevance, enhancing the potential for meaningful impact. Finally, regularly assessing progress toward development goals is critical as development stands as an overarching objective.

Interventions for improving critical care in low- and middle-income countries: a systematic review

PURPOSE

To systematically review the typology, impact, quality of evidence, barriers, and facilitators to implementation of Quality Improvement (QI) interventions for adult critical care in low- and middle-income countries (LMICs).

METHODS

MEDLINE, EMBASE, Cochrane Library and ClinicalTrials.gov were searched on 1st September 2022. The studies were included if they described the implementation of QI interventions for adult critical care in LMICs, available as full text, in English and published after 2000. The risks of bias were assessed using the ROB 2.0/ROBINS-I tools. Intervention strategies were categorised according to a Knowledge Translation framework. Interventions' effectiveness were synthesised by vote counting and assessed with a binomial test. Barriers and facilitators to implementation were narratively synthesised using the Consolidated Framework for Implementation Research.

RESULTS

78 studies were included. Risk of bias was high. The most common intervention strategies were Education, Audit & Feedback (A&F) and Protocols/Guidelines/Bundles/Checklists (PGBC). Two multifaceted strategies improved both process and outcome measures: Education and A&F (p = 0.008); and PGBC with Education and A&F (p = 0.001, p < 0.001). Facilitators to implementation were stakeholder engagement, organisational readiness for implementation, and adaptability of interventions. Barriers were lack of resources and incompatibility with clinical workflows.

CONCLUSIONS

The evidence for QI in critical care in LMICs is sparse and at high risk of bias but suggests that multifaceted interventions are most effective. Co-designing interventions with and engaging stakeholders, communicating relative advantages, employing local champions and adapting to feedback can improve implementation. Hybrid study designs, process evaluations and adherence to reporting guidelines would improve the evidence base.

The Healthy Start scheme in England "is a lifeline for families but many are missing out": a rapid qualitative analysis

BACKGROUND

Healthy Start (HS) is a government scheme in England, Wales and Northern Ireland that offers a financial payment card and free vitamins to families experiencing low income. Pregnant women and families with children < 4 years can use the HS card to buy fruit, vegetables, cow's milk, infant formula and pulses. HS was fully digitalised in March 2022. While digitalisation has improved the user experience for many families, in the context of the cost-of-living crisis and increasing dietary inequalities, it is important to understand why HS is not reaching more families. This study aimed to (i) assess the perceptions and experiences of HS from stakeholders across the system including those who promote, implement and are eligible for HS, and (ii) identify recommendations to improve the scheme's effectiveness and uptake.

METHODS

The study design was a post-implementation rapid qualitative evaluation using stakeholder interviews. Data were collected between January and June 2023 via semi-structured interviews (50% online; 50% in person) with 112 stakeholders, including parents (n = 59), non-government organisations (n = 13), retailers (n = 11) and health and community professionals (n = 29) at national and local levels. Findings were confirmed by a sub-sample of participants.

RESULTS

Six core themes cut across stakeholders' perceptions and experiences, and stakeholders collectively outlined seven recommendations they felt could be acted upon to maximise uptake and efficiency of HS, with actions at both national and local levels. A novel finding from this study is that raising awareness about HS alone is unlikely to result automatically or universally in higher uptake rate. Recommendations include: continuing to provide this scheme that is universally valued; the need for many families to be provided with a helping hand to successfully complete the application; reframing of the scheme as a child's right to food and development to ensure inclusivity; improved leadership, coordination and accountability at both national and local levels.

CONCLUSIONS

HS provides benefits for child development and family wellbeing. The study's recommendations should be actioned by national and local governments to enable all families eligible for the scheme to benefit from this nutritional safety net.

Effectiveness and cost-effectiveness of a sustainable obesity prevention programme for preschool children delivered at scale 'HENRY' (Health, Exercise, Nutrition for the Really Young): protocol for the HENRY III cluster randomised controlled trial

INTRODUCTION

One-fifth of children start school already overweight or living with obesity, with rates disproportionately impacting those living in the most deprived areas. Social, environmental and biological factors contribute to excess weight gain and programmes delivered in early years settings aim to support families to navigate these in order to prevent obesity. One of these programmes (Health, Exercise and Nutrition for the Really Young, HENRY) has been delivered in UK community venues (hereon named 'centres') in high deprivation areas since 2008 and aims to help families to provide a healthy start for their preschool children. We aim to establish the effectiveness and cost-effectiveness of HENRY, including its potential role from a wider systems perspective.

METHODS AND ANALYSIS

This is a multicentre, open-labelled, two-group, prospective, cluster randomised controlled trial, with cost-effectiveness analysis, systems-based process evaluation and internal pilot. Primary analysis will compare body mass index (BMI) z-score at 12 months in children (n=984) whose parents have attended HENRY to those who have not attended. Secondary outcomes include parent and staff BMI and waist circumference, parenting efficacy, feeding, eating habits, quality of life, resource use and medium term (3 years) BMI z-scores (child and siblings). 82 centres in ~14 local authority areas will be randomised (1:1) to receive HENRY or continue with standard practice. Intention-to-treat analysis will compare outcomes using mixed effects linear regression. Economic evaluation will estimate a within-trial calculation of cost-per unit change in BMI z-score and longer-term trajectories to determine lifelong cost savings (long-term outcomes). A systems process evaluation will explore whether (and how) implementation of HENRY impacts (and is impacted by) the early years obesity system. An established parent advisory group will support delivery and dissemination.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the University of York, Health Sciences' Research Governance Committee (HSRGC/2022/537/E). Dissemination includes policy reports, community resources, social media and academic outputs.

TRIAL REGISTRATION NUMBER

ISRCTN16529380.

Process evaluation of the Belgian one-month-without alcohol campaign 'Tournée Minérale': a mixed method approach

BACKGROUND

The Tournée Minérale campaign [TMC] is a mass media prevention campaign challenging Belgian adults to refrain from alcohol during one month. A process evaluation may help us better understand the effect of TMC and to formulate recommendations for future editions. The current study aimed to examine reach, experiences, perceived effectiveness and maintenance of TMC.

METHODS

A mixed method design was used to assess the process, using pre- and post-questionnaires (n = 49.022, 44.5 ± 13.1 years old, 37.0% men) and focus groups (n = 31, 47.3 ± 14.3 years old, 33.3% men).

RESULTS

Most campaign materials were considered useful and/or motivating. Facilitators for taking part in TMC were connectedness with other participants, stimulus control (e.g. removing alcohol at home) and a supportive social environment. Most difficulties were encountered with abstaining during social occasions as participants had to change a habit or find alternative non-alcoholic beverages. Participants reported both beneficial (e.g. sleeping better) and adverse effects (e.g. drinking more soft drinks).

CONCLUSIONS

Future editions of TMC should try to decrease perceived adverse effects (e.g. by providing attractive non-alcoholic beverages) and can benefit from having a forum where people can share experiences.

The current state of complex systems research on socioeconomic inequalities in health and health behavior-a systematic scoping review

BACKGROUND

Interest in applying a complex systems approach to understanding socioeconomic inequalities in health is growing, but an overview of existing research on this topic is lacking. In this systematic scoping review, we summarize the current state of the literature, identify shared drivers of multiple health and health behavior outcomes, and highlight areas ripe for future research.

METHODS

SCOPUS, Web of Science, and PubMed databases were searched in April 2023 for peer-reviewed, English-language studies in high-income OECD countries containing a conceptual systems model or simulation model of socioeconomic inequalities in health or health behavior in the adult general population. Two independent reviewers screened abstracts and full texts. Data on study aim, type of model, all model elements, and all relationships were extracted. Model elements were categorized based on the Commission on Social Determinants of Health framework, and relationships between grouped elements were visualized in a summary conceptual systems map.

RESULTS

A total of 42 publications were included; 18 only contained a simulation model, 20 only contained a conceptual model, and 4 contained both types of models. General health outcomes (e.g., health status, well-being) were modeled more often than specific outcomes like obesity. Dietary behavior and physical activity were by far the most commonly modeled health behaviors. Intermediary determinants of health (e.g., material circumstances, social cohesion) were included in nearly all models, whereas structural determinants (e.g., policies, societal values) were included in about a third of models. Using the summary conceptual systems map, we identified 15 shared drivers of socioeconomic inequalities in multiple health and health behavior outcomes.

CONCLUSIONS

The interconnectedness of socioeconomic position, multiple health and health behavior outcomes, and determinants of socioeconomic inequalities in health is clear from this review. Factors central to the complex system as it is currently understood in the literature (e.g., financial strain) may be both efficient and effective policy levers, and factors less well represented in the literature (e.g., sleep, structural determinants) may warrant more research. Our systematic, comprehensive synthesis of the literature may serve as a basis for, among other things, a complex systems framework for socioeconomic inequalities in health.

Process evaluation of school-based high-intensity interval training interventions for children and adolescents: a systematic review and meta-analysis of randomized controlled trials

BACKGROUND

Several systematic reviews have been published to investigate the effectiveness of high-intensity interval training (HIIT) in schools. However, there has been limited attention given to understanding the functioning of the intervention processes, which is of paramount importance for interpreting and translating the intervention effectiveness. The aim of this systematic review is to determine the extent to which process evaluation is measured in school-based HIIT interventions and to explore the effects of process evaluation and intervention characteristics on cardiorespiratory fitness (CRF), body composition, muscular strength, and blood pressure.

METHODS

A comprehensive search was conducted in SPORT Discus (EBSCOhost), Web of Science, Scopus, Medline (Ovid) and Cochrane Central Register of Controlled Trials. The extent to which process evaluation is measured was narratively reported, alongside with the guidance of process evaluation of complex interventions by UK Medical Research Council. Meta-analyses and meta-regressions were conducted to determine the effects of process evaluation and intervention characteristics to the intervention outcomes.

RESULTS

The literature search identified 77 studies reporting on 45 school-based HIIT interventions. In total, five interventions reported process evaluation in a section or in a separate study, and only one intervention adopted a process evaluation framework. On average, 6 out of 12 process evaluation measures were reported in all interventions. Subgroup analyses did not indicate any beneficial treatment effects for studies with process evaluation group, whereas all pooled data and studies without process evaluation group showed significant improvement for CRF and body composition.

CONCLUSION

Process evaluation is frequently omitted in the literature of school-based HIIT in children and adolescents. Although reporting of process evaluation measures may not directly associate with better intervention outcomes, it allows accurate interpretation of intervention outcomes, thereby enhancing the generalisability and dissemination of the interventions.

Real-life experiments in supermarkets to encourage healthy dietary-related behaviours: opportunities, challenges and lessons learned

BACKGROUND

Supermarkets are the primary source of food for many people yet their full potential as a setting to encourage healthy dietary-related behaviours remains underutilised. Sharing the experiences from research groups who have worked with supermarket chains to evaluate strategies that promote healthy eating could improve the efficiency of building such relationships and enhance the design quality of future research studies.

METHODS

A collective case study approach was used to synthesise experiences of engaging and sustaining research collaborations with national supermarket chains to test the effectiveness of health-focused in-store interventions. The collective narrative covers studies conducted in three high-income countries: Australia, the Netherlands and the United Kingdom.

RESULTS

We have distilled our experiences and lessons learned into six recommendations for conducting high quality public health research with commercial supermarket chains. These include: (i) using personal contacts, knowledge of supermarket activities and engaging executive management to establish a partnership and allowing time to build trust; (ii) using scientifically robust study designs with appropriate sample size calculations; (iii) formalising data exchange arrangements and allocating adequate resource for data extraction and re-categorisation; (iv) assessing effects at individual/households level where possible; (v) designing a mixed-methods process evaluation to measure intervention fidelity, dose and unintended consequences; and (vi) ensuring scientific independence through formal contract agreements.

CONCLUSIONS

Our collective experiences of working in non-financial partnerships with national supermarket chains could be useful for other research groups looking to develop and implement supermarket studies in an efficient manner. Further evidence from real-life supermarket interventions is necessary to identify sustainable strategies that can improve population diet and maintain necessary commercial outcomes.

Meta-evaluation of a whole systems programme, ActEarly: A study protocol

INTRODUCTION

Living in an area with high levels of child poverty predisposes children to poorer mental and physical health. ActEarly is a 5-year research programme that comprises a large number of interventions (>20) with citizen science and co-production embedded. It aims to improve the health and well-being of children and families living in two areas of the UK with high levels of deprivation; Bradford in West Yorkshire, and the London Borough of Tower Hamlets. This protocol outlines the meta-evaluation (an evaluation of evaluations) of the ActEarly programme from a systems perspective, where individual interventions are viewed as events in the wider policy system across the two geographical areas. It includes investigating the programme's impact on early life health and well-being outcomes, interdisciplinary prevention research collaboration and capacity building, and local and national decision making.

METHODS

The ActEarly meta-evaluation will follow and adapt the five iterative stages of the 'Evaluation of Programmes in Complex Adaptive Systems' (ENCOMPASS) framework for evaluation of public health programmes in complex adaptive systems. Theory-based and mixed-methods approaches will be used to investigate the fidelity of the ActEarly research programme, and whether, why and how ActEarly contributes to changes in the policy system, and whether alternative explanations can be ruled out. Ripple effects and systems mapping will be used to explore the relationships between interventions and their outcomes, and the degree to which the ActEarly programme encouraged interdisciplinary and prevention research collaboration as intended. A computer simulation model ("LifeSim") will also be used to evaluate the scale of the potential long-term benefits of cross-sectoral action to tackle the financial, educational and health disadvantages faced by children in Bradford and Tower Hamlets. Together, these approaches will be used to evaluate ActEarly's dynamic programme outputs at different system levels and measure the programme's system changes on early life health and well-being.

DISCUSSION

This meta-evaluation protocol presents our plans for using and adapting the ENCOMPASS framework to evaluate the system-wide impact of the early life health and well-being programme, ActEarly. Due to the collaborative and non-linear nature of the work, we reserve the option to change and query some of our evaluation choices based on the feedback we receive from stakeholders to ensure that our evaluation remains relevant and fit for purpose.

Knowledge exchange in crisis settings: A scoping review

BACKGROUND

Public health practice and efforts to improve the social determinants of health operate within a climate characterised by multiple and intersecting crises. This includes the Covid-19 pandemic as well as more protracted crises such as climate change and persistent social inequalities that impact health. We sought to understand and compare how knowledge exchange (KE) processes occur across different crises, and how knowledge on improving social determinants of health can be utilised at times of crisis to reduce health inequalities and strengthen public systems.

METHODS

We conducted a scoping review to understand how KE on improving social determinants of health can occur across different types of crises (e.g. environmental, pandemics, humanitarian). Relevant studies were identified through electronic searching of Medline, EMBASE, Global Health, Scopus and Web of Science databases.

RESULTS

We identified 86 studies for inclusion in the review. Most studies concerned pandemic or environmental crises. Fewer studies explored KE during technical (e.g. nuclear), terror-related or humanitarian crises. This may reflect a limitation of the searches. Few studies assessed KE as part of longer-term responses to social and economic impacts of crises, with studies more likely to focus on immediate response or early recovery stages. Exchange of research evidence or data with policy or practice contextual knowledge was common but there was variation in the extent that lay (public) knowledge was included as part of KE processes.

CONCLUSION

As ongoing crises continue with significant public health implications, KE processes should appropriately reflect the complexity inherent in crises and foreground health inequalities. Doing so could include the utilisation of systems or complexity-informed methods to support planning and evaluation of KE, a greater focus on KE to support action to address social determinants of health, and the inclusion of a plurality of knowledge-including lived experience-in planning and responding to crises.

Child and family experiences of a whole-systems approach to physical activity in a multiethnic UK city: a citizen science evaluation protocol

INTRODUCTION

Whole-systems approaches are being adopted to tackle physical inactivity. The mechanisms contributing to changes resulting from whole-systems approaches are not fully understood. The voices of children and families that these approaches are designed for need to be heard to understand what is working, for whom, where and in what context. This paper describes the protocol for the children and families' citizen science evaluation of the Join Us: Move, Play (JU:MP) programme, a whole-systems approach to increasing physical activity in children and young people aged 5-14 years in Bradford, UK.

METHODS AND ANALYSIS

The evaluation aims to understand the lived experiences of children and families' relationship with physical activity and participation in the JU:MP programme. The study takes a collaborative and contributory citizen science approach, including focus groups, parent-child dyad interviews and participatory research. Feedback and data will guide changes within this study and the JU:MP programme. We also aim to examine participant experience of citizen science and the suitability of a citizen science approach to evaluate a whole-systems approach. Data will be analysed using framework approach alongside iterative analysis with and by citizen scientists in the collaborative citizen science study.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the University of Bradford: study one (E891-focus groups as part of the control trial, E982-parent-child dyad interviews) and study two (E992). Results will be published in peer-reviewed journals and summaries will be provided to the participants, through schools or directly. The citizen scientists will provide input to create further dissemination opportunities.

UK government's new placement legislation is a 'good first step': a rapid qualitative analysis of consumer, business, enforcement and health stakeholder perspectives

BACKGROUND

The current food system in England promotes a population diet that is high in fat, sugar and salt (HFSS). To address this, the UK government has implemented legislation to restrict the promotion of HFSS products in prominent locations (e.g. store entrances, checkouts) in qualifying retailers since October 2022. This study investigated the perceived impact of the legislation for affected stakeholders.

METHODS

A pre-implementation rapid qualitative evaluation of stakeholder interviews. One hundred eight UK stakeholders participated in the study including 34 consumers, 24 manufacturers and retailers, 22 local authority enforcement officers and 28 academic and charitable health representatives. A participatory conference was used to enable policy recommendations to be confirmed by stakeholders.

RESULTS

Stakeholders perceived the legislation to be a 'good first step' towards improving population diet but recognised this needed to be considered amongst a range of long-term obesity policies. Areas of further support were identified and these are presented as six recommendations for government to support the successful implementation of the legislation: (1) provide a free central HFSS calculator, (2) refine legislation to enhance intent and clarity, (3) conduct a robust evaluation to assess intended and unintended outcomes, (4) provide greater support for smaller businesses, (5) provide ring-fenced resources to local authorities and (6) create and communicate a long-term roadmap for food and health.

CONCLUSIONS

This legislation has the potential to reduce impulse HFSS purchases and makes a solid start towards creating healthier retail outlets for consumers. Immediate government actions to create a freely accessible HFSS calculator, support smaller businesses and provide additional resources to local authorities would support successful implementation and enforcement. Independent evaluation of the implementation of the legislation will enable monitoring of potential unintended consequences identified in this study and support refinement of the legislation. A long-term roadmap is necessary to outline strategies to support equal access to healthier and sustainable food across the whole food system within the next 20-30 years.

What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework

BACKGROUND

Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create.

METHODS

We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework.

RESULTS

We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent.

CONCLUSION

The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.

A scoping review of systems approaches for increasing physical activity in populations

Introduction

The past decade has increasingly seen systems approaches as a featured theme in public health studies and policy documents. This trend is evident in the area of physical activity, which is a significant global health risk factor that is addressed in WHO’s Global Action Plan on Physical Activity. We undertook a comprehensive scoping review to characterize the application of systems approaches to physical activity, to develop a typology of the objectives, themes and methods of research papers that purported to apply systems thinking to this issue.

Methods

We searched electronic databases (PubMed, Web of Science, Scopus and PsycINFO) for studies published during the period 2010–2021 that explicitly applied systems approaches or methods to investigate and/or address population physical activity. A framework using systems-based methodological approaches was adapted to classify physical activity studies according to their predominant approach, covering basic descriptive, complex analytical and advanced forms of practice. We selected case studies from retained studies to depict the current “state of the art”.

Results

We included 155 articles in our narrative account. Literature reporting the application of systems approaches to physical activity is skewed towards basic methods and frameworks, with most attention devoted to conceptual framing and predictive modelling. There are few well-described examples of physical activity interventions which have been planned, implemented and evaluated using a systems perspective. There is some evidence of “retrofitted” complex system framing to describe programmes and interventions which were not designed as such.

Discussion

We propose a classification of systems-based approaches to physical activity promotion together with an explanation of the strategies encompassed. The classification is designed to stimulate debate amongst policy-makers, practitioners and researchers to inform the further implementation and evaluation of systems approaches to physical activity.

Conclusion

The use of systems approaches within the field of physical activity is at an early stage of development, with a preponderance of descriptive approaches and a dearth of more complex analyses. We need to see movement towards a more sophisticated research agenda spanning the development, implementation and evaluation of systems-level interventions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-022-00906-2.

Developing and implementing 20-mph speed limits in Edinburgh and Belfast: mixed-methods study

Background

Transport initiatives such as 20-mph (≈30-km/h) speed limits are anticipated to result in fewer road casualties and improve perceptions of safety, leading to increases in active travel. Lower speeds may also lead to more pleasant environments in which to live, work and play.

Objectives

The main objective was to evaluate and understand the processes and effects of developing and implementing 20-mph speed limits in Edinburgh and Belfast. The focus was on health-related outcomes (casualties and active travel) that may lead to public health improvements. An additional objective was to investigate the political and policy factors (conditions) that led to the decision to introduce the new speed limits.

Design

This was a mixed-methods study that comprised an outcome, process, policy and economic evaluation of two natural experiments.

Setting

The study was set in Edinburgh, Scotland, and Belfast, Northern Ireland, from 2000 to 2018.

Participants

The whole population of each city were participants, as well as stakeholders involved in implementation and decision-making processes.

Intervention

The intervention was the implementation of 20-mph legislation, signage, enforcement, and education and awareness-raising in Edinburgh (citywide) and Belfast (city centre).

Main outcome measures

The main outcomes measured were speed; number, type and severity of road collisions; perceptions; and liveability.

Data sources

The following data sources were used – routinely and locally collected quantitative data for speed, volume of traffic, casualties and collisions, and costs; documents and print media; surveys; interviews and focus groups; and Google Street View (Google Inc., Mountain View, CA, USA).

Results

Collisions and casualties – the overall percentage reduction in casualty rates was 39% (the overall percentage reduction in collision rates was 40%) in Edinburgh. The percentage reduction for each level of severity was 23% for fatal casualties, 33% for serious casualties and 37% for minor casualties. In Belfast there was a 2% reduction in casualties, reflecting differences in the size, reach and implementation of the two schemes. Perceptions – in Edinburgh there was an increase in two factors (support for 20 mph and rule-following after implementation) supported by the qualitative data. Liveability – for both cities, there was a small statistical increase in liveability. Speed – mean and median speeds reduced by 1.34 mph and 0.47 mph, respectively, at 12 months in Edinburgh, with no statistically significant changes in Belfast. History, political context, local policy goals, local priorities and leadership influenced decision-making and implementation in the two cities.

Limitations

There was no analysis of active travel outcomes because the available data were not suitable.

Conclusions

The pre-implementation period is important. It helps frame public and political attitudes. The scale of implementation and additional activities in the two cities had a bearing on the impacts. The citywide approach adopted by Edinburgh was effective in reducing speeds and positively affected a range of public health outcomes. The city-centre approach in Belfast (where speeds were already low) was less effective. However, the main outcome of these schemes was a reduction in road casualties at all levels of severity.

Future work

Future work should develop a statistical approach to public health interventions that incorporates variables from multiple outcomes. In this study, each outcome was analysed independently of each other. Furthermore, population measures of active travel that can be administered simply, inexpensively and at scale should be developed.

Study registration

This study is registered as ISRCTN10200526.

Funding

This project was funded by the National Institute for Health and Care Research (NIHR) Public Health Research programme and will be published in full in Public Health Research; Vol. 10, No. 9. See the NIHR Journals Library website for further project information.

Addressing alcohol-related harms in the local night-time economy: a qualitative process evaluation from a complex systems perspective

OBJECTIVES

English local authorities (LAs) are interested in reducing alcohol-related harms and may use discretionary powers such as the Late Night Levy (LNL) to do so. This study aims to describe how system stakeholders hypothesise the levy may generate changes and to explore how the system, its actors and the intervention adapt and co-evolve over time.

DESIGN

A process evaluation from a complex systems perspective, using qualitative methods.

SETTING

A London LA with high densities of residential and commercial properties, which implemented the LNL in 2014.

PARTICIPANTS

Data were generated through interviews with LNL implementers and alcohol consumers, observations in bars and during LNL patrols and documentary review.

INTERVENTION

The LNL allows LAs to charge late-night alcohol retailers an annual fee (£299-£4440) to manage and police the night-time economy (NTE).

RESULTS

When the LNL was being considered, stakeholders from different interest groups advanced diverse opinions about its likely impacts while rarely referencing supporting research evidence. Proponents of the levy argued it could reduce crime and anti-social behaviour by providing additional funds to police and manage the NTE. Critics of the levy hypothesised adverse consequences linked to claims that the intervention would force venues to vary their hours or close, cluster closing times, reduce NTE diversity and undermine public-private partnerships. In the first 2 years, levy-funded patrols developed relationships with the licensed trade and the public. The LNL did not undermine public-private partnerships and while some premises varied their hours, these changes did not undermine the intervention's viability, nor significantly cluster venue closing times, nor obviously damage the area's reputation for having a diverse NTE.

CONCLUSIONS

This study applies a framework for process evaluation from a complex systems perspective. The evaluation could be extended to measure alcohol-related outcomes and to consider the interplay between the national and local systems.

Challenges in Implementing the National Health Response to COVID-19 in Senegal

Since the beginning of the COVID-19 pandemic in Africa, many epidemiological or anthropological studies have been published. However, few studies have yet been conducted to understand the implementation of State interventions to fight the COVID-19 pandemic. In Senegal, the national response plan was planned before the country experienced its first official case of COVID-19 on 2 March 2020. This qualitative study, conducted in March and April 2021, based on 189 interviews, aims to understand how the national response has been implemented in several regions of Senegal. Implementation of the response to the pandemic was favoured by good preparation, capacity to adapt, responsiveness of health actors, and commitment for both the political and religious authorities. The implementation response was confronted by several constraining factors such as the coercive approach, the challenges of coordinating actors, and the lack of intersectoral response. The central level has sometimes used reflexivity processes to adapt its response, but it has remained highly politicized, centralized, directive, and with little involvement of civil society. In Senegal, the response to the pandemic has been implemented in a relatively political and directive, even coercive manner, without necessarily considering prior knowledge and the need to adapt it to local contexts and to involve civil society and community actors in the process.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43477-022-00053-4.

Navigating the river(s) of systems change: a multi-methods, qualitative evaluation exploring the implementation of a systems approach to physical activity in Gloucestershire, England

OBJECTIVES

Systems approaches aim to change the environments in which people live, through cross-sectoral working, by harnessing the complexity of the problem. This paper sought to identify: (1) the strategies which support the implementation of We Can Move (WCM), (2) the barriers to implementation, (3) key contextual factors that influence implementation and (4) impacts associated with WCM.

DESIGN

A multi-methods evaluation of WCM was completed between April 2019 and April 2021. Ripple Effects Mapping (REM) and semi-structured interviewers were used. Framework and content analysis were systematically applied to the dataset.

SETTING

WCM-a physical activity orientated systems approach being implemented in Gloucestershire, England.

PARTICIPANTS

31 stakeholder interviews and 25 stakeholders involved in 15 REM workshops.

RESULTS

A white-water rafting analogy was developed to present the main findings. The successful implementation of WCM required a facilitative, well-connected and knowledgeable guide (ie, the lead organisation), a crew (ie, wider stakeholders) who's vision and agenda aligned with WCM's purpose, and a flexible delivery approach that could respond to ever-changing nature of the river (ie, local and national circumstances). The context surrounding WCM further strengthened and hampered its implementation. Barriers included evaluative difficulties, a difference in stakeholder and organisational perspectives, misaligned expectations and understandings of WCM, and COVID-19 implications (COVID-19 also presented as a facilitative factor). WCM was said to strengthen cohesion and collaboration between partners, benefit other agendas and policies (eg, mental health, town planning, inequality), and improve physical activity opportunities and environments.

CONCLUSIONS

This paper is one of the first to evaluate a systems approach to increasing physical activity. We highlight key strategies and contextual factors that influenced the implementation of WCM and demonstrate some of the wider benefits from such approaches. Further research and methodologies are required to build the evidence base surrounding systems approaches in Public Health.

[Evaluation of the effects of a complex intervention]

Appraising the effects of a complex intervention is one step in a more broadly based research process, from the construction or modelling of the intervention to its actual deployment. It consists in measuring the effectiveness or impact of the intervention, i.e. analyzing its capacity to produce change. The aim of this article is to obtain unbiased measurement of the average effects of an intervention, based on a panel of predetermined parameters and on the assumption of a causal link between the intervention and the measured result. This article is consequently devoted to evaluation of the effects of a complex intervention and focuses on the methodological challenges of its three key stages : 1) modelling of the intervention and feasibility analysis, essential prerequisites ; 2) the choice of study design and of the effects to be measured, that is to say the methodological premises ; and 3) process analysis, carried out in parallel with the evaluation of effects, leading to an indispensable appraisal of the intervention implementation and of the context into which it is integrated. The article is illustrated by five intervention impact assessment projects. A specific objective when evaluating the effects of a complex intervention consists in (a) moving away from a simple search for causality involving the intervention and its effects and (b) toward understanding of the effectiveness mechanisms, once again taking into account the context and the actual conditions of implementation. The challenge is to embrace rather than limit the complexity of the intervention, this being an essential prerequisite for its successful deployment and eventual generalization.

Analyzing implementation of public health interventions : a need for rigor, and the challenges of stakeholder involvement

OBJECTIVES

This article shows how conceptual models can help to develop and evaluate public health interventions. It also reports on the challenges of getting stakeholders involved.

METHOD

The analysis is based on the reflexive approach applied by the authors during their participation in two public health intervention research (PHIR) projects, in France and in Burkina Faso.

RESULTS

In Paris, PHIR aimed to enable sub-Saharan immigrants to appropriate the existing means of prevention and sexual health care and to strengthen their empowerment in view of preserving their health. Evaluation was carried out using mixed methods. The intervention process theory is based on Ninacs' conceptual model of individual empowerment. The Consolidated Framework For Implementation Research (CFIR) was mobilized a posteriori to analyze the process. PHIR stemmed from collaboration between a research team and two associations. The different stakeholders were involved in the evaluation process, as were, at certain times, persons in highly precarious situations. In Ouagadougou, a community-based dengue vector control intervention was deployed to address an essential but neglected need. As regards evaluation, we opted for a holistic, mixed method approach (effectiveness and process). The contents of the intervention were determined based on tacit knowledge, a community preference survey and solid evidence. The theoretical framework of the intervention consisted in an eco-biological model of vector control. The implementation analysis combined an internal assessment of implementation fidelity with an external CFIR process analysis. All stakeholders were involved in the evaluation process.

DISCUSSION

Analysis confirmed not only the value of process evaluations in PHIR, but also the primordial importance of a rigorous approach. Stakeholder involvement is a major challenge to be addressed early in the planning of RISPs; with this in mind, effective and ethically sound assessment mechanisms need to be drawn up. Interdisciplinary evaluative approaches should be preferred, and the use of justified, relevant, and flexible frameworks is highly recommended.

CONCLUSION

Lessons learned for those wishing to engage in the process evaluation of a public health intervention are hereby presented.

Process evaluations of mental health and psychosocial support interventions for populations affected by humanitarian crises

BACKGROUND

Randomised controlled trials (RCTs) have been increasingly used to test the effectiveness of mental health and psychosocial support(MHPSS) interventions for populations affected by humanitarian crises. Process evaluations are often integrated within RCTs of psychological interventions to investigate the implementation of the intervention, the impact of context, and possible mechanisms of action. We aimed to explore limitations and strengths of how process evaluations are currently conceptualised and implemented within MHPSS RCTs specifically.

METHODS

In April-June 2021 we conducted semi-structured interviews with 24 researchers involved in RCTs of MHPSS interventions in 23 different countries. Participants were selected based on systematic reviews of MHPSS interventions, funders' databases, and personal networks. Data were analysed using codebook thematic analysis.

RESULTS

The conduct of process evaluations was characterized by high heterogeneity in perceived function, implementation outcomes assessed, and methods used. While process evaluations were overwhelmingly considered as an important component of an RCT, there were different opinions on their perceived quality. This could be explained by the varying prioritization of effectiveness data over implementation data, confusion around the nature of process evaluations, and challenges in the collection and analysis of process data in humanitarian settings. Various practical recommendations were made by participants to improve future process evaluations in relation to: (i) study design (e.g., embedding process evaluations in study protocol and overall study objectives); (ii) methods (e.g., use of mixed methods); and (iii) increased financial and human resources dedicated to process evaluations.

CONCLUSION

The current state of process evaluations in MHPSS RCTs is heterogeneous. The quality of process evaluations should be improved to strengthen implementation science of the growing number of evidence-informed MHPSS interventions.

Applying systems thinking to identify enablers and challenges to scale-up interventions for hypertension and diabetes in low-income and middle-income countries: protocol for a longitudinal mixed-methods study

INTRODUCTION

There is an urgent need to reduce the burden of non-communicable diseases (NCDs), particularly in low-and middle-income countries, where the greatest burden lies. Yet, there is little research concerning the specific issues involved in scaling up NCD interventions targeting low-resource settings. We propose to examine this gap in up to 27 collaborative projects, which were funded by the Global Alliance for Chronic Diseases (GACD) 2019 Scale Up Call, reflecting a total funding investment of approximately US$50 million. These projects represent diverse countries, contexts and adopt varied approaches and study designs to scale-up complex, evidence-based interventions to improve hypertension and diabetes outcomes. A systematic inquiry of these projects will provide necessary scientific insights into the enablers and challenges in the scale up of complex NCD interventions.

METHODS AND ANALYSIS

We will apply systems thinking (a holistic approach to analyse the inter-relationship between constituent parts of scaleup interventions and the context in which the interventions are implemented) and adopt a longitudinal mixed-methods study design to explore the planning and early implementation phases of scale up projects. Data will be gathered at three time periods, namely, at planning (TP), initiation of implementation (T0) and 1-year postinitiation (T1). We will extract project-related data from secondary documents at TP and conduct multistakeholder qualitative interviews to gather data at T0 and T1. We will undertake descriptive statistical analysis of TP data and analyse T0 and T1 data using inductive thematic coding. The data extraction tool and interview guides were developed based on a literature review of scale-up frameworks.

ETHICS AND DISSEMINATION

The current protocol was approved by the Monash University Human Research Ethics Committee (HREC number 23482). Informed consent will be obtained from all participants. The study findings will be disseminated through peer-reviewed publications and more broadly through the GACD network.

A Combined Theory of Change-Group Model Building Approach to Evaluating “Farm to Fork” Models for School Feeding in the Caribbean

There is a scarcity of research on building nutrition-sensitive value chains (NSVCs) to improve diets and nutrition outcomes of populations in the Caribbean. This study contributes to filling this research gap by outlining a participatory approach to evaluating a NSVC model for “farm to fork” (F2F) school feeding in the Eastern Caribbean Island of St. Kitts. Using a combined group model building (GMB) and theory of change (ToC) approach, policy actors and other stakeholders (n = 37) across the school feeding value chain were guided through a facilitated process to evaluate the ToC underlying a series of F2F interventions designed to enhance childhood nutrition. Stakeholders at the workshop engaged collaboratively to create a causal map of interconnected “system factors” that help explain behaviors contributing to unhealthy eating among children that extended well-beyond the original F2F project ToC that had been used to inform interventions. Through this facilitated GMB process, stakeholders proposed additional food system interventions, and identified multiple “impact pathways” and “mediating influences” underlying local availability and consumption of nutritious foods in local school environments. Workshop participants were also able to identify leverage points where community-level efforts, alongside research interventions, may ensure that initiatives for building local NSVCs are ultimately institutionalized. Results of this study suggest that developing NSVCs for school feeding and food systems in the Caribbean requires both locally driven innovation and the leveraging of system-wide resources, with lessons for project intervention strategies.

Evaluation proposal of a national community-based obesity prevention programme: a novel approach considering the complexity perspective

Community-based obesity prevention programmes are considered an important strategy to curb the obesity epidemic. The JOGG (Youth At a Healthy Weight) approach is a large-scale community-based programme for childhood obesity prevention in the Netherlands that has been implemented over the past ten years. Practice-based development of the programme, both at the national and local level, increasingly poses challenges for its evaluation. One considerable challenge is the increasing acknowledgement of the complexity in the JOGG-approach, characterized by (a) objectives that vary locally, (b) adaptions to the programme over time in response to a community's shifting needs, challenges and opportunities, and (c) emergent outcomes and non-linear causality.We propose an evaluation framework that highlights elements of the complex local practice, including the local programme theory, implementation, adaption, the influence of context and feedback loops and intended as well as emergent and unintended outcomes. By studying each of these elements in practice, we hope to learn about principles that guide effective obesity prevention across contexts. The results of the proposed evaluation will inform both practice and research.Considering complexity in evaluation is a relatively new challenge in public health and therefore an emergent research area. The proposed framework for complex evaluations allows to retrospectively evaluate a programme that was implemented and developed in practice, and enables us to learn from practice-based experiences. Following the ISBNPA Dare2Share initiative, we kindly invite other researchers in the field to share their ideas and experiences regarding integration of complexity in evaluation.

Identifying priorities for research on financial risk protection to achieve universal health coverage: a scoping overview of reviews

OBJECTIVES

Financial risk protection (FRP) is an indicator of the Sustainable Development Goal 3 universal health coverage (UHC) target. We sought to characterise what is known about FRP in the UHC context and to identify evidence gaps to prioritise in future research.

DESIGN

Scoping overview of reviews using the Arksey & O'Malley and Levac & Colquhoun framework and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews reporting guidelines.

DATA SOURCES

MEDLINE, PsycINFO, CINAHL-Plus and PAIS Index were systematically searched for studies published between 1 January 1995 and 20 July 2021.

ELIGIBILITY CRITERIA

Records were screened by two independent reviewers in duplicate using the following criteria: (1) literature review; (2) focus on UHC achievement through FRP; (3) English or French language; (4) published after 1995 and (5) peer-reviewed.

DATA EXTRACTION AND SYNTHESIS

Two reviewers extracted data using a standard form and descriptive content analysis was performed to synthesise findings.

RESULTS

50 studies were included. Most studies were systematic reviews focusing on low-income and middle-income countries. Study periods spanned 1990 and 2020. While FRP was recognised as a dimension of UHC, it was rarely defined as a concept. Out-of-pocket, catastrophic and impoverishing health expenditures were most commonly used to measure FRP. Pooling arrangements, expansion of insurance coverage and financial incentives were the main interventions for achieving FRP. Evidence gaps pertained to the effectiveness, cost-effectiveness and equity implications of efforts aimed at increasing FRP. Methodological gaps related to trade-offs between single-country and multicountry analyses; lack of process evaluations; inadequate mixed-methods evidence, disaggregated by relevant characteristics; lack of comparable and standardised measurement and short follow-up periods.

CONCLUSIONS

This scoping overview of reviews characterised what is known about FRP as a UHC dimension and found evidence gaps related to the effectiveness, cost-effectiveness and equity implications of FRP interventions. Theory-informed mixed-methods research using high-quality, longitudinal and disaggregated data is needed to address these objectives.

Critical factors that affect the functioning of a research and evaluation capacity building partnership: A causal loop diagram

Introduction

Public health policy and practice is strengthened by the application of quality evidence to decision making. However, there is limited understanding of how initiatives that support the generation and use of evidence in public health are operationalised. This study examines factors that support the internal functioning of a partnership, the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN). SiREN aims to build research and evaluation capacity and increase evidence-informed decision making in a public health context.

Methods

This study was informed by systems concepts. It developed a causal loop diagram, a type of qualitative system model that illustrated the factors that influence the internal operation of SiREN. The causal loop diagram was developed through an iterative and participatory process with SiREN staff and management (n = 9) via in-depth semi-structured interviews (n = 4), workshops (n = 2), and meetings (n = 6).

Results

Findings identified critical factors that affected the functioning of SiREN. Central to SiREN’s ability to meet its aims was its capacity to adapt within a dynamic system. Adaptation was facilitated by the flow of knowledge between SiREN and system stakeholders and the expertise of the team. SiREN demonstrated credibility and capability, supporting development of new, and strengthening existing, partnerships. This improved SiREN’s ability to be awarded new funding and enhanced its sustainability and growth. SiREN actively balanced divergent stakeholder interests to increase sustainability.

Conclusion

The collaborative development of the diagram facilitated a shared understanding of SiREN. Adaptability was central to SiREN achieving its aims. Monitoring the ability of public health programs to adapt to the needs of the systems in which they work is important to evaluate effectiveness. The detailed analysis of the structure of SiREN and how this affects its operation provide practical insights for those interested in establishing a similar project.

Diverse Approaches to Creating and Using Causal Loop Diagrams in Public Health Research: Recommendations From a Scoping Review

Objectives: Complex systems thinking methods are increasingly called for and used as analytical lenses in public health research. The use of qualitative system mapping and in particular, causal loop diagrams (CLDs) is described as one promising method or tool. To our knowledge there are no published literature reviews that synthesize public health research regarding how CLDs are created and used. Methods: We conducted a scoping review to address this gap in the public health literature. Inclusion criteria included: 1) focused on public health research, 2) peer reviewed journal article, 3) described and/or created a CLD, and 4) published in English from January 2018 to March 2021. Twenty-three articles were selected from the search strategy. Results: CLDs were described as a new tool and were based upon primary and secondary data, researcher driven and group processes, and numerous data analysis methods and frameworks. Intended uses of CLDs ranged from illustrating complexity to informing policy and practice. Conclusion: From our learnings we propose nine recommendations for building knowledge and skill in creating and using CLDs for future public health research.

There is no trustable data: the state and data accuracy of violence against women in Indonesia

The number of violence against women in Indonesia continues to increase every year, this can be seen from data released by the National Commission on Violence Against Women and the State Ministry of Women's Empowerment and Child Protection. The problem is, although the two state institutions regularly release data on violence against women, the data released does not nationally represent the data on violence. This study aims to find reasons on why data on violence in Indonesia cannot be used as material for policy making. By using ethnographic methods in the National Commission on Violence Against Women, the Ministry, and six partner institutions, this study found a number of facts why the data on violence in Indonesia is so unreliable. At the ministry level, the main constraints are with the forms that are difficult to fill in, the model of tiered bureaucracy that is useful for reporting, but with the higher the level, the less violence data there are, the limited number of registrar. Whereas National Commission on Violence Against Women data are constrained by a centralized and voluntary data collection model, that more than 65% of partners do not return the data collection forms. This results in both the Ministry and the Komnas Perempuan data being just the tip of the iceberg of the problem of violence against women in Indonesia.

A whole system approach to increasing children's physical activity in a multi-ethnic UK city: a process evaluation protocol

BACKGROUND

Engaging in regular physical activity requires continued complex decision-making in varied and dynamic individual, social and structural contexts. Widespread shortfalls of physical activity interventions suggests the complex underlying mechanisms of change are not yet fully understood. More insightful process evaluations are needed to design and implement more effective approaches. This paper describes the protocol for a process evaluation of the JU:MP programme, a whole systems approach to increasing physical activity in children and young people aged 5-14 years in North Bradford, UK.

METHODS

This process evaluation, underpinned by realist philosophy, aims to understand the development and implementation of the JU:MP programme and the mechanisms by which JU:MP influences physical activity in children and young people. It also aims to explore behaviour change across wider policy, strategy and neighbourhood systems. A mixed method data collection approach will include semi-structured interview, observation, documentary analysis, surveys, and participatory evaluation methods including reflections and ripple effect mapping.

DISCUSSION

This protocol offers an innovative approach on the use of process evaluation feeding into an iterative programme intended to generate evidence-based practice and deliver practice-based evidence. This paper advances knowledge regarding the development of process evaluations for evaluating systems interventions, and emphasises the importance of process evaluation.

Methodological Considerations in the Assessment of Effectiveness of Homeopathic Care: A Critical Review of the EPI3 Study

BACKGROUND

  EPI3 is an observational study of a representative sample of general practitioners (GPs) and patients in France, demonstrating that patient characteristics differ according to the prescribing preferences of their GPs for homeopathy. For selected conditions (musculoskeletal disorders, sleep disorders, anxiety/depression, upper respiratory tract infections), progression of symptoms and adverse events over follow-up in the homeopathy preference group did not significantly differ from other practice preferences, but there was a two-fold to four-fold lower usage of conventional medicines. The EPI3 study's validity was challenged due to absence of head-to-head comparison of medicines to conclude on a causal association between homeopathy and outcomes.

METHODS

 A critical review of the nine EPI3 publications was conducted, focusing on generalizability, selection bias, outcome measurements and confounding.

RESULTS

 The conceptual framework of EPI3 rests on a systemic construct, i.e., the homeopathic treatment concept assessed using the type of GP prescribing preference, taking into account the clinical, human and social aspects. The enrollment process enhanced the generalizability of findings. Validated instruments for outcome measurements were used for three conditions, and control of confounding was rigorous.

CONCLUSION

 EPI3 was conducted according to best practices. Homeopathy prescribing preference met specific patient needs with less use of conventional medicines and without an apparent loss in therapeutic opportunity.

Systems thinking and complexity science methods and the policy process in non-communicable disease prevention: a systematic scoping review protocol

INTRODUCTION

Given the complex causal origins of many non-communicable diseases (NCDs), and the complex landscapes in which policies designed to tackle them are made and unfold, the need for systems thinking and complexity science (STCS) in developing effective policy solutions has been emphasised. While numerous methods informed by STCS have been applied to the policy process in NCD prevention, these applications have not been systematically catalogued. The aim of this scoping review is to identify existing applications of methods informed by STCS to the policy process for NCD prevention, documenting which domains of the policy process they have been applied to.

METHODS AND ANALYSIS

A systematic scoping review methodology will be used.

IDENTIFICATION

We will search Medline, SCOPUS, Embase and Web of Science using search terms combining STCS, NCD prevention and the policy process. All records published in English will be eligible for inclusion, regardless of study design.

SELECTION

We will screen titles and abstracts and extract data according to published guidelines for scoping reviews. In order to determine the quality of the included studies, we will use the approach developed by Dixon-Woods et al, excluding studies identified as fatally flawed, and determining the credibility and contribution of included studies.

SYNTHESIS

We will identify relevant studies, summarising key data from each study and mapping applications of methods informed by STCS to different parts of the policy process. Review findings will provide a useful reference for policy-makers, outlining which domains of the policy process different methods have been applied to.

ETHICS AND DISSEMINATION

Formal ethical approval is not required, as the study does not involve primary data collection. The findings of this study will be disseminated through a peer-reviewed publication, presentations and summaries for key stakeholders.

Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

BACKGROUND

The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods.

METHODS

There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments.

RESULTS

Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six 'core elements' that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout.

LIMITATIONS

The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development.

CONCLUSIONS

This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as 'events in systems' that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers.

FUTURE WORK

We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources.

FUNDING

This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).

One Health Evaluation of Antimicrobial Use and Resistance Surveillance: A Novel Tool for Evaluating Integrated, One Health Antimicrobial Resistance and Antimicrobial Use Surveillance Programs

We describe the development, application and utility of our novel, One Health Evaluation of Antimicrobial Use and Resistance Surveillance (OHE-AMURS) tool that we created to evaluate progress toward integrated, One Health surveillance of antimicrobial resistance (AMR) and antimicrobial use (AMU) as a complex system in Canada. We conducted a qualitative inquiry into the current state of policy and programs for integrated AMR/AMU surveillance using explicit and tacit knowledge. To assess the "messy" state of public health surveillance program development, we synthesized recommendations from previous reports by the National Collaborating Centre for Infectious Diseases and the Canadian Council of Chief Veterinary Officers; conducted an environmental scan to find all federal, provincial, and territorial AMR/AMU surveillance programs in Canada; and conducted semi-structured interviews with Canadian subject matter experts. To integrate evidence from these different sources we adapted two published tools to create a new evaluation matrix, deriving 36 components of the ideal integrated AMR/AMU surveillance system. Our two-way matrix tool allowed us to examine seven common, foundational elements of sustainable programs for each component, and assign a stage of development/sustainability ranking for each component according to the matrix definitions. Our adaptable novel tool allowed for granular and repeatable assessment of the many components of a complex surveillance system. The assessment proved robust and exacting to ensure transparency in our methods and results. The matrix allows flexible assignment of program components based on program principles, and stages can be adapted to evaluate any aspect of an AMR/AMU surveillance or other multi-faceted, multi-jurisdictional system. Future refinement should include an assessment of the scope of surveillance components.