Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

Building a systems-thinking community workforce to scale action on determinants of health in New Zealand

This article describes findings from the evaluation of Healthy Families NZ (HFNZ), an equity-driven, place-based community health initiative. Implemented in nine diverse communities across New Zealand, HFNZ aims to strengthen the systems that can improve health and well-being. Findings highlight local needs and priorities including the social mechanisms important for reorienting health and policy systems towards place-based communities. Lessons encompass the importance of local lived experience in putting evidence into practice; the strength of acting with systems in mind; the need for relational, learning, intentional, and well-resourced community organisation; examples of how to foster place-based 'community-up' leadership; and how to enable responsiveness between communities and local and national policy systems. A reconceptualisation of scaling in the context of complexity and systems change is offered, which recognises that relationships and agency are key to making progress on the determinants of health.

Understanding 'value' in the context of community-based interventions for people affected by dementia: A concept analysis

AIM

This study aimed to conduct a concept analysis of value in the context of community-based interventions for people affected by dementia.

BACKGROUND

Concepts of value play a critical role in shaping the delivery and distribution of community-based health interventions through related concepts. However, the use and meaning of 'value' is rarely clarified limiting the term's utility in practice and research. Increasing need for community healthcare and scarce public resources means developing understanding of value in community-based interventions for people affected by dementia is timely, and may support more informed approaches to exploring, explaining and delivering value.

DESIGN

Evolutionary Concept Analysis was used to systematically determine the characteristics of value.

DATA SOURCES

Peer-reviewed and grey literature databases were searched between April and July 2021, with 32 pieces of literature from different disciplines included in the final sample. No limits were set for the years of literature retrieved.

METHODS

Literature was thematically analysed for information on the antecedents, attributes and consequences of value.

RESULTS AND DISCUSSION

The analysis uncovered a need and/or desire to understand the experience of people affected by or that affect interventions; and to demonstrate, prove/disprove the (best) quality and nature of results of interventions as antecedents of value. Attributes of value were stakeholder/person centred, measurable, time and context dependent and multidimensional. Consequences of the concept included shared decision-making, valuation of interventions and internal/external investment and development of interventions.

CONCLUSION

Through concept analysis value can now be better understood and applied. The development of a conceptual model to illustrate the constituent elements and relationships of the concept adds transparency to where, why and how concepts of value are enabled that supports future concept development.

PATIENT AND PUBLIC CONTRIBUTION

No patient or public contribution.

Evaluating an online module for sexual violence prevention in a tertiary educational setting: An exploratory study

ISSUE ADDRESSED

Sexual violence is an important public health issue affecting significant numbers of university students across Australia and internationally. In response, online modules have been widely implemented and there is an urgent need to better understand their effectiveness. The aim of this study was to evaluate an online sexual violence prevention and response education module designed for and implemented in one Australian university.

METHODS

We used a mixed-methods approach that included pre/post module completion surveys of key measures relating to sexual consent, being a bystander, and response to disclosures as well as knowledge of resources and support services. We conducted post module completion semi-structured interviews.

RESULTS

Results indicated potential effectiveness of the module on beliefs about sexual consent, confidence intervening when witnessing potentially harmful situations, willingness to report incidents, confidence supporting a peer who discloses an incident, and knowledge of support services. Qualitative results indicated support for the online module as an accessible, private and self-paced tool for sexual violence education. Interactive, relevant and engaging content that can be applied in real-life contexts was noted as key for effectiveness.

CONCLUSIONS

This exploratory study suggests there may be potential for online modules to be effective, as one aspect of universities' sexual violence prevention and response strategies-particularly modules aimed at addressing primary, secondary and tertiary prevention. Further rigorous research is required to strengthen best practice in the development and implementation of online modules as part of whole-of-campus strategies. SO WHAT?: Universities across Australia and internationally are grappling with sexual violence response and prevention in light of high prevalence rates among students. Online modules may be one effective tool when implemented as part of a wider strategy.

Process evaluation of the Invictus Pathways Program

INTRODUCTION

UniSA's Invictus Pathways Program (IPP), a service program, was originally developed to assist veterans to train for and participate in the Invictus Games. More recently, the scope of the IPP has widened to support and improve wellbeing and facilitate post traumatic growth and recovery among participants who are living with physical and mental health injuries and conditions. This paper describes the components of the IPP and reports its process evaluation.

METHODS

Underpinned by a pragmatic approach, data related to participant and student involvement in the IPP, the number of participant training sessions, session attendance, program activities and events, and program fidelity were compiled from process documentation that had been collected between 2017 and 2020, inclusive. Following ethics approval, semi-structured interviews were conducted with participants of the IPP, members of their family support network, and university staff to understand the operations of the IPP and satisfaction with the program.

FINDINGS

There was high fidelity for the student-led exercise training aspects of IPP; however, data collection relevant to participants' psychological outcomes, and non-training IPP events and activities did not always occur as intended. Between 2017 and 2020, 53 veterans had participated in or were still participating in the IPP, and 63 allied health students had completed placements as student trainers. Fifty-three individual training sessions were delivered in 2017, increasing to 1,024 in 2020. Seventy-one interviews were completed with key IPP stakeholders. The qualitative analysis identified four higher order themes: Implementation and fidelity of the IPP, Satisfaction with the IPP, Areas of the IPP requiring improvement and suggestions for change, and Sustainability of the IPP. Satisfaction was generally high for the IPP, although there were factors that negatively impacted the experience for some participants and their family support network. Suggestions for improvement to program components and delivery aspects were made during the interviews, and the precariousness of IPP funding and sustainability was raised as an ongoing concern.

CONCLUSION

This process evaluation has demonstrated that the physical activity training components of the IPP were delivered with high levels of fidelity, and that satisfaction with the IPP is mostly high, although there are areas that could be improved. There is a need for a more structured approach to the ongoing evaluation of the IPP. This includes ensuring that program staff have a shared understanding of the purpose of evaluation activities and that these activities occur as intended. Beyond this is the need to secure funding to support the sustainability of the IPP, so that it can continue to contribute to the wellbeing of veterans living with physical and mental health conditions, and their families.

Evaluation of a pre-professional pathway program: perspectives of former students in the rural pre-medicine program at Selkirk College

Background

Having a rural background is one of the most predictive factors in eventually having a rural practice, but people from rural areas face several barriers to post-secondary education. Pre-professional rural pathway initiatives are a potential solution. The Rural Pre-Medicine Program (RPM) at Selkirk College, British Columbia was developed to provide students with the credits necessary to apply to medicine and other health professional programs, an introduction to rural healthcare issues, and a unique and comprehensive support program to enable success.

Methods

We administered a cross-sectional survey to former students who left the program from its inception in September 2014 to May 2020 to explore the extent to which program aims are being met.

Results

The response rate was 49.4% (40/81). Respondents agreed the program increased their skills, their understanding of rural healthcare issues, and enhanced their competitiveness for applying to health professional programs. Most agreed the program increased their future rural work intentions. Respondents suggested that academic programming be more flexible to allow for more varied post-program pathways.

Conclusion

This survey provides preliminary evidence the RPM Program is on track to increase the number of people with a rural affinity who prepare to become health professionals.

Experiences in the application of logic models in the context of workplace health promotion - A focus group discussion

Gathering evidence on complex workplace health promotion interventions faces methodological challenges. Therefore, the application of logic models as a theory of change is recommended to support outcome and process evaluations. The present study explores challenges and opportunities of applying logic models in application-oriented intervention research on workplace health promotion. A focus group (n = 6), consisting of scientists and workplace health promotion practitioners, was conducted using a semi-structured interview guide. The recorded qualitative data were transcribed and analysed using the structuring content analysis method. According to the focus group, logic models provide several opportunities for planning and evaluating complex workplace health promotion interventions. Logic models support the communication between science and practice, and have benefits for the provider of workplace health promotion interventions. The main challenges in working with logic models were dealing with the complex and constantly developing intervention and with the derivation and implementation of reasonable evaluation methods. The focus group exposed repeated application and a shared understanding between stakeholders as facilitators for working with logic models. In conclusion, at the science-practice interface, logic models could enhance the integrative understanding and the further development of evidence-based workplace health promotion.

A mixed methods evaluation of Quit for new life, a smoking cessation initiative for women having an Aboriginal baby

BACKGROUND

Quit for new life (QFNL) is a smoking cessation initiative developed to support mothers of Aboriginal babies to quit smoking during pregnancy. The state-wide initiative provides support for pregnant women and their households including free nicotine replacement therapy (NRT) and follow up cessation advice. Services are also supported to implement systems-level changes and integrate QFNL into routine care. This study aimed to evaluate: (1) models of implementation of QFNL; (2) the uptake of QFNL; (3) the impact of QFNL on smoking behaviours; and (4) stakeholder perceptions of the initiative.

METHODS

A mixed methods study was conducted comprising semi-structured interviews and analysis of routinely collected data. Interviews were conducted with 6 clients and 35 stakeholders involved in program implementation. Data were analysed using inductive content analysis. Aboriginal Maternal and Infant Health Service Data Collection (AMDC) records for the period July 2012-June 2015 were investigated to examine how many eligible women attended a service implementing QFNL and how many women took up a QFNL support. Smoking cessation rates were compared in women attending a service offering QFNL with women attending the same service prior to the implementation of QFNL to determine program impact.

RESULTS

QFNL was implemented in 70 services located in 13 LHDs across New South Wales. Over 430 staff attended QFNL training, including 101 staff in Aboriginal-identified roles. In the period July 2012-June 2015 27% (n = 1549) of eligible women attended a service implementing QFNL and 21% (n = 320) of these were recorded as taking up a QFNL support. While stakeholders shared stories of success, no statistically significant impact of QFNL on smoking cessation rates was identified (N = 3502; Odds ratio (OR) = 1.28; 95% Confidence Interval (CI) = 0.96-1.70; p-value = 0.0905). QFNL was acceptable to both clients and stakeholders, increased awareness about smoking cessation, and gave staff resources to support clients.

CONCLUSION

QFNL was perceived as acceptable by stakeholders and clients and provided care providers with knowledge and tangible support to offer women who presented at antenatal care as smokers, however, no statistically significant impact on rates of smoking cessation were found using the measures available.

Impact of the Domestic Violence Housing First Model on Survivors' Safety and Housing Stability: Six Month Findings

Intimate partner violence (IPV) is a leading cause of homelessness, yet little evidence exists about effective strategies to assist IPV survivors as they work to avoid homelessness while freeing themselves from abuse. An ongoing demonstration evaluation is examining if and how one promising model assists IPV survivors in obtaining safe and stable housing over time. The Domestic Violence Housing First (DVHF) model involves providing IPV survivors with mobile advocacy and/or flexible funding, depending on individual needs, in order to attain these goals. We hypothesized that those receiving DVHF would experience greater housing stability and less re-abuse compared to survivors receiving services as usual. The current study evaluated the short-term efficacy of the DVHF model with a sample of 345 homeless or unstably housed survivors who sought services and who completed in-person interviews shortly after contacting the DV agency, as well as six months later. Those who received the DVHF model showed greater improvement in their housing stability compared to those receiving services as usual, as well as decreased economic abuse. Both groups experienced a sharp decline in all forms of abuse. The Domestic Violence Housing First model shows promise in helping unstably housed DV survivors achieve safe and stable housing. Study findings have implications for DV agencies as well as those funding such services. Understanding which interventions work best for which survivors is critical to ensuring that service providers are effectively working toward long-term housing stability and well-being for IPV survivors and their children.

Pragmatic Evaluation of a Low-Threshold Sports Program for Older Adults in Group Homes

Physical activity (PA) reduces with older age, ill health, and disability. For these groups, guidance recognizes the benefit of small increases in PA and low-intensity PA. This study evaluated a low-threshold intervention that addressed known barriers to older people's participation in PA in residential care and sheltered housing. Ten, competitive sport sessions were delivered by coaches at 49 sites with the aim that they be sustained in-house. Using quasi-experimental methods, participants reported reduced sitting time, increased moderate/vigorous PA, increased participation in sports and improved scores for both health quality of life and fear of falling at 6 months. The program engaged 29% of residents and was sustained at 50% of sites at 8 months. The findings suggest that low-threshold sports programs that overcome known barriers to older people's participation in PA have the potential to provide a gateway to increased PA in group homes and to be sustained in-house.

Co-creating physical activity interventions: Findings from a multiple case study using mixed methods

Introduction

In health and physical activity promotion, there is growing interest in co-creation approaches that involve researchers and non-academic stakeholders in developing new interventions. Previous research has shown the promising results of cooperative planning as a co-creation approach in building new capacities and implementing physical activity-promoting interventions in nursing care and automotive mechatronics. However, it remains unclear whether (1) cooperative planning for physical activity promotion can be successfully transferred to other settings in the nursing care and automotive mechatronic sectors and (2) what key factors influence its success or failure.

Methods

We conducted a multiple case study in three settings in the nursing care and automotive mechatronics sectors. Following a mixed methods approach, we collected, analyzed, and triangulated data from documents (n = 17), questionnaires (n = 66), and interviews (n = 6). Quantitative data were analyzed descriptively and through using nonparametric analyses of variance; qualitative data were analyzed using qualitative content analysis by extraction.

Results

The transfer of cooperative planning to new settings was realized, though the impact varied by setting. While the interventions were developed and implemented in nursing care settings, interventions were developed but not implemented in the automotive mechatronics setting. In this context, intervention implementation was influenced by 11 key factors: champion, commitment, embedment, empowerment, engagement, health-promoting leadership, ownership, relevance, resources, responsibility, and strategic planning. Furthermore, the transfer of cooperative planning was influenced by different activity characteristics, namely elaboration & reconsideration, group composition, number of meetings, participation, period, prioritization, and researchers' input & support.

Discussion

The present article contributes to a better understanding of a co-creation approach utilized for physical activity promotion and provides new insights into (1) the transferability of cooperative planning and (2) the associated key factors influencing intervention implementation. The success of cooperative planning varied by setting and was influenced by several activity characteristics and key factors, some of which showed complex relationships. This raises the question of whether some settings might benefit more from a co-creation approach than others. Therefore, future co-creation initiatives should carefully consider the specific characteristics of a setting to select and apply the most appropriate approach.

Examining the sustainability and effectiveness of co-created physical activity interventions in vocational education and training: a multimethod evaluation

Background

Co-creation approaches are increasingly used in physical activity promotion to develop interventions tailored to the target group and setting. The resulting complexity of such interventions raises challenges in evaluation. Accordingly, little is known about the effectiveness of co-created interventions and the underlying processes that impact their sustainable implementation. In this study, we attempt to fill this gap by evaluating co-created multi-component physical activity interventions in vocational education and training in nursing care and automotive mechatronics regarding (1) their sustainable implementation at the institutional level and (2) the effectiveness of single intervention components at the individual level.

Methods

Following a multimethod design, we conducted a questionnaire survey (n = 7) and semi-structured interviews (n = 4) to evaluate the sustainability of the interventions. Quantitative data were analyzed descriptively, and qualitative data were analyzed using qualitative content analysis. To examine the interventions’ effectiveness, we conducted two non-randomized controlled trials (n = 111). Analysis of variance was used to examine differences between groups.

Results

At the institutional level, long-term implementation of single intervention components in nursing care was observed; in contrast, long-term implementation in automotive mechatronics was not observed. In this context, various factors at the outer contextual (e.g., COVID-19 pandemic), inner contextual (e.g., health-promoting leadership), intervention (e.g., acceptance), and personal levels (e.g., champion) influenced sustainability. At the individual level, no significant intervention effects were found for changes in physical activity behavior and physical activity-related health competence.

Conclusion

The role of co-creation on the effectiveness and sustainability of physical activity promotion in vocational education and training cannot be answered conclusively. Only in the nursing care sector, a co-creation approach appeared promising for long-term intervention implementation. Sustainable implementation depends on various influencing factors that should be considered from the outset. Demonstrating effectiveness at the individual level was challenging. To conclusively clarify both the role and impact of co-creation, methodologically complex and elaborate evaluation designs will be required in future research projects.

Trial registration

This study was retrospectively registered at clinicaltrials.gov on 24/08/2021 (NCT05018559).

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-022-13133-9.

Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia

Complex interventions, such as innovation platforms, pose challenges for evaluators. A variety of methodological approaches are often required to build a more complete and comprehensive understanding of how complex interventions work. In this paper, we outline and critically appraise a methodologically pluralist evaluation of an innovation platform to strengthen primary care for Aboriginal and Torres Strait Islander Australians. In doing so, we aim to identify lessons learned from the approach taken and add to existing literature on implementing evaluations in complex settings, such as innovation platforms. The pluralist design used four evaluation approaches-developmental evaluation, principles-focused evaluation, network analysis, and framework analysis-with differing strengths and challenges. Taken together, the multiple evaluation approaches yielded a detailed description and nuanced understanding of the formation, functioning and outcomes of the innovation platform that would be difficult to achieve with any single evaluation method. While a methodologically pluralist design may place additional pressure on logistical and analytic resources available, it enables a deeper understanding of the mechanisms that underlie complex interventions.

Pathways for collaboratively strengthening water and sanitation systems

Collaborative approaches are seen as a promising way to strengthen Water, Sanitation, and Hygiene (WASH) service delivery systems when challenges exceed the mandates and capabilities of any single entity. While collaborative approaches are well studied in high-income country contexts, current understanding of their application to international development contexts is limited. This paper uses fuzzy-set Qualitative Comparative Analysis to assess what conditions and pathways drove or impeded progress within eleven collaborative approaches for WASH service delivery in Eastern Africa. Evidence supported three main findings: (1) Government uptake of recommendations is necessary for progress but cannot be guaranteed solely by government participation in the collaboration, (2) different forms of problem identification are possible; problem scopes are often predefined to align with funders and partner government agendas, but flexible scopes that foster collective problem identification can reap benefits, and (3) hub convening power can be critical and convening power can be gained in different ways. Political dynamics, shifting priorities, and turnover undermine collaborative efforts, but collaborative approaches can still make progress in spite of turnover if funds are available for implementation of activities (i.e. in addition to funds for meetings and hub roles) and program implementers either facilitate collective problem identification or establish a hub with convening power. Yet even these tactics are vulnerable to instability, thus in highly unstable contexts, stakeholders and funders should be realistic from the outset about what they may be able to achieve. Building on existing theories of collaborative approaches, this work revealed that there is no single best design for collaborative approaches in WASH, rather, core elements worked together in different ways depending on the context.

Tingbjerg Changing Diabetes: a protocol for a long-term Supersetting initiative to promote health and prevent type 2 diabetes among people living in an ethnically and socioeconomically diverse neighbourhood in Copenhagen, Denmark

INTRODUCTION

Type 2 diabetes is an escalating public health problem closely related to socioeconomic position. There is increased risk of type 2 diabetes in disadvantaged neighbourhoods where education, occupation and income levels are low. Meanwhile, studies show positive health outcomes of participatory community interventions pointing towards the need for increased health promotion and prevention of type 2 diabetes in local communities. This study protocol describes Tingbjerg Changing Diabetes (TCD), a community-based health promotion and type 2 diabetes prevention initiative in Tingbjerg, a disadvantaged neighbourhood in Copenhagen, Denmark.

METHODS AND ANALYSIS

TCD is a long-term, complex intervention, implemented in three phases from 2014 to 2032, focusing on partnership formation (phase 1, 2014-2019), developing and implementing action for health (phase 2, 2019-2030) and diffusion of knowledge (phase 3, 2022-2032). The Supersetting principles act as guidelines for development and implementation of all intervention activities of TCD, involving several population groups in a variety of everyday life settings. The implementation of TCD draws on Community Action Research design and methodologies. TCD's evaluation and research strategy is interdisciplinary, pragmatic and multimethod, unfolding at three levels of operation: (A) evaluating activities, (B) researching cross-cutting topics, and (C) researching methods and approaches.

ETHICS AND DISSEMINATION

TCD has been approved by the Danish Data Protection Agency. Accordingly, the initiative is carried out in adherence to rules and regulations of the Danish Data Protection Agency. As data contain no personal identifiable or sensitive data, no clearance from the Danish National Ethical Review Board can be obtained according to Danish regulations. Citizen, local agents and stakeholders are engaged in the design and execution of TCD to ensure usefulness, reflexive interpretation of data, relevance and iterative progression of interventions. Results will be published in international peer-reviewed scientific journals, presented at conferences and through public media including TCD home page, podcasts and videos.

Service provider perspectives on implementing the NSW Get Healthy at Work program

Purpose: One approach increasingly used by governments to deliver on public initiatives is to partner with private enterprise through public–private partnerships. This study is a qualitative process evaluation of an Australian state-wide workplace health programme “Get Healthy at Work” from the currently under-researched perspective of the private service providers. Methods: Semi-structured interviews were conducted with nine service providers. Interviews were transcribed and analysed inductively. Results: Service providers reported an alignment of motives and skills between the programme and their organizations as a benefit of the partnership. However, they also described misalignments: between the potential and realized value of the programme to businesses and service providers; the programme cycle and business operational processes; and the capacity building approach and businesses’ expectations of the service.Conclusions: Although several hallmarks of a well-functioning private–public partnership were evident, misalignments of process and expectations challenged sustained partnership involvement by providers. Careful consideration must be given to the ongoing management functioning of cross-sector engagement and partnering in health promotion practice in order to ensure public health goals are being met, but also that the model is mutually sustainable.

Evaluation in the Real World: Decision Points and Rationales in Creating A Rigorous Study Designed to Convey Ecologically Valid Findings

Rigorously evaluating community-based interventions for multiply marginalized populations is fraught with challenges under the best of circumstances. This manuscript describes the methodology chosen to evaluate an innovative model designed to help survivors of intimate partner violence obtain safe and stable housing. We justify the choice of evaluation design from a community psychology perspective and detail why we believe the multi-method, multi-source design, that also focuses on social context, will maximize ecological validity and, therefore, propel the scale-up of the intervention if it is found to be effective. Longitudinal data are being collected from program recipients over time, the advocates who worked with them, agency service records, and monthly documentation of agency resources on hand that can impact services provided. Special attention is focused on capturing contextual information that can impact program success. While randomized control trials are still too often heralded as "the gold standard" for measuring intervention effectiveness, we maintain that the current design, which was developed in partnership with key community stakeholders, holds more promise when evaluating many community-based programs.

Co-creating physical activity interventions: a mixed methods evaluation approach

BACKGROUND

Co-creation strategies, such as cooperative planning, are promising as a means to ensure that physical activity interventions address real-world problems and are tailored to the target group. This has already been validated in diverse settings. However, questions targeting the transferability of cooperative planning to new settings and the key factors influencing its success or failure remain unclear. At the same time, co-creation processes are complex, and evaluation can be challenging. Following calls for detailed reporting, this paper describes the programme activities, the underlying logic, and methodological design of a study that aims to evaluate the transfer of cooperative planning to new settings and to explore the associated key determinants.

METHODS

Cooperative planning was utilized as a strategy to target physical activity promotion in three real-world German settings in the nursing care and automotive mechatronics sectors. This involved researchers working alongside stakeholders from practice and policy to conjointly develop new interventions to promote physical activity in physically demanding jobs. A pragmatic approach is used to evaluate both the transferability and key determinants of this strategy. We developed a logic model for this co-creation process that describes the underlying assumptions and guides the evaluation. The evaluation outcomes of this study include planning meetings, newly developed interventions, and the determinants that are likely to affect cooperative planning. Quantitative and qualitative data will be collected using questionnaires, documents, and interviews. The quantitative data will be analysed descriptively, while the qualitative data will mainly be analysed using qualitative content analysis, split by settings. Subsequently, data triangulation will be used to integrate the quantitative and qualitative findings, which will then be compared across all three settings.

DISCUSSION

The study findings will contribute to a better understanding of co-creation strategies, their transferability, and key determinants. The practical implications can include a checklist for assessing key determinants and a guideline for transferring cooperative planning into new settings to benefit more people. Ultimately, this study will help to advance co-creation strategies and may be relevant for researchers, practitioners, and policy-makers targeting physical activity promotion in various contexts.

TRIAL REGISTRATION

Open Science Framework: https://osf.io/r6xnt/ (retrospectively registered).

Evaluation of public health interventions from a complex systems perspective: A research methods review

INTRODUCTION

Applying a complex systems perspective to public health evaluation may increase the relevance and strength of evidence to improve health and reduce health inequalities. In this review of methods, we aimed to: (i) classify and describe different complex systems methods in evaluation applied to public health; and (ii) examine the kinds of evaluative evidence generated by these different methods.

METHODS

We adapted critical review methods to identify evaluations of public health interventions that used systems methods. We conducted expert consultation, searched electronic databases (Scopus, MEDLINE, Web of Science), and followed citations of relevant systematic reviews. Evaluations were included if they self-identified as using systems- or complexity-informed methods and if they evaluated existing or hypothetical public health interventions. Case studies were selected to illustrate different types of complex systems evaluation.

FINDINGS

Seventy-four unique studies met our inclusion criteria. A framework was developed to map the included studies onto different stages of the evaluation process, which parallels the planning, delivery, assessment, and further delivery phases of the interventions they seek to inform; these stages include: 1) theorising; 2) prediction (simulation); 3) process evaluation; 4) impact evaluation; and 5) further prediction (simulation). Within this framework, we broadly categorised methodological approaches as mapping, modelling, network analysis and 'system framing' (the application of a complex systems perspective to a range of study designs). Studies frequently applied more than one type of systems method.

CONCLUSIONS

A range of complex systems methods can be utilised, adapted, or combined to produce different types of evaluative evidence. Further methodological innovation in systems evaluation may generate stronger evidence to improve health and reduce health inequalities in our complex world.

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

BACKGROUND

Public health evaluation methods have been criticized for being overly reductionist and failing to generate suitable evidence for public health decision-making. A "complex systems approach" has been advocated to account for real world complexity. Qualitative methods may be well suited to understanding change in complex social environments, but guidance on applying a complex systems approach to inform qualitative research remains limited and underdeveloped. This systematic review aims to analyze published examples of process evaluations that utilize qualitative methods that involve a complex systems perspective and proposes a framework for qualitative complex system process evaluations.

METHODS AND FINDINGS

We conducted a systematic search to identify complex system process evaluations that involve qualitative methods by searching electronic databases from January 1, 2014-September 30, 2019 (Scopus, MEDLINE, Web of Science), citation searching, and expert consultations. Process evaluations were included if they self-identified as taking a systems- or complexity-oriented approach, integrated qualitative methods, reported empirical findings, and evaluated public health interventions. Two reviewers independently assessed each study to identify concepts associated with the systems thinking and complexity science traditions. Twenty-one unique studies were identified evaluating a wide range of public health interventions in, for example, urban planning, sexual health, violence prevention, substance use, and community transformation. Evaluations were conducted in settings such as schools, workplaces, and neighborhoods in 13 different countries (9 high-income and 4 middle-income). All reported some utilization of complex systems concepts in the analysis of qualitative data. In 14 evaluations, the consideration of complex systems influenced intervention design, evaluation planning, or fieldwork. The identified studies used systems concepts to depict and describe a system at one point in time. Only 4 evaluations explicitly utilized a range of complexity concepts to assess changes within the system resulting from, or co-occurring with, intervention implementation over time. Limitations to our approach are including only English-language papers, reliance on study authors reporting their utilization of complex systems concepts, and subjective judgment from the reviewers relating to which concepts featured in each study.

CONCLUSION

This study found no consensus on what bringing a complex systems perspective to public health process evaluations with qualitative methods looks like in practice and that many studies of this nature describe static systems at a single time point. We suggest future studies use a 2-phase framework for qualitative process evaluations that seek to assess changes over time from a complex systems perspective. The first phase involves producing a description of the system and identifying hypotheses about how the system may change in response to the intervention. The second phase involves following the pathway of emergent findings in an adaptive evaluation approach.

Using natural experiments to improve public health evidence: a review of context and utility for obesity prevention

Background

Natural experiments are increasingly valued as a way to assess the health impact of health and non-health interventions when planned controlled experimental research designs may be infeasible or inappropriate to implement. This study sought to investigate the value of natural experiments by exploring how they have been used in practice. The study focused on obesity prevention research as one complex programme area for applying natural experiment studies.

Methods

A literature search sought obesity prevention research from January 1997 to December 2017 and identified 46 population health studies that self-described as a natural experiment.

Results

The majority of studies identified were published in the last 5 years, illustrating a more recent adoption of such opportunities. The majority of studies were evaluations of the impact of policies (n = 19), such as assessing changes to food labelling, food advertising or taxation on diet and obesity outcomes, or were built environment interventions (n = 17), such as the impact of built infrastructure on physical activity or access to healthy food. Research designs included quasi-experimental, pre-experimental and non-experimental methods. Few studies applied rigorous research designs to establish stronger causal inference, such as multiple pre/post measures, time series designs or comparison of change against an unexposed group. In general, researchers employed techniques to enhance the study utility but often were limited in the use of more rigorous study designs by ethical considerations and/or the particular context of the intervention.

Conclusion

Greater recognition of the utility and versatility of natural experiments in generating evidence for complex health issues like obesity prevention is needed. This review suggests that natural experiments may be underutilised as an approach for providing evidence of the effects of interventions, particularly for evaluating health outcomes of interventions when unexpected opportunities to gather evidence arise.

ActEarly: a City Collaboratory approach to early promotion of good health and wellbeing

Economic, physical, built, cultural, learning, social and service environments have a profound effect on lifelong health. However, policy thinking about health research is dominated by the ‘biomedical model’ which promotes medicalisation and an emphasis on diagnosis and treatment at the expense of prevention. Prevention research has tended to focus on ‘downstream’ interventions that rely on individual behaviour change, frequently increasing inequalities. Preventive strategies often focus on isolated leverage points and are scattered across different settings. This paper describes a major new prevention research programme that aims to create City Collaboratory testbeds to support the identification, implementation and evaluation of upstream interventions within a whole system city setting. Prevention of physical and mental ill-health will come from the cumulative effect of multiple system-wide interventions. Rather than scatter these interventions across many settings and evaluate single outcomes, we will test their collective impact across multiple outcomes with the goal of achieving a tipping point for better health. Our focus is on early life (ActEarly) in recognition of childhood and adolescence being such critical periods for influencing lifelong health and wellbeing.