Making sense of the emerging conversation in evaluation about systems thinking and complexity science

Exploring the complexity of the implementation determinants of human papillomavirus vaccination in Africa through a systems thinking lens: A rapid review

A rapid review was conducted to explore the implementation determinants of human papillomavirus (HPV) vaccination in the World Health Organization African Region and describe their dynamic relationship. PubMed and Google Scholar were searched in October 2023 to find relevant literature. A total of 64 published studies that reported factors affecting HPV vaccination were identified. Analysis of identified factors yielded 74 implementation determinants of HPV vaccination across the five domains of the Consolidated Framework for Implementation Research (CFIR): two (2.70%) were in the innovation domain, seven (9.46%) were in the outer setting domain, 14 (18.92%) were in the inner setting domain, 37 (50%) were in the individual domain and 14 (18.92%) were in the implementation process domain. A causal loop diagram of these implementation determinants revealed four balancing and seven reinforcing loops. Applying systems lens promoted a more holistic understanding of the implementation determinants of HPV vaccination, exposing leverage points for interventions.

How to Evaluate the Effectiveness of Health Promotion Actions Developed Through Youth-Centered Participatory Action Research

Most actions targeting children's health behaviors have limited involvement of children in the development, potentially contributing to disappointing effectiveness. Therefore, in the 3-year "Kids in Action" study, 9- to 12-year-old children from a lower-socioeconomic neighborhood were involved as coresearchers in the development, implementation, and evaluation of actions targeting health behaviors. The current study describes the controlled trial that evaluated the effects on children's energy balance-related behaviors, physical fitness, and self-rated health, as well as experienced challenges and recommendations for future evaluations. Primary school children from the three highest grades of four intervention and four control schools were eligible for participation. Outcome measures assessed at baseline, and at 1- and 2-year follow-up were as follows: motor fitness by the MOPER test (N = 656, N = 485, N = 608, respectively), physical activity and sedentary behavior by accelerometry (N = 223, N = 149, N = 164, respectively), and consumption of sugar sweetened beverages and snacks and self-rated health by a questionnaire (N = 322, N = 281, N = 275, respectively). Mixed-model analyses were performed adjusted for clustering within schools and relevant confounders. Significant beneficial intervention effects were found on self-reported consumption of energy/sports drinks at T2 versus T0, and on total time and ≥5-minute bouts of moderate-to-vigorous physical activity at T1 versus T0. Significant adverse effects were found on "speed and agility" and "coordination and upper-limb speed." No other significant effects were found. The inconsistent intervention effects may be explained by the dynamic cohort and suboptimal outcome measures. We advise future studies with a similar approach to apply alternative evaluation designs, such as the delayed baseline design.

Systems Thinking and Complexity Science Methods and the Policy Process in Non-communicable Disease Prevention: A Systematic Scoping Review

BACKGROUND

Given the complex determinants of non-communicable diseases (NCDs), and the dynamic policy landscape, researchers and policymakers are exploring the use of systems thinking and complexity science (STCS) in developing effective policies. The aim of this review is to systematically identify and analyse existing applications of STCS-informed methods in NCD prevention policy.

METHODS

Systematic scoping review: We searched academic databases (Medline, Scopus, Web of Science, EMBASE) for all publications indexed by 13 October 2020, screening titles, abstracts and full texts and extracting data according to published guidelines. We summarised key data from each study, mapping applications of methods informed by STCS to policy process domains. We conducted a thematic analysis to identify advantages, limitations, barriers and facilitators to using STCS.

RESULTS

4681 papers were screened and 112 papers were included in this review. The most common policy areas were tobacco control, obesity prevention and physical activity promotion. Methods applied included system dynamics modelling, agent-based modelling and concept mapping. Advantages included supporting evidence-informed decision-making; modelling complex systems and addressing multi-sectoral problems. Limitations included the abstraction of reality by STCS methods, despite aims of encompassing greater complexity. Challenges included resource-intensiveness; lack of stakeholder trust in models; and results that were too complex to be comprehensible to stakeholders. Ensuring stakeholder ownership and presenting findings in a user-friendly way facilitated STCS use.

CONCLUSION

This review maps the proliferating applications of STCS methods in NCD prevention policy. STCS methods have the potential to generate tailored and dynamic evidence, adding robustness to evidence-informed policymaking, but must be accessible to policy stakeholders and have strong stakeholder ownership to build consensus and change stakeholder perspectives. Evaluations of whether, and under what circumstances, STCS methods lead to more effective policies compared to conventional methods are lacking, and would enable more targeted and constructive use of these methods.

A problem-bound evaluation approach

Traditional modes of evaluation can be constraining when seeking to improve persistent societal problems within complex systems. Frameworks that assume stability and predictability may not match the system dynamics within which a program resides. This paper seeks to further evaluators' understanding of how to improve problems in complex systems by offering a "problem-bound" evaluation approach. This model is grounded in theoretical concepts from complexity science, complex adaptive systems, systems thinking, and improvement science. As such, it provides an agile and flexible framework for continually learning from emergent findings related to the problem. By offering this model, we seek to contribute to the limited knowledge base on how to apply systems change theoretical ideas in evaluation practice, along with practical implications regarding the evaluator's role.

Addressing alcohol-related harms in the local night-time economy: a qualitative process evaluation from a complex systems perspective

OBJECTIVES

English local authorities (LAs) are interested in reducing alcohol-related harms and may use discretionary powers such as the Late Night Levy (LNL) to do so. This study aims to describe how system stakeholders hypothesise the levy may generate changes and to explore how the system, its actors and the intervention adapt and co-evolve over time.

DESIGN

A process evaluation from a complex systems perspective, using qualitative methods.

SETTING

A London LA with high densities of residential and commercial properties, which implemented the LNL in 2014.

PARTICIPANTS

Data were generated through interviews with LNL implementers and alcohol consumers, observations in bars and during LNL patrols and documentary review.

INTERVENTION

The LNL allows LAs to charge late-night alcohol retailers an annual fee (£299-£4440) to manage and police the night-time economy (NTE).

RESULTS

When the LNL was being considered, stakeholders from different interest groups advanced diverse opinions about its likely impacts while rarely referencing supporting research evidence. Proponents of the levy argued it could reduce crime and anti-social behaviour by providing additional funds to police and manage the NTE. Critics of the levy hypothesised adverse consequences linked to claims that the intervention would force venues to vary their hours or close, cluster closing times, reduce NTE diversity and undermine public-private partnerships. In the first 2 years, levy-funded patrols developed relationships with the licensed trade and the public. The LNL did not undermine public-private partnerships and while some premises varied their hours, these changes did not undermine the intervention's viability, nor significantly cluster venue closing times, nor obviously damage the area's reputation for having a diverse NTE.

CONCLUSIONS

This study applies a framework for process evaluation from a complex systems perspective. The evaluation could be extended to measure alcohol-related outcomes and to consider the interplay between the national and local systems.

How to do (or not to do)…Using Causal Loop Diagrams for Health System Research in Low- and Middle-Income Settings

Causal loop diagrams (CLDs) are a systems thinking method that can be used to visualize and unpack complex health system behaviour. They can be employed prospectively or retrospectively to identify the mechanisms and consequences of policies or interventions designed to strengthen health systems and inform discussion with policymakers and stakeholders on actions that may alleviate sub-optimal outcomes. Whilst the use of CLDs in health systems research has generally increased, there is still limited use in low- and middle-income settings. In addition to their suitability for evaluating complex systems, CLDs can be developed where opportunities for primary data collection may be limited (such as in humanitarian or conflict settings) and instead be formulated using secondary data, published or grey literature, health surveys/reports and policy documents. The purpose of this paper is to provide a step-by-step guide for designing a health system research study that uses CLDs as their chosen research method, with particular attention to issues of relevance to research in low- and middle-income countries (LMICs). The guidance draws on examples from the LMIC literature and authors’ own experience of using CLDs in this research area. This paper guides researchers in addressing the following four questions in the study design process; (1) What is the scope of this research? (2) What data do I need to collect or source? (3) What is my chosen method for CLD development? (4) How will I validate the CLD? In providing supporting information to readers on avenues for addressing these key design questions, authors hope to promote CLDs for wider use by health system researchers working in LMICs.

Utilizing Causal Loop Diagramming to Explore a Research and Evaluation Capacity Building Partnership

The capacity to engage in research, evaluation and evidence-informed decision-making supports effective public health policy and practice. Little is known about partnership-based approaches that aim to build capacity across a system or how to evaluate them. This study examines the impacts of a research and evaluation capacity building partnership called the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (hereafter, SiREN). SiREN aims to strengthen capacity across a system of clinical and medical services and government and non-government organizations. These organizations are connected through their shared aim of preventing and managing sexually transmissible infections and blood-borne viruses. To examine SiREN, systems concepts and methods were used. Data were collected from SiREN organizational documents (n = 42), a survey tool (n = 104), in-depth interviews (n = 17), a workshop and three meetings with SiREN stakeholders and used to develop two causal loop diagrams. Findings show engagement with SiREN was influenced by a complex interplay of contextual (e.g., organizational capacity) and process (e.g., presence of trusting relationships) factors. SiREN contributed to system level changes, including increased resources for research and evaluation, the development of networks and partnerships that led to more efficient responses to emerging health issues, evidence sharing, and sustainable research and evaluation practice. The use of causal loop diagrams enabled the identification of key leverage points that SiREN can use for continuous improvement or evaluation. The focus on how contextual factors influenced SiREN's ability to create change provides valuable information for researchers, policymakers or practitioners seeking to develop a similar partnership.

Embedded Researchers as Part of a Whole Systems Approach to Physical Activity: Reflections and Recommendations

Whole systems approaches are increasingly being advocated as a way of responding to complex public health priorities such as obesity and physical inactivity. Due to the complex and adaptive nature of such systems, researchers are increasingly being embedded within host organisations (i.e., those which facilitate the whole systems approach) to work with key stakeholders to illuminate and understand mechanisms of change and develop a culture of continuous improvement. While previous literature has reported on the benefits and challenges of embedded researchers in health care, little is known about the experiences and learnings of those situated within these complex whole systems approaches. In this paper, we present our reflections of being embedded researchers within four distinct whole systems approaches and outline recommendations and considerations for commissioners working with or seeking support from an embedded researcher.

Presenting a comprehensive multi-scale evaluation framework for participatory modelling programs: A scoping review

INTRODUCTION

Systems modelling and simulation can improve understanding of complex systems to support decision making, better managing system challenges. Advances in technology have facilitated accessibility of modelling by diverse stakeholders, allowing them to engage with and contribute to the development of systems models (participatory modelling). However, despite its increasing applications across a range of disciplines, there is a growing need to improve evaluation efforts to effectively report on the quality, importance, and value of participatory modelling. This paper aims to identify and assess evaluation frameworks, criteria, and/or processes, as well as to synthesize the findings into a comprehensive multi-scale framework for participatory modelling programs.

MATERIALS AND METHODS

A scoping review approach was utilized, which involved a systematic literature search via Scopus in consultation with experts to identify and appraise records that described an evaluation framework, criteria, and/or process in the context of participatory modelling. This scoping review is registered with the Open Science Framework.

RESULTS

The review identified 11 studies, which varied in evaluation purposes, terminologies, levels of examination, and time points. The review of studies highlighted areas of overlap and opportunities for further development, which prompted the development of a comprehensive multi-scale evaluation framework to assess participatory modelling programs across disciplines and systems modelling methods. The framework consists of four categories (Feasibility, Value, Change/Action, Sustainability) with 30 evaluation criteria, broken down across project-, individual-, group- and system-level impacts.

DISCUSSION & CONCLUSION

The presented novel framework brings together a significant knowledge base into a flexible, cross-sectoral evaluation effort that considers the whole participatory modelling process. Developed through the rigorous synthesis of multidisciplinary expertise from existing studies, the application of the framework can provide the opportunity to understand practical future implications such as which aspects are particularly important for policy decisions, community learning, and the ongoing improvement of participatory modelling methods.

Evaluation proposal of a national community-based obesity prevention programme: a novel approach considering the complexity perspective

Community-based obesity prevention programmes are considered an important strategy to curb the obesity epidemic. The JOGG (Youth At a Healthy Weight) approach is a large-scale community-based programme for childhood obesity prevention in the Netherlands that has been implemented over the past ten years. Practice-based development of the programme, both at the national and local level, increasingly poses challenges for its evaluation. One considerable challenge is the increasing acknowledgement of the complexity in the JOGG-approach, characterized by (a) objectives that vary locally, (b) adaptions to the programme over time in response to a community's shifting needs, challenges and opportunities, and (c) emergent outcomes and non-linear causality.We propose an evaluation framework that highlights elements of the complex local practice, including the local programme theory, implementation, adaption, the influence of context and feedback loops and intended as well as emergent and unintended outcomes. By studying each of these elements in practice, we hope to learn about principles that guide effective obesity prevention across contexts. The results of the proposed evaluation will inform both practice and research.Considering complexity in evaluation is a relatively new challenge in public health and therefore an emergent research area. The proposed framework for complex evaluations allows to retrospectively evaluate a programme that was implemented and developed in practice, and enables us to learn from practice-based experiences. Following the ISBNPA Dare2Share initiative, we kindly invite other researchers in the field to share their ideas and experiences regarding integration of complexity in evaluation.

Critical factors that affect the functioning of a research and evaluation capacity building partnership: A causal loop diagram

Introduction

Public health policy and practice is strengthened by the application of quality evidence to decision making. However, there is limited understanding of how initiatives that support the generation and use of evidence in public health are operationalised. This study examines factors that support the internal functioning of a partnership, the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN). SiREN aims to build research and evaluation capacity and increase evidence-informed decision making in a public health context.

Methods

This study was informed by systems concepts. It developed a causal loop diagram, a type of qualitative system model that illustrated the factors that influence the internal operation of SiREN. The causal loop diagram was developed through an iterative and participatory process with SiREN staff and management (n = 9) via in-depth semi-structured interviews (n = 4), workshops (n = 2), and meetings (n = 6).

Results

Findings identified critical factors that affected the functioning of SiREN. Central to SiREN’s ability to meet its aims was its capacity to adapt within a dynamic system. Adaptation was facilitated by the flow of knowledge between SiREN and system stakeholders and the expertise of the team. SiREN demonstrated credibility and capability, supporting development of new, and strengthening existing, partnerships. This improved SiREN’s ability to be awarded new funding and enhanced its sustainability and growth. SiREN actively balanced divergent stakeholder interests to increase sustainability.

Conclusion

The collaborative development of the diagram facilitated a shared understanding of SiREN. Adaptability was central to SiREN achieving its aims. Monitoring the ability of public health programs to adapt to the needs of the systems in which they work is important to evaluate effectiveness. The detailed analysis of the structure of SiREN and how this affects its operation provide practical insights for those interested in establishing a similar project.

Beyond financial proxies in Cohesion Policy inputs' monitoring: A system dynamics approach

European Union's Cohesion Policy aims to foster development and reduce disparities among regions by redistributing more than one-third of the European budget. Given the policy's importance and complexity, an elaborated monitoring and evaluation system has been established. While attention has been dedicated to evaluating policy impact, the monitoring of inputs (i.e., allocated financial resources) has been limited to the control of financial dimensions (i.e., funds' absorption rate). As the implementation process entails a sequence of steps, this research explores whether financial proxies alone are adequate to monitor the policy inputs. To test this hypothesis, a system dynamics model is built. Simulations highlight that the absorption rate captures shocks that might occur during the inputs' expenditure with significant delay. To that end, we elaborate three novel operative monitoring indicators (i.e., funds' demand, funds' offer, procedural efficiency), which may overcome the financial indicators' mono-dimensionality and time lags' limitations.

Systems thinking and complexity science methods and the policy process in non-communicable disease prevention: a systematic scoping review protocol

INTRODUCTION

Given the complex causal origins of many non-communicable diseases (NCDs), and the complex landscapes in which policies designed to tackle them are made and unfold, the need for systems thinking and complexity science (STCS) in developing effective policy solutions has been emphasised. While numerous methods informed by STCS have been applied to the policy process in NCD prevention, these applications have not been systematically catalogued. The aim of this scoping review is to identify existing applications of methods informed by STCS to the policy process for NCD prevention, documenting which domains of the policy process they have been applied to.

METHODS AND ANALYSIS

A systematic scoping review methodology will be used.

IDENTIFICATION

We will search Medline, SCOPUS, Embase and Web of Science using search terms combining STCS, NCD prevention and the policy process. All records published in English will be eligible for inclusion, regardless of study design.

SELECTION

We will screen titles and abstracts and extract data according to published guidelines for scoping reviews. In order to determine the quality of the included studies, we will use the approach developed by Dixon-Woods et al, excluding studies identified as fatally flawed, and determining the credibility and contribution of included studies.

SYNTHESIS

We will identify relevant studies, summarising key data from each study and mapping applications of methods informed by STCS to different parts of the policy process. Review findings will provide a useful reference for policy-makers, outlining which domains of the policy process different methods have been applied to.

ETHICS AND DISSEMINATION

Formal ethical approval is not required, as the study does not involve primary data collection. The findings of this study will be disseminated through a peer-reviewed publication, presentations and summaries for key stakeholders.

A qualitative system dynamics approach to modeling the causes and effects of design deficiencies in construction projects

Purpose

Poor quality of design documents and Design Deficiencies (DDs) are critical factors that can potentially cause time and cost overrun and lead to project failure. Despite the importance of this issue, few studies have been conducted to explore the underlying relations among the causes and effects of DDs, and it is not clear how they exert influence on each other. However, it is imperative to have a clear understanding of their interrelationships so as to devise intervention strategies. The purpose of this study is, therefore, to explore these causes and effects.

Design/methodology/approach

To this aim, a qualitative system dynamics model was provided to visualize how various factors interact with one another. The model was developed and validated by drawing on the expertise of industry-related experts. The key causes and effects of the DDs were identified through the investigation of the model.

Findings

It was concluded that design management of project and the composition of design teams are critical. The obtained results also showed that more attention should be paid to the enhancement of knowledge circulation and communication within design firms and increasing the awareness of managers and owners from the technical consequences of their managerial decisions.

Research limitations/implications

It is worth mentioning that the design process could be deeply affected by the quality, frequency and strength of relationships among the project’s parties. Early involvement of contractors, for example, can significantly contribute to enhancing design quality. This study investigates factors’ interactions in projects delivered using the traditional design-bid-build method. As relationships between the project’s parties could be different in various project delivery methods, further research is indeed needed to assess factors’ interactions in projects delivered using other methods.

Originality/value

It is believed that the proposed model by providing a better insight into the connections of causes and effects could aid project managers of different parties with planning the strategies to reduce DDs and mitigate their effects.

Re-Strategising Government Palliative Support Systems in Tackling the Challenges of COVID-19 Lockdown in Lagos State, Nigeria

This research covered the palliative material distribution project in Lagos State, Nigeria, during the lockdown and restriction on movement because of COVID-19 pandemic. The study engaged reports from extant literature and tertiary data sources such as the newspapers, the Internet, and website source. Relevant data gathered from these sources were conceptually analysed and discussed along with extant literature. The research also adopted the use of rich pictures for further presentation of the research. A key finding from the research is the need to redesign the distribution process and consider proper structuring of the project. And further engagement with the affected stakeholders to address crucial issues such as marginalisation that can defeat the purpose, which is to emancipate the affected indigents of the State from the harrowing effects of COVID-19 lockdown and other emergencies-related critical incidences such as starvation and criminalities. The study proposed a model for a resilience approach to effectively address the current palliative distribution process challenges in Lagos State, the epicentre of COVID-19 in Nigeria.

Evaluation of public health interventions from a complex systems perspective: A research methods review

INTRODUCTION

Applying a complex systems perspective to public health evaluation may increase the relevance and strength of evidence to improve health and reduce health inequalities. In this review of methods, we aimed to: (i) classify and describe different complex systems methods in evaluation applied to public health; and (ii) examine the kinds of evaluative evidence generated by these different methods.

METHODS

We adapted critical review methods to identify evaluations of public health interventions that used systems methods. We conducted expert consultation, searched electronic databases (Scopus, MEDLINE, Web of Science), and followed citations of relevant systematic reviews. Evaluations were included if they self-identified as using systems- or complexity-informed methods and if they evaluated existing or hypothetical public health interventions. Case studies were selected to illustrate different types of complex systems evaluation.

FINDINGS

Seventy-four unique studies met our inclusion criteria. A framework was developed to map the included studies onto different stages of the evaluation process, which parallels the planning, delivery, assessment, and further delivery phases of the interventions they seek to inform; these stages include: 1) theorising; 2) prediction (simulation); 3) process evaluation; 4) impact evaluation; and 5) further prediction (simulation). Within this framework, we broadly categorised methodological approaches as mapping, modelling, network analysis and 'system framing' (the application of a complex systems perspective to a range of study designs). Studies frequently applied more than one type of systems method.

CONCLUSIONS

A range of complex systems methods can be utilised, adapted, or combined to produce different types of evaluative evidence. Further methodological innovation in systems evaluation may generate stronger evidence to improve health and reduce health inequalities in our complex world.

Applying systems thinking to knowledge mobilisation in public health

CONTEXT

Knowledge mobilisation (KM) is a vital strategy in efforts to improve public health policy and practice. Linear models describing knowledge transfer and translation have moved towards multi-directional and complexity-attuned approaches where knowledge is produced and becomes meaningful through social processes. There are calls for systems approaches to KM but little guidance on how this can be operationalised. This paper describes the contribution that systems thinking can make to KM and provides guidance about how to put it into action.

METHODS

We apply a model of systems thinking (which focuses on leveraging change in complex systems) to eight KM practices empirically identified by others. We describe how these models interact and draw out some key learnings for applying systems thinking practically to KM in public health policy and practice. Examples of empirical studies, tools and targeted strategies are provided.

FINDINGS

Systems thinking can enhance and fundamentally transform KM. It upholds a pluralistic view of knowledge as informed by multiple parts of the system and reconstituted through use. Mobilisation is conceived as a situated, non-prescriptive and potentially destabilising practice, no longer conceptualised as a discrete piece of work within wider efforts to strengthen public health but as integral to and in continual dialogue with those efforts. A systems approach to KM relies on contextual understanding, collaborative practices, addressing power imbalances and adaptive learning that responds to changing interactions between mobilisation activities and context.

CONCLUSION

Systems thinking offers valuable perspectives, tools and strategies to better understand complex problems in their settings and for strengthening KM practice. We make four suggestions for further developing empirical evidence and debate about how systems thinking can enhance our capacity to mobilise knowledge for solving complex problems - (1) be specific about what is meant by 'systems thinking', (2) describe counterfactual KM scenarios so the added value of systems thinking is clearer, (3) widen conceptualisations of impact when evaluating KM, and (4) use methods that can track how and where knowledge is mobilised in complex systems.

The coronavirus response: Boxed in by models

Science has a mixed record when it comes to predicting the future. Engineers build bridges based on foreknowledge of the forces that they are likely to encounter – and their constructions tend to withstand the test of time. Predicting the future course of epidemics and building intervention to contain them are much more precarious. And yet simulation models produced in prestigious centres for mathematical biology have played a significant role informing coronavirus policy in the United Kingdom and elsewhere. The predictive uncertainties include the inherent variability of the pathogen, considerable variation in host population immunity as well as the concern of this article, namely, the constantly adapting human judgements of those designing, implementing and experiencing the national response to an outbreak. Assumptions about how interventions are implemented and how people will react are, of course, built into modelling scenarios – but these estimates depict behavioural change in fixed, stimulus-response terms. Real reactions to the complex restrictions introduced to combat the virus unfold in scores of different pathways – people comply, they resist, they learn, they grow weary, they change their minds, they seek exceptions and so on. Model building is intrinsically speculative, and it is important that crisis management is not boxed in by its latent simplifications. A more pluralistic evidence base needs to be drawn on, to understand how complex interventions operate within complex societies.

Qualitative process evaluation from a complex systems perspective: A systematic review and framework for public health evaluators

BACKGROUND

Public health evaluation methods have been criticized for being overly reductionist and failing to generate suitable evidence for public health decision-making. A "complex systems approach" has been advocated to account for real world complexity. Qualitative methods may be well suited to understanding change in complex social environments, but guidance on applying a complex systems approach to inform qualitative research remains limited and underdeveloped. This systematic review aims to analyze published examples of process evaluations that utilize qualitative methods that involve a complex systems perspective and proposes a framework for qualitative complex system process evaluations.

METHODS AND FINDINGS

We conducted a systematic search to identify complex system process evaluations that involve qualitative methods by searching electronic databases from January 1, 2014-September 30, 2019 (Scopus, MEDLINE, Web of Science), citation searching, and expert consultations. Process evaluations were included if they self-identified as taking a systems- or complexity-oriented approach, integrated qualitative methods, reported empirical findings, and evaluated public health interventions. Two reviewers independently assessed each study to identify concepts associated with the systems thinking and complexity science traditions. Twenty-one unique studies were identified evaluating a wide range of public health interventions in, for example, urban planning, sexual health, violence prevention, substance use, and community transformation. Evaluations were conducted in settings such as schools, workplaces, and neighborhoods in 13 different countries (9 high-income and 4 middle-income). All reported some utilization of complex systems concepts in the analysis of qualitative data. In 14 evaluations, the consideration of complex systems influenced intervention design, evaluation planning, or fieldwork. The identified studies used systems concepts to depict and describe a system at one point in time. Only 4 evaluations explicitly utilized a range of complexity concepts to assess changes within the system resulting from, or co-occurring with, intervention implementation over time. Limitations to our approach are including only English-language papers, reliance on study authors reporting their utilization of complex systems concepts, and subjective judgment from the reviewers relating to which concepts featured in each study.

CONCLUSION

This study found no consensus on what bringing a complex systems perspective to public health process evaluations with qualitative methods looks like in practice and that many studies of this nature describe static systems at a single time point. We suggest future studies use a 2-phase framework for qualitative process evaluations that seek to assess changes over time from a complex systems perspective. The first phase involves producing a description of the system and identifying hypotheses about how the system may change in response to the intervention. The second phase involves following the pathway of emergent findings in an adaptive evaluation approach.

'The opportunity to have their say'? Identifying mechanisms of community engagement in local alcohol decision-making

BACKGROUND

Engaging the community in decisions-making is recognised as important for improving public health, and is recommended in global alcohol strategies, and in national policies on controlling alcohol availability. Yet there is little understanding of how to engage communities to influence decision-making to help reduce alcohol-related harms. We sought to identify and understand mechanisms of community engagement in decision-making concerning the local alcohol environment in England.

METHODS

We conducted case studies in three local government areas in England in 2018, purposively selected for examples of community engagement in decisions affecting the local alcohol environment. We conducted 20 semi-structured interviews with residents, workers, local politicians and local government practitioners, and analysed documents linked to engagement and alcohol decision-making.

RESULTS

Four rationales for engaging the community in decision-making affecting the alcohol environment were identified: i) as part of statutory decision-making processes; ii) to develop new policies; iii) as representation on committees; and iv) occurring through relationship building. Many of the examples related to alcohol licensing processes, but also local economy and community safety decision-making. The impact of community inputs on decisions was often not clear, but there were a few instances of engagement influencing the process and outcome of decision-making relating to the alcohol environment.

CONCLUSIONS

While influencing statutory licensing decision-making is challenging, community experiences of alcohol-related harms can be valuable 'evidence' to support new licensing policies. Informal relationship-building between communities and local government is also beneficial for sharing information about alcohol-related harms and to facilitate future engagement. However, care must be taken to balance the different interests among diverse community actors relating to the local alcohol environment, and extra support is needed for those with least capacity to engage but who face more burden of alcohol-related harms, to avoid compounding existing inequalities.

What Can Policy-Makers Get Out of Systems Thinking? Policy Partners' Experiences of a Systems-Focused Research Collaboration in Preventive Health

BACKGROUND

There is increasing interest in using systems thinking to tackle 'wicked' policy problems in preventive health, but this can be challenging for policy-makers because the literature is amorphous and often highly theoretical. Little is known about how best to support health policy-makers to gain skills in understanding and applying systems thinking for policy action.

METHODS

In-depth interviews were conducted with 18 policy-makers who are participating in an Australian research collaboration that uses a systems approach. Our aim was to explore factors that support policy-makers to use systems approaches, and to identify any impacts of systems thinking on policy thinking or action, including the pathways through which these impacts occurred.

RESULTS

All 18 policy-makers agreed that systems thinking has merit but some questioned its practical policy utility. A small minority were confused about what systems thinking is or which approaches were being used in the collaboration. The majority were engaged with systems thinking and this group identified concrete impacts on their work. They reported using systems-focused research, ideas, tools and resources in policy work that were contributing to the development of practical methodologies for policy design, scaling up, implementation and evaluation; and to new prevention narratives. Importantly, systems thinking was helping some policy-makers to reconceptualise health problems and contexts, goals, potential policy solutions and methods. In short, they were changing how they think about preventive health.

CONCLUSION

These results show that researchers and policy-makers can put systems thinking into action as part of a research collaboration, and that this can result in discernible impacts on policy processes. In this case, action-oriented collaboration and capacity development over a 5-year period facilitated mutual learning and practical application. This indicates that policy-makers can get substantial applied value from systems thinking when they are involved in extended co-production processes that target policy impact and are supported by responsive capacity strategies.

The relevance of U.S. Strategic Highway Safety Plans in a future context

While road safety in the United States (U.S.) has been continually improving since the 1970's, there are indications that these improvements are becoming increasingly difficult to sustain. Strategic Highway Safety Plans (SHSPs) are prepared by States to guide road safety management, however assessing the appropriateness of these plans remains a significant challenge, especially for the future in which they are to be applied. This study developed a new methodology to assess SHSPs from the perspectives of comprehensive system-based safety management and relevant future issues that can be applied before SHSPs are implemented, thereby avoiding long periods after implementation before assessing the appropriateness of the plans. A rating scale was developed and applied to assess 48 U.S. SHSPs against two key criteria: 1. a comprehensive framework for road safety, and 2. the anticipated changing, difficult and unpredictable nature of future transport and its context. The analysis concluded that current SHSPs have good national oversight with several strengths but were weak in some areas of content and did not address future challenges. Improvements are suggested to strengthen the plans' thoroughness by being consistent with systems theory and practice, as well as ways that these SHSPs can be more resilient to future circumstances. Implementing the recommendations in this paper provides the opportunity to adopt a system-based safety management practice that has been successful in other hazardous industries. Doing so is expected to most efficiently and effectively continue the recent improvements to road safety, which is likely to be increasingly difficult otherwise.

Systemic View to Understanding Design Change Causation and Exploitation of Communications and Knowledge

Frequent changes pervasively undermine project schedule performance. Despite voluminous research into project delays, however, the persistence of the problem demands that a systemic approach be adopted to investigate design change causation and to explore the efficacy of communications and knowledge as strategic project controls. Drawing on a hybrid research design, the critical variables are identified and mapped onto causal loop diagrams to enable practical holism. The findings indicate that effective project communications engenders collaborative team dynamics and collective learning, whereas project learning contributes to knowledge reuse and the improved expert judgment needed for transforming design change management and schedule control.

Periodic reflections: a method of guided discussions for documenting implementation phenomena

BACKGROUND

Ethnography has been proposed as a valuable method for understanding how implementation occurs within dynamic healthcare contexts, yet this method can be time-intensive and challenging to operationalize in pragmatic implementation. The current study describes an ethnographically-informed method of guided discussions developed for use by a multi-project national implementation program.

METHODS

The EMPOWER QUERI is conducting three projects to implement innovative care models in VA women's health for high-priority health concerns - prediabetes, cardiovascular risk, and mental health - utilizing the Replicating Effective Programs (REP) implementation strategy enhanced with stakeholder engagement and complexity science. Drawing on tenets of ethnographic research, we developed a lightly-structured method of guided "periodic reflections" to aid in documenting implementation phenomena over time. Reflections are completed as 30-60 min telephone discussions with implementation team members at monthly or bi-monthly intervals, led by a member of the implementation core. Discussion notes are coded to reflect key domains of interest and emergent themes, and can be analyzed singly or in triangulation with other qualitative and quantitative assessments to inform evaluation and implementation activities.

RESULTS

Thirty structured reflections were completed across the three projects during a 15-month period spanning pre-implementation, implementation, and sustainment activities. Reflections provide detailed, near-real-time information on projects' dynamic implementation context, including characteristics of implementation settings and changes in the local or national environment, adaptations to the intervention and implementation plan, and implementation team sensemaking and learning. Reflections also provide an opportunity for implementation teams to engage in recurring reflection and problem-solving.

CONCLUSIONS

To implement new, complex interventions into dynamic organizations, we must better understand the implementation process as it unfolds in real time. Ethnography is well suited to this task, but few approaches exist to aid in integrating ethnographic insights into implementation research. Periodic reflections show potential as a straightforward and low-burden method for documenting events across the life cycle of an implementation effort. They offer an effective means for capturing information on context, unfolding process and sensemaking, unexpected events, and diverse viewpoints, illustrating their value for use as part of an ethnographically-minded implementation approach.

TRIAL REGISTRATION

The two implementation research studies described in this article have been registered as required: Facilitating Cardiovascular Risk Screening and Risk Reduction in Women Veterans (NCT02991534); and Implementation of Tailored Collaborative Care for Women Veterans (NCT02950961).

Agent-based modeling as a tool for program design and evaluation

Recently, systems thinking and systems science approaches have gained popularity in the field of evaluation; however, there has been relatively little exploration of how evaluators could use quantitative tools to assist in the implementation of systems approaches therein. The purpose of this paper is to explore potential uses of one such quantitative tool, agent-based modeling, in evaluation practice. To this end, we define agent-based modeling and offer potential uses for it in typical evaluation activities, including: engaging stakeholders, selecting an intervention, modeling program theory, setting performance targets, and interpreting evaluation results. We provide demonstrative examples from published agent-based modeling efforts both inside and outside the field of evaluation for each of the evaluative activities discussed. We further describe potential pitfalls of this tool and offer cautions for evaluators who may chose to implement it in their practice. Finally, the article concludes with a discussion of the future of agent-based modeling in evaluation practice and a call for more formal exploration of this tool as well as other approaches to simulation modeling in the field.