Interfacing theories of program with theories of evaluation for advancing evaluation practice: Reductionism, systems thinking, and pragmatic synthesis

Human Factors in Leveraging Systems Science to Shape Public Policy for Obesity: A Usability Study

Background: despite a broad consensus on their importance, applications of systems thinking in policymaking and practice have been limited. This is partly caused by the longstanding practice of developing systems maps and software in the intention of supporting policymakers, but without knowing their needs and practices. Objective: we aim to ensure the effective use of a systems mapping software by policymakers seeking to understand and manage the complex system around obesity, physical, and mental well-being. Methods: we performed a usability study with eight policymakers in British Columbia based on a software tool (ActionableSystems) that supports interactions with a map of obesity. Our tasks examine different aspects of systems thinking (e.g., unintended consequences, loops) at several levels of mastery and cover common policymaking needs (identification, evaluation, understanding). Video recordings provided quantitative usability metrics (correctness, time to completion) individually and for the group, while pre- and post-usability interviews yielded qualitative data for thematic analysis. Results: users knew the many different factors that contribute to mental and physical well-being in obesity; however, most were only familiar with lower-level systems thinking concepts (e.g., interconnectedness) rather than higher-level ones (e.g., feedback loops). Most struggles happened at the lowest level of the mastery taxonomy, and predominantly on network representation. Although participants completed tasks on loops and multiple pathways mostly correctly, this was at the detriment of spending significant time on these aspects. Results did not depend on the participant, as their experiences with the software were similar. The thematic analysis revealed that policymakers did not have a typical workflow and did not use any special software or tools in their policy work; hence, the integration of a new tool would heavily depend on individual practices. Conclusions: there is an important discrepancy between what constitutes systems thinking to policymakers and what parts of systems thinking are supported by software. Tools may be more successfully integrated when they include tutorials (e.g., case studies), facilitate access to evidence, and can be linked to a policymaker’s portfolio.

Utilizing Causal Loop Diagramming to Explore a Research and Evaluation Capacity Building Partnership

The capacity to engage in research, evaluation and evidence-informed decision-making supports effective public health policy and practice. Little is known about partnership-based approaches that aim to build capacity across a system or how to evaluate them. This study examines the impacts of a research and evaluation capacity building partnership called the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (hereafter, SiREN). SiREN aims to strengthen capacity across a system of clinical and medical services and government and non-government organizations. These organizations are connected through their shared aim of preventing and managing sexually transmissible infections and blood-borne viruses. To examine SiREN, systems concepts and methods were used. Data were collected from SiREN organizational documents (n = 42), a survey tool (n = 104), in-depth interviews (n = 17), a workshop and three meetings with SiREN stakeholders and used to develop two causal loop diagrams. Findings show engagement with SiREN was influenced by a complex interplay of contextual (e.g., organizational capacity) and process (e.g., presence of trusting relationships) factors. SiREN contributed to system level changes, including increased resources for research and evaluation, the development of networks and partnerships that led to more efficient responses to emerging health issues, evidence sharing, and sustainable research and evaluation practice. The use of causal loop diagrams enabled the identification of key leverage points that SiREN can use for continuous improvement or evaluation. The focus on how contextual factors influenced SiREN's ability to create change provides valuable information for researchers, policymakers or practitioners seeking to develop a similar partnership.

Critical factors that affect the functioning of a research and evaluation capacity building partnership: A causal loop diagram

Introduction

Public health policy and practice is strengthened by the application of quality evidence to decision making. However, there is limited understanding of how initiatives that support the generation and use of evidence in public health are operationalised. This study examines factors that support the internal functioning of a partnership, the Western Australian Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN). SiREN aims to build research and evaluation capacity and increase evidence-informed decision making in a public health context.

Methods

This study was informed by systems concepts. It developed a causal loop diagram, a type of qualitative system model that illustrated the factors that influence the internal operation of SiREN. The causal loop diagram was developed through an iterative and participatory process with SiREN staff and management (n = 9) via in-depth semi-structured interviews (n = 4), workshops (n = 2), and meetings (n = 6).

Results

Findings identified critical factors that affected the functioning of SiREN. Central to SiREN’s ability to meet its aims was its capacity to adapt within a dynamic system. Adaptation was facilitated by the flow of knowledge between SiREN and system stakeholders and the expertise of the team. SiREN demonstrated credibility and capability, supporting development of new, and strengthening existing, partnerships. This improved SiREN’s ability to be awarded new funding and enhanced its sustainability and growth. SiREN actively balanced divergent stakeholder interests to increase sustainability.

Conclusion

The collaborative development of the diagram facilitated a shared understanding of SiREN. Adaptability was central to SiREN achieving its aims. Monitoring the ability of public health programs to adapt to the needs of the systems in which they work is important to evaluate effectiveness. The detailed analysis of the structure of SiREN and how this affects its operation provide practical insights for those interested in establishing a similar project.

Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia

Complex interventions, such as innovation platforms, pose challenges for evaluators. A variety of methodological approaches are often required to build a more complete and comprehensive understanding of how complex interventions work. In this paper, we outline and critically appraise a methodologically pluralist evaluation of an innovation platform to strengthen primary care for Aboriginal and Torres Strait Islander Australians. In doing so, we aim to identify lessons learned from the approach taken and add to existing literature on implementing evaluations in complex settings, such as innovation platforms. The pluralist design used four evaluation approaches-developmental evaluation, principles-focused evaluation, network analysis, and framework analysis-with differing strengths and challenges. Taken together, the multiple evaluation approaches yielded a detailed description and nuanced understanding of the formation, functioning and outcomes of the innovation platform that would be difficult to achieve with any single evaluation method. While a methodologically pluralist design may place additional pressure on logistical and analytic resources available, it enables a deeper understanding of the mechanisms that underlie complex interventions.

Opioid risk screening: Program evaluation from the community pharmacists' perspective

BACKGROUND

Pharmacist-implemented screening programs can be improved through continuous program evaluation. Pharmacists are in a position to determine whether interventions are realistic and efficacious when used in practice.

OBJECTIVE

The purpose of this study is to evaluate how community pharmacists perceive the use of an opioid risk screening for patients receiving opioid prescriptions and the associated implications for improved patient-centered care.

METHODS

North Dakota community pharmacists received training on the use of an opioid risk tool for all patients filling an opioid prescription to evaluate for opioid misuse and overdose risk potential. Pharmacists then implemented the screening in their community pharmacy to screen all patients prescribed an opioid. Six months after implementation, pharmacists across the state were surveyed regarding their perception of the value of screening patients for the risk of opioid misuse and overdose. The survey questions used the Joint Committee on Standards for Educational Evaluation focusing on utility, propriety, feasibility, and accuracy.

RESULTS

All pharmacists (n = 35) indicated the opioid risk screening improved patient communication and patient-centered interventions. A total of 97% of pharmacists agreed the opioid screening tool provided an objective measure in providing care to patients and improved the potential for patient safety during prescription opioid use. Although 66% of pharmacists disagreed that the screening process was time consuming, 14% of respondents agreed with this statement indicating they may require additional assistance to optimize their workflow.

CONCLUSION

The results of this study support that opioid risk screening ensures utility for opioid risk stratification, feasibility to incorporate into existing workflow, and propriety for patient safety and well-being.

Five ways to get a grip on the shortcomings of logic models in program evaluation

Logic models are perhaps the most widely used tools in program evaluation work. They provide reasonably straightforward, visual illustrations of plausible links between program activities and outcomes. Consequently, they are employed frequently in stakeholder engagement, communication, and evaluation project planning. However, their relative simplicity comes with multiple drawbacks that can compromise the integrity of evaluation studies. In this Black Ice article, we outline key considerations and provide practical strategies that can help those engaged in evaluation work to identify and mitigate some limitations of logic models.

A guide to systems-level, participatory, theory-informed implementation research in global health

Implementation research is a multidisciplinary field that addresses the complex phenomenon of how context influences our ability to deliver evidence-informed healthcare. There is increasing realisation of the importance of applying robust implementation research to scale-up life-saving interventions that meet health-related sustainable development goals. However, the lack of high-quality implementation research is impeding our ability to meet these targets, globally. Within implementation research, theory refers to the proposed hypothesis and/or explanation of how an intervention is expected to interact with the local context and actors to bring about change. Although there is increasing interest in applying theory to understand how and why implementation programmes work in real-world settings, global health actors still tend to favour impact evaluations conducted in controlled environments. This may, in part, be due to the relative novelty as well as methodological complexity of implementation research and the need to draw on divergent disciplines, including epidemiology, implementation science and social sciences. Because of this, implementation research is faced with a particular set of challenges about how to reconcile different ways of thinking and constructing knowledge about healthcare interventions. To help translate some of the ambiguity surrounding how divergent theoretical approaches and methods contribute to implementation research, we draw on our multidisciplinary expertise in the field, particularly in global health. We offer an overview of the different theoretical approaches and describe how they are applied to continuously select, monitor and evaluate implementation strategies throughout the different phases of implementation research. In doing so, we offer a relatively brief, user-focused guide to help global health actors implement and report on evaluation of evidence-based and scalable interventions, programmes and practices.

Case Study With a Participatory Approach: Rethinking Pragmatics of Stakeholder Engagement for Implementation Research

The case study design is particularly useful for implementation analysis of complex health care innovations in primary care that can be influenced by the context of dynamic environments. Case studies may be combined with participatory approaches where academics conduct joint research with nonacademic stakeholders, to foster translation of findings results into practice. The aim of this article is to clarify epistemological and methodological considerations of case studies with a participatory approach. It also aims to propose best practice recommendations when using this case study approach. We distinguish between the participatory case study with full co-construction and co-governance, and the case study with a participatory approach whereby stakeholders are consulted in certain phases of the research. We then compare the epistemological posture of 3 prominent case study methodologists, Yin, Stake, and Merriam, to present the epistemological posture of case studies with a participatory approach. The relevance, applications, and procedures of a case study with a participatory approach methodology are illustrated through a concrete example of a primary care research program (PriCARE). We propose 12 steps for designing and conducting a case study with a participatory approach that may help guide researchers in the implementation analysis of complex health care innovations in primary care.

Developing a programme theory for a transdisciplinary research collaboration: Complex Urban Systems for Sustainability and Health

Background: Environmental improvement is a priority for urban sustainability and health and achieving it requires transformative change in cities. An approach to achieving such change is to bring together researchers, decision-makers, and public groups in the creation of research and use of scientific evidence. Methods: This article describes the development of a programme theory for Complex Urban Systems for Sustainability and Health (CUSSH), a four-year Wellcome-funded research collaboration which aims to improve capacity to guide transformational health and environmental changes in cities. Results: Drawing on ideas about complex systems, programme evaluation, and transdisciplinary learning, we describe how the programme is understood to "work" in terms of its anticipated processes and resulting changes. The programme theory describes a chain of outputs that ultimately leads to improvement in city sustainability and health (described in an 'action model'), and the kinds of changes that we expect CUSSH should lead to in people, processes, policies, practices, and research (described in a 'change model'). Conclusions: Our paper adds to a growing body of research on the process of developing a comprehensive understanding of a transdisciplinary, multiagency, multi-context programme. The programme theory was developed collaboratively over two years. It involved a participatory process to ensure that a broad range of perspectives were included, to contribute to shared understanding across a multidisciplinary team. Examining our approach allowed an appreciation of the benefits and challenges of developing a programme theory for a complex, transdisciplinary research collaboration. Benefits included the development of teamworking and shared understanding and the use of programme theory in guiding evaluation. Challenges included changing membership within a large group, reaching agreement on what the theory would be 'about', and the inherent unpredictability of complex initiatives.

Developing a programme theory for a transdisciplinary research collaboration: Complex Urban Systems for Sustainability and Health

Background: Environmental improvement is a priority for urban sustainability and health and achieving it requires transformative change in cities. An approach to achieving such change is to bring together researchers, decision-makers, and public groups in the creation of research and use of scientific evidence.

Methods: This article describes the development of a programme theory for Complex Urban Systems for Sustainability and Health (CUSSH), a four-year Wellcome-funded research collaboration which aims to improve capacity to guide transformational health and environmental changes in cities.

Results: Drawing on ideas about complex systems, programme evaluation, and transdisciplinary learning, we describe how the programme is understood to “work” in terms of its anticipated processes and resulting changes. The programme theory describes a chain of outputs that ultimately leads to improvement in city sustainability and health (described in an ‘action model’), and the kinds of changes that we expect CUSSH should lead to in people, processes, policies, practices, and research (described in a ‘change model’).

Conclusions: Our paper adds to a growing body of research on the process of developing a comprehensive understanding of a transdisciplinary, multiagency, multi-context programme. The programme theory was developed collaboratively over two years. It involved a participatory process to ensure that a broad range of perspectives were included, to contribute to shared understanding across a multidisciplinary team. Examining our approach allowed an appreciation of the benefits and challenges of developing a programme theory for a complex, transdisciplinary research collaboration. Benefits included the development of teamworking and shared understanding and the use of programme theory in guiding evaluation. Challenges included changing membership within a large group, reaching agreement on what the theory would be ‘about’, and the inherent unpredictability of complex initiatives.

Using implementation science theories and frameworks in global health

In global health, researchers and decision makers, many of whom have medical, epidemiology or biostatistics background, are increasingly interested in evaluating the implementation of health interventions. Implementation science, particularly for the study of public policies, has existed since at least the 1930s. This science makes compelling use of explicit theories and analytic frameworks that ensure research quality and rigour. Our objective is to inform researchers and decision makers who are not familiar with this research branch about these theories and analytic frameworks. We define four models of causation used in implementation science: intervention theory, frameworks, middle-range theory and grand theory. We then explain how scientists apply these models for three main implementation studies: fidelity assessment, process evaluation and complex evaluation. For each study, we provide concrete examples from research in Cuba and Africa to better understand the implementation of health interventions in global health context. Global health researchers and decision makers with a quantitative background will not become implementation scientists after reading this article. However, we believe they will be more aware of the need for rigorous implementation evaluations of global health interventions, alongside impact evaluations, and in collaboration with social scientists.

Taking a systems approach to explore the impacts and outcomes of a research and evaluation capacity building partnership: a protocol

INTRODUCTION

Partnership models that bring researchers, policymakers and service providers closer together are gaining traction as a strategy to improve public health practice. Yet, there is little evidence of how these models work, or indeed if they do work. The Sexual Health and Blood-borne Virus Applied Research and Evaluation Network (SiREN) is one such model. SiREN is a partnership between researchers, policymakers and service providers that aims to develop the research and evaluation capacity and evidence-informed decision making capability of professionals working to address sexual health and bloodborne virus issues in Western Australia. This study will use a systems approach to identify the mechanisms of action, impacts and outcomes of SiREN and inform the development of evaluation tools.

METHODS AND ANALYSIS

Data will be collected from organisational documents, surveys, in-depth interviews and a workshop. It will be analysed using a complex adaptive systems lens and findings will be used to inform the development of a type of qualitative systems model called a causal loop diagram. The causal loop diagram will illustrate the: contextual factors influencing engagement; mechanisms of action; and impacts and outcomes of SiREN. Evaluation tools will then be developed that can be used to assess the indicators identified in the causal loop diagram.

ETHICS AND DISSEMINATION

Ethics approval was obtained from the Curtin University Human Research Ethics Committee (approval number: HRE2017-0090). Participants will be free to withdraw from the study at any point and confidentiality will be maintained by de-identifying participant responses in any published or shared data. The findings from this study will be shared in conference presentations, reports, peer-reviewed journals and online through websites and social media.

Teacher Physical Education Practices and Student Outcomes in a Sample of Middle Schools Participating in the Presidential Youth Fitness Program

Obesity and lack of physical activity among children and adolescents are public health problems in the United States. This Presidential Youth Fitness Program (PYFP) evaluation measured program implementation in 13 middle schools and its effect on physical education practices, student fitness knowledge, and student physical activity and fitness levels. PYFP, a free program with the potential to positively affect student health and fitness outcomes, was designed to improve fitness education practices that are easily integrated into existing physical education programs. We used a 2-group (13 PYFP and 13 comparison schools) quasi-experimental design to collect FitnessGram assessments, accelerometry data, and surveys of students, physical education teachers, and administrators. Although the program was positively associated with student cardiovascular endurance and physical activity gains during the semester, schools underused professional development courses and fitness recognition resources.

Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

ISSUE ADDRESSED

Complex health promotion programs, which can have multilevels of implementation and multi-components with nonlinear causal pathways, present many evaluation challenges. Traditional evaluation methods often fail to account for the complexity inherent in assessing these programs. In real-world settings, evaluations of complex programs are often beset by additional constraints of limited budgets and short timeframes. Determining whether a complex program is successful and how a program worked requires evaluators of complex programs to adopt a level of pragmatism.

METHODS

This paper describes a pragmatic evaluation approach used to evaluate the Get Healthy at Work workplace health promotion program, implemented in New South Wales, Australia. Using the program as a case study, we describe some key principles for applying a pragmatic evaluation approach and use these principles to develop an appropriate evaluation strategy.

RESULTS

The evaluation includes multiple research methods to assess program outputs and implementation; and identify emergent program impacts, within constrained resources. The evaluation was guided by epistemological flexibility, methodological comprehensiveness and operational practicality.

CONCLUSION

Health promotion programs, such as state-wide obesity prevention programs, require appropriate evaluation methods which address their inherent complexity amidst the real-world evaluation constraints, and focuses on the essential evaluation needs.

SO WHAT

The main complex program evaluation principles are applicable to other multilevel health promotion programs, challenged by methodological and practical or political constraints.

Evaluation of Get Healthy at Work, a state-wide workplace health promotion program in Australia

BACKGROUND

Workplace health programs (WHPs) may improve adult health but very little evidence exists on multi-level WHPs implemented at-scale and so the relationship between program implementation factors and outcomes of WHPs are poorly understood. This study evaluated Get Healthy at Work (GHaW), a state-wide government-funded WHP in Australia.

METHODS

A mixed-method design included a longitudinal quasi-experimental survey of businesses registered with GHaW and a comparison group of businesses surveyed over a 12-month period. Semi-structured interviews and focus groups with key contacts and employees of selected intervention group businesses and the service providers of the program were conducted to assess program adoption and adaptation.

RESULTS

Positive business-level changes in workplace culture were observed over time among GHaW businesses compared with the control group. Multilevel regression modelling revealed perceptions that employees were generally healthy (p = 0.045 timeXgroup effect) and that the workplace promoted healthy behaviours (p = 0.004 timeXgroup effect) improved significantly while the control group reported no change in work culture perceptions. Changes in perceptions about work productivity were not observed; however only one third of businesses registered for the program had adopted GHaW during the evaluation period. Qualitative results revealed a number of factors contributing to program adoption: which depended on program delivery (e.g., logistics, technology and communication channels), design features of the program, and organisational factors (primarily business size and previous experience of WHPs).

CONCLUSIONS

Evaluation of program factors is important to improve program delivery and uptake and to ensure greater scalability. GHaW has the potential to improve workplace health culture, which may lead to better health promoting work environments. These results imply that government can play a central role in enabling prioritisation and incentivising health promotion in the workplace.

Early evaluation of experiences of health care providers in reception centers with a patient-held personal health record for asylum seekers: a multi-sited qualitative study in a German federal state

BACKGROUND

The provision of high-quality medical care to asylum seekers represents a key challenge in many countries of the European Union. Especially continuity of care has been difficult to achieve as the migrant trajectory moves asylum seekers across and within European countries. Patient-held personal health records (PHR) have been proposed to facilitate the transfer of medical history between health sectors and providers, but so far there is no data to support its use in the migrant setting. The present paper addresses this knowledge gap by exploring the experiences and practices of healthcare providers in reception centers for asylum seekers using a patient-held PHR as well as the perceived associated benefits and shortcomings.

METHODS

Early evaluation by means of a multi-sited qualitative study in six asylum seeker reception centers in five cities in the German state of Baden-Wuerttemberg, conducted between November 2016 and January 2017. The PHR evaluated in this study was implemented in five of these reception centers between February and October 2016; the remaining one only receiving patients with the PHR through transfer from the other facilities. 17 interviews were conducted with physicians and nurses working at these reception centers exploring their experiences, routines, and perspectives regarding the patient-held PHR. The interviews were recorded, transcribed and analyzed following the approach of thematic analysis.

RESULTS

Healthcare providers recognise the potential of a patient-held PHR to improve access to medical history. They use the PHR to document their medical consultations and to collect other medical reports. However, physician adherence to the patient-held PHR was described as unsatisfactory, in particular among external doctors, thus limiting its immediate benefit. Reasons given for this low adherence included lack of information before implementation, demanding working conditions with little support, low perceived benefits depending on the degree of fragmentation of settings, parallel existence of other documentation platforms and strained patient relationships.

CONCLUSION

A patient-held PHR could improve the availability of health-related information in reception centers if a context-sensitive implementation process achieves high adherence to the PHR among physicians as well as high patient compliance and includes guidelines regarding its adequate integration into local routines.

The need for theory evaluation in global citizenship programmes: The case of the GCSA programme

Many education programmes lack a documented programme theory. This is a problem for programme planners and evaluators as the ability to measure programme success is grounded in the plausibility of the programme's underlying causal logic. Where the programme theory has not been documented, conducting a theory evaluation offers a foundational evaluation step as it gives an indication of whether the theory behind a programme is sound. This paper presents a case of a theory evaluation of a Global Citizenship programme at a top-ranking university in South Africa, subsequently called the GCSA Programme. This evaluation highlights the need for documented programme theory in global citizenship-type programmes for future programme development. An articulated programme theory produced for the GCSA Programme, analysed against the available social science literature, indicated it is comparable to other such programmes in terms of its overarching framework. What the research found is that most other global citizenship programmes do not have an articulated programme theory. These programmes also do not explicitly link their specific activities to their intended outcomes, making demonstrating impact impossible. In conclusion, we argue that taking a theory-based approach can strengthen and enable outcome evaluations in global citizenship programmes.