Why so many "rigorous" evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Examining the Complex Interaction Among Technological Innovation, Company Performance, and Occupational Safety and Health: A Mixed-Methods Study

Technological innovation and Industry 5.0 are gaining increasing attention among researchers as they offer companies a significant competitive advantage. On the other hand, introducing these technologies also brings new risks for workers. The current literature reveals a lack of studies that effectively integrate occupational safety and health (OSH) within this emerging technological context and analyse the impacts of their use. This study aims to explore how companies interact with macro-level interventions that promote technological innovation and to understand their impact on different dimensions of company performance, including aspects related to OSH. Based on the existing literature, a research framework is presented that identifies the stakeholders involved, the inputs facilitating their interaction, and the cascading effects and changes. A mixed-methods approach was adopted by employing an in-depth survey with 89 companies responding and composed of both open-ended questions, to capture rich, qualitative insights, and multiple-choice questions, to gather quantifiable data. Two change levels have been identified: general changes and specific changes related to OSH. The analysis also delved into the main drivers and barriers that lead companies to engage with technological improvements and the multiple changes these interventions generate across company dimensions.

Uncovering the key working mechanisms of a complex community-based obesity prevention programme in the Netherlands using ripple effects mapping

BACKGROUND

Much remains unknown about how complex community-based programmes can successfully achieve long-term impact. More insight is needed to understand the key mechanisms through which these programmes work. Therefore, we conducted an in-depth study in five communities that implemented the Healthy Youth, Healthy Future (JOGG) approach, a Dutch community-based obesity prevention programme. We aimed to identify perceived outcomes and long-term impacts among local stakeholders and explore potential causal pathways and working mechanisms.

METHODS

We used ripple effects mapping (REM), a qualitative participatory method to map outcomes and identify causal pathways, in five communities. We involved 26 stakeholders, professionals and policy-makers affiliated with the local JOGG approach, spread over eight REM sessions and conducted individual interviews with 24 additional stakeholders. To uncover working mechanisms, we compared outcomes and causal pathways across communities.

RESULTS

Over 5-9 years of implementation, participants perceived that JOGG had improved ownership of local stakeholders, health policies, intersectoral collaboration and social norms towards promoting healthy lifestyles. Causal pathways comprised small initial outcomes that created the preconditions to enable the achievement of long-term impact. Although exact JOGG actions varied widely between communities, we identified five common working mechanisms through which the JOGG approach contributed to causal pathways: (1) creating a positive connotation with JOGG, (2) mobilizing stakeholders to participate in the JOGG approach, (3) facilitating projects to promote knowledge and awareness among stakeholders while creating successful experiences with promoting healthy lifestyles, (4) connecting stakeholders, thereby stimulating intersectoral collaboration and (5) sharing stakeholder successes that promote healthy lifestyles, which gradually created a social norm of participation.

CONCLUSIONS

The JOGG approach seems to work through activating initial stakeholder participation and bolstering the process towards ownership, policy change, and intersectoral collaboration to promote healthy lifestyles. Key working mechanisms can inform further development of JOGG as well as other complex community-based prevention programmes.

A qualitative investigation of gender-based violence prevention and response using digital technologies in low resource settings and refugee populations

Background: Governmental and non-governmental organizations across medical, legal, and psychosocial sectors providing care to survivors of gender-based violence (GBV) and their families rapidly digitalized services during the COVID-19 pandemic. GBV prevention/response services working with women and children who are forcibly displaced and/or living in low-and-middle income countries (LMIC) were no exception to the rapid digitalization trend. Literature is lacking a critical synthesis of best practices and lessons learned since digitalization replaced major operations involved in GBV prevention/response.Objective: This research qualitatively investigated how GBV service providers, located in a range of socio-political settings, navigated the process of digitalizing GBV prevention/response during the COVID-19 crisis.Method: Semi-structured key informant interviews (KII) with GBV service providers in varied sectors were implemented virtually (2020-2021) in Brazil, Guatemala, Iraq, and Italy (regarding forcibly displaced women/girls for the latter). Participants were recruited using purposive and snowball sampling. Interview guides covered a range of topics: perceived changes in violence and service provision, experiences with virtual services, system coordination, and challenges. The KIIs were conducted in Portuguese, Spanish, Arabic, and Italian. Interviews were audio-recorded, transcribed, and translated into English. The research team conducted thematic analysis within and between countries using a structured codebook of data driven and theory driven codes.Results: Major themes concerned the: (1) spectrum of services that were digitalized during the COVID-19 crisis; (2) gender digital divide as a barrier to equitable, safe, and effective service digitalization; (3) digital violence as an unintended consequence of increased digitalization across social/public services.Conclusion: Digitalization is a balancing act with respect to (1) the variety of remotely-delivered services that are possible and (2) the access/safety considerations related to the gender digital divide and digital violence.

Exploring the use and impact of the Australian living guidelines for the clinical care of people with COVID-19: where to from here?

OBJECTIVES

The Australian National COVID-19 Clinical Evidence Taskforce has been developing, maintaining, and disseminating living guidelines and decision support tools (clinical flowcharts) for the care of people with suspected or confirmed COVID-19 since 2020. Living guidelines, a form of living evidence, are a relatively new approach; hence, more work is required to determine how to optimize their use to inform practice, policy, and decision-making and to explore implementation, uptake, and impact implications. An update of an earlier impact evaluation was conducted to understand sustained awareness and use of the guidelines; the factors that facilitate the widespread adoption of the guidelines and to explore the perceived strengths and opportunities for improvement of the guidelines.

STUDY DESIGN AND SETTING

A mixed-methods impact evaluation was conducted. Surveys collected both quantitative and qualitative data and were supplemented with qualitative interviews. Participants included Australian healthcare practitioners providing care to individuals with suspected or confirmed COVID-19 and people involved in policy-making. Data were collected on awareness, use, impact, strengths, and opportunities for improvement of the guidelines and flow charts.

RESULTS

A total of 148 participants completed the survey and 21 people were interviewed between January and March 2022. Awareness of the work of the Taskforce was high and more than 75% of participants reported that the guidelines were used within their workplace. Participants described the Taskforce website and guidelines as trustworthy, valuable, and reliable sources of up-to-date evidence-based information. The evaluation highlighted the varied ways the guidelines were being used across a range of settings and the diverse impacts they have from those at a clinical level to impacts at a policy level. Barriers to and enablers of impact and uptake of the guideline were explored.

CONCLUSION

This evaluation highlights the value of living guidelines during a pandemic when the evidence base is rapidly changing and expanding. It presents useful understanding of the ways clinicians and others use living evidence to inform their clinical practice and decision-making and the diverse impacts the guidelines are having around Australia.

Individual interviews versus focus groups for evaluations of international development programs: Systematic testing of method performance to elicit sensitive information in a justice study in Haiti

Focus group discussions (FGDs) and individual interviews (IIs) with community members are common methods used in evaluations of all kinds of projects, including those in international development. As resources are often limited, evaluators must carefully choose methods that yield the best information for their particular program. A concern with FGDs and IIs is how well they elicit information on potentially sensitive topics; very little is known about differences in disclosure by methodology in the domain of justice. Using FGDs (n = 16) and IIs (n = 46) from a USAID project in Haiti, we systematically coded responses based on a shared elicitation guide around access to and engagement with the formal and informal justice systems and performed thematic and statistical comparisons across the two methods. We introduce the continuous thought as the novel standard unit for statistical comparison. Participants in IIs were statistically more likely to provide themes relevant to genderbased violence. Importantly, sensitive themes extracted in IIs (e.g., related to sexual violence, economic dimensions, and restorative justice) did not emerge in FGDs. Given these results and other limitations to the FGD, prioritizing interviews over focus group modalities may be appropriate to guide targeted, effective programming on justice or other socially sensitive topics.

Aid and forgetting the enemy: A systematic review of the unintended consequences of international development in fragile and conflict-affected situations

Fragile and conflict-affected situations (FCAS) are a significant focus area of international development efforts, with corresponding sizable investments by global donors. When implementing development assistance programmes in this context, the risk of unintended consequences tends to be higher as the conflict tends to increase complexity, and typically, FCAS development programmes are designed to deliver benefits only to parties on one side of the conflict. Searching for new insights, a systematic review of 121 documents was used to identify specific types of unintended consequences, their frequency and the nature of management strategies used to address them. Examination of these documents identified aggravation of the conflict and unintended support either for the opposing side's military regime or a non-state actor was the most common unintended consequence. As nearly all assessments only considered the nature of consequences from the donor's viewpoint, we conclude that there may be a case for assessing impacts from the alternative viewpoint - that is, of unsupported parties to the conflict. More generally, we conclude that FCAS programme guidelines would benefit from the development and deployment of more rigorous evaluation methods and the codifying of unintended consequences terminology.

Five ways to get a grip on the shortcomings of logic models in program evaluation

Logic models are perhaps the most widely used tools in program evaluation work. They provide reasonably straightforward, visual illustrations of plausible links between program activities and outcomes. Consequently, they are employed frequently in stakeholder engagement, communication, and evaluation project planning. However, their relative simplicity comes with multiple drawbacks that can compromise the integrity of evaluation studies. In this Black Ice article, we outline key considerations and provide practical strategies that can help those engaged in evaluation work to identify and mitigate some limitations of logic models.

Ripple effects of research capacity strengthening: a study of the effects of a project to support test facilities in three African countries towards Good Laboratory Practice certification

Background: Strengthening capacity for public health research is essential to the generation of high-quality, reliable scientific data. This study focuses on a research capacity strengthening project supporting seven test facilities in Africa conducting studies on mosquito vector control products towards Good Laboratory Practice (GLP) certification. It captures the primary effects of the project on each facility's research capacity, the secondary effects at the individual and institutional level, and the ripple effects that extend beyond the research system. The relationships between effects at different levels are identified and compared to an existing framework for the evaluation of research capacity strengthening initiatives. Methods: To capture the views of individuals engaged in the project at all levels within each facility, a maximum-variation purposive sampling strategy was used. This allowed triangulation between different data sources. Semi-structured interviews were conducted with individuals in three facilities and a combination of email and remote video-call interviews were conducted with individuals at two further facilities. Results: We found that, despite a focus of the GLP certification project at the institutional level, the project had effects also at individual (including enhanced motivation, furtherment of careers) and national/international levels (including development of regional expertise). In addition, we detected ripple effects of the project which extended beyond the research system. Conclusion: This study shows that research capacity strengthening interventions that are focussed on institutional level goals require actions also at individual and national/international levels. The effects of engagement at all three levels can be amplified by collaborative actions at the national/international level. These findings show that research capacity strengthening projects must develop plans that address and evaluate impact at all three levels. Capturing the ripple effects of investment in research capacity strengthening should also be planned for from the beginning of projects to support further engagement of all stakeholders.

The unintended consequences of COVID-19 mitigation measures matter: practical guidance for investigating them

BACKGROUND

COVID-19 has led to the adoption of unprecedented mitigation measures which could trigger many unintended consequences. These unintended consequences can be far-reaching and just as important as the intended ones. The World Health Organization identified the assessment of unintended consequences of COVID-19 mitigation measures as a top priority. Thus far, however, their systematic assessment has been neglected due to the inattention of researchers as well as the lack of training and practical tools.

MAIN TEXT

Over six years our team has gained extensive experience conducting research on the unintended consequences of complex health interventions. Through a reflexive process, we developed insights that can be useful for researchers in this area. Our analysis is based on key literature and lessons learned reflexively in conducting multi-site and multi-method studies on unintended consequences. Here we present practical guidance for researchers wishing to assess the unintended consequences of COVID-19 mitigation measures. To ensure resource allocation, protocols should include research questions regarding unintended consequences at the outset. Social science theories and frameworks are available to help assess unintended consequences. To determine which changes are unintended, researchers must first understand the intervention theory. To facilitate data collection, researchers can begin by forecasting potential unintended consequences through literature reviews and discussions with stakeholders. Including desirable and neutral unintended consequences in the scope of study can help minimize the negative bias reported in the literature. Exploratory methods can be powerful tools to capture data on the unintended consequences that were unforeseen by researchers. We recommend researchers cast a wide net by inquiring about different aspects of the mitigation measures. Some unintended consequences may only be observable in subsequent years, so longitudinal approaches may be useful. An equity lens is necessary to assess how mitigation measures may unintentionally increase disparities. Finally, stakeholders can help validate the classification of consequences as intended or unintended.

CONCLUSION

Studying the unintended consequences of COVID-19 mitigation measures is not only possible but also necessary to assess their overall value. The practical guidance presented will help program planners and evaluators gain a more comprehensive understanding of unintended consequences to refine mitigation measures.

Ripple effects of research capacity strengthening: a study of the effects of a project to support test facilities in three African countries towards Good Laboratory Practice certification

Background: Strengthening capacity for public health research is essential to the generation of high-quality, reliable scientific data. This study focuses on a research capacity strengthening project supporting seven test facilities in Africa conducting studies on mosquito vector control products towards Good Laboratory Practice (GLP) certification. It captures the primary effects of the project on each facility's research capacity, the secondary effects at the individual and institutional level, and the ripple effects that extend beyond the research system. The relationships between effects at different levels are identified and compared to an existing framework for the evaluation of research capacity strengthening initiatives. Methods: To capture the views of individuals engaged in the project at all levels within each facility, a maximum-variation purposive sampling strategy was used. This allowed triangulation between different data sources. Semi-structured interviews were conducted with individuals in three facilities and a combination of email and remote video-call interviews were conducted with individuals at two further facilities. Results: We found that, despite a focus of the GLP certification project at the institutional level, the project had effects also at individual (including enhanced motivation, furtherment of careers) and national/international levels (including development of regional expertise). In addition, we detected ripple effects of the project which extended beyond the research system. Conclusion: This study shows that research capacity strengthening interventions that are focussed on institutional level goals require actions also at individual and national/international levels. The effects of engagement at all three levels can be amplified by collaborative actions at the national/international level. These findings show that research capacity strengthening projects must develop plans that address and evaluate impact at all three levels. Capturing the ripple effects of investment in research capacity strengthening should also be planned for from the beginning of projects to support further engagement of all stakeholders.

Designing Rural Policies for Sustainable Innovations through a Participatory Approach

The added value coming from involvement of stakeholders in changing attitudes and cultures towards a more sustainable-oriented society has been repeatedly emphasized in documents of the European Union. Those documents emphasize the advisability of creating a more inclusive system from the early planning stages and for the whole process with regard to the development of EU policies, involving the stakeholder as a referring partnership both at the national and regional levels. This paper focuses on a case study related to an Italian region where the local partnership has been involved during the setting up of the Rural Development Program 2014–2020 and where a participatory approach has been applied. In order to create an effective output coming from these open consultations, a participatory approach has been carried out using a dynamized Strengths and Weaknesses, Opportunities and Threats analysis (SWOT Analysis). The results presented here come from a specific thematic table where experts and stakeholders discussed a specific priority focused on innovation, training and advisory in rural areas. As expected, sustainable innovations and services as well as training courses need to be improved, while financial resources addressing those issues need to be increased.

How to design, implement and evaluate organizational interventions for maximum impact: the Sigtuna Principles

Research on organizational interventions needs to meet the objectives of both researchers and participating organizations. This duality means that real-world impact has to be considered throughout the research process, simultaneously addressing both scientific rigour and practical relevance. This discussion paper aims to offer a set of principles, grounded in knowledge from various disciplines that can guide researchers in designing, implementing, and evaluating organizational interventions. Inspired by Mode 2 knowledge production, the principles were developed through a transdisciplinary, participatory and iterative process where practitioners and academics were invited to develop, refine and validate the principles. The process resulted in 10 principles: 1) Ensure active engagement and participation among key stakeholders; 2) Understand the situation (starting points and objectives); 3) Align the intervention with existing organizational objectives; 4) Explicate the program logic; 5) Prioritize intervention activities based on effort-gain balance; 6) Work with existing practices, processes, and mindsets; 7) Iteratively observe, reflect, and adapt; 8) Develop organizational learning capabilities; 9) Evaluate the interaction between intervention, process, and context; and 10) Transfer knowledge beyond the specific organization. The principles suggest how the design, implementation, and evaluation of organizational interventions can be researched in a way that maximizes both practical and scientific impact.

What can we learn about unintended consequences from a textual analysis of monitoring reports and evaluations for U.S. foreign assistance programs?

In recent years, articles in Evaluation and Program Planning have noted the importance of evaluating programs' unintended consequences, and the need to increase our knowledge in that area. To that end, this paper considers the information that can be obtained about the unintended consequences of foreign assistance programs through an automated textual analysis and review of publicly-available monitoring reports and evaluations. Automated full text searches for terms synonymous with 'unintended consequences' were conducted of more than 1,300 monitoring reports and evaluations downloaded from a publicly-available database of foreign assistance programs. The reports identified by the automated searches were screened and analyzed to determine which had considered and/or reported about such consequences. Positive and negative consequences were identified, as were the assistance sectors and recipient countries. While this study makes available more information on the unintended consequences of foreign assistance programs, it also emphasizes the need for greater research in this area, and outlines how a future research project of this nature might obtain more data.

Evaluating unintended program outcomes through Ripple Effects Mapping (REM): Application of REM using grounded theory

Several evaluation models exist for investigating unintended outcomes, including goal-free and systems evaluation. Yet methods for collecting and analyzing data on unintended outcomes remain under-utilized. Ripple Effects Mapping (REM) is a promising qualitative evaluation method with a wide range of program planning and evaluation applications. In situations where program results are likely to occur over time within complex settings, this method is useful for uncovering both intended and unintended outcomes. REM applies an Appreciative Inquiry facilitation technique to engage stakeholders in visually mapping sequences of program outcomes. Although it has been used to evaluate community development and health promotion initiatives, further methodological guidance for applying REM is still needed. The purpose of this paper is to contribute to the methodological development of evaluating unintended outcomes and extend the foundations of REM by describing steps for integrating it with grounded theory.

Using mixed method approach in measuring effects of training in firms: Case study of the European Social Fund support

Public support of training in firms corresponds to the long-term importance of the quality of human capital in the competitiveness of firms and nations. Thus, the EU supports such training via the European Social Fund (ESF). The evaluation community evaluates the support by using either qualitative or quantitative methods. The simultaneous application of these two approaches is rare. The purpose of this paper is to combine quantitative (counterfactual impact evaluation) and qualitative (qualitative comparative analysis) methods in order to fill the methodological gap. Based on the combination of both approaches, it explores their strengths, complementarity and disadvantages to evaluate public support for employee training in the Czech Republic. The combination of methods makes it possible to identify not only the impacts but also their causes. Linking the ESF support to corporate competitiveness is crucial for demonstrating the effectiveness of public spending.

Introduction to the special issue 'unintended effects of international cooperation'

The 'Evaluation and Program Planning' journal has contributed to the launch of an academic discussion of unintended effects of international cooperation, notably by publishing in 2016 articles by Bamberger, Tarsilla, & Hesse-Biber and by Jabeen. This special issue aims to take up the academic challenges as laid down by those authors, by providing among others a clear typology and applying it, by outlining various methodological options and testing them, and elaborating on suggestions on how to deal with the barriers that prevent unintended effects being taken into account. This special issue makes clear that it is possible to reduce the share of unforeseen effects of international cooperation. Turning the spotlight on unintended effects that can be anticipated, and aiming to make progress on uncovering those that are particularly difficult to detect and debunking those that are exaggerated is the task that lies ahead of us.

Systematic iteration between model and methodology: A proposed approach to evaluating unintended consequences

This article argues that evaluators could better deal with unintended consequences if they improved their methods of systematically and methodically combining empirical data collection and model building over the life cycle of an evaluation. This process would be helpful because it can increase the timespan from when the need for a change in methodology is first suspected to the time when the new element of the methodology is operational. The article begins with an explanation of why logic models are so important in evaluation, and why the utility of models is limited if they are not continually revised based on empirical evaluation data. It sets the argument within the larger context of the value and limitations of models in the scientific enterprise. Following will be a discussion of various issues that are relevant to model development and revision. What is the relevance of complex system behavior for understanding predictable and unpredictable unintended consequences, and the methods needed to deal with them? How might understanding of unintended consequences be improved with an appreciation of generic patterns of change that are independent of any particular program or change effort? What are the social and organizational dynamics that make it rational and adaptive to design programs around single-outcome solutions to multi-dimensional problems? How does cognitive bias affect our ability to identify likely program outcomes? Why is it hard to discern change as a result of programs being embedded in multi-component, continually fluctuating, settings? The last part of the paper outlines a process for actualizing systematic iteration between model and methodology, and concludes with a set of research questions that speak to how the model/data process can be made efficient and effective.

Twinning "Practices of Change" With "Theory of Change": Room for Emergence in Advocacy Evaluation

Theory of change (ToC) is currently the approach for the evaluation and planning of international development programs. This approach is considered especially suitable for complex interventions. We question this assumption and argue that ToC's focus on cause-effect logic and intended outcomes does not do justice to the recursive nature of complex interventions such as advocacy. Supported by our work as evaluators, and specifically our case study of an advocacy program on child rights, we illustrate how advocacy evolves through recursive interactions, with outcomes that are emergent rather than predictable. We propose putting "practices of change" at the center by emphasizing human interactions, using the analytical lenses of strategies as practice and recursiveness. This provides room for emergent outcomes and implies a different use of ToC. In this article, we make a clear distinction between theoretical reality models and the real world of practices.

Population cardiovascular health and urban environments: the Heart Healthy Hoods exploratory study in Madrid, Spain

BACKGROUND

Our aim is to conduct an exploratory study to provide an in-depth characterization of a neighborhood's social and physical environment in relation to cardiovascular health. A mixed-methods approach was used to better understand the food, alcohol, tobacco and physical activity domains of the urban environment.

METHODS

We conducted this study in an area of 16,000 residents in Madrid (Spain). We obtained cardiovascular health and risk factors data from all residents aged 45 and above using Electronic Health Records from the Madrid Primary Health Care System. We used several quantitative audit tools to assess: the type and location of food outlets and healthy food availability; tobacco and alcohol points of sale; walkability of all streets and use of parks and public spaces. We also conducted 11 qualitative interviews with key informants to help understanding the relationships between urban environment and cardiovascular behaviors. We integrated quantitative and qualitative data following a mixed-methods merging approach.

RESULTS

Electronic Health Records of the entire population of the area showed similar prevalence of risk factors compared to the rest of Madrid/Spain (prevalence of diabetes: 12 %, hypertension: 34 %, dyslipidemia: 32 %, smoking: 10 %, obesity: 20 %). The food environment was very dense, with many small stores (n = 44) and a large food market with 112 stalls. Residents highlighted the importance of these small stores for buying healthy foods. Alcohol and tobacco environments were also very dense (n = 91 and 64, respectively), dominated by bars and restaurants (n = 53) that also acted as food services. Neighbors emphasized the importance of drinking as a socialization mechanism. Public open spaces were mostly used by seniors that remarked the importance of accessibility to these spaces and the availability of destinations to walk to.

CONCLUSION

This experience allowed testing and refining measurement tools, drawn from epidemiology, geography, sociology and anthropology, to better understand the urban environment in relation to cardiovascular health.