Strengthening open disclosure after incidents in maternity care: a realist synthesis of international research evidence

BACKGROUND

Open Disclosure (OD) is open and timely communication about harmful events arising from health care with those affected. It is an entitlement of service-users and an aspect of their recovery, as well as an important dimension of service safety improvement. Recently, OD in maternity care in the English National Health Service has become a pressing public issue, with policymakers promoting multiple interventions to manage the financial and reputational costs of communication failures. There is limited research to understand how OD works and its effects in different contexts.

METHODS

Realist literature screening, data extraction, and retroductive theorisation involving two advisory stakeholder groups. Data relevant to families, clinicians, and services were mapped to theorise the relationships between contexts, mechanisms, and outcomes. From these maps, key aspects for successful OD were identified.

RESULTS

After realist quality appraisal, 38 documents were included in the synthesis (22 academic, 2 training guidance, and 14 policy report). 135 explanatory accounts were identified from the included documents (with n = 41 relevant to families; n = 37 relevant to staff; and n = 37 relevant to services). These were theorised as five key mechanism sets: (a) meaningful acknowledgement of harm, (b) opportunity for family involvement in reviews and investigations, (c) possibilities for families and staff to make sense of what happened, (d) specialist skills and psychological safety of clinicians, and (e) families and staff knowing that improvements are happening. Three key contextual factors were identified: (a) the configuration of the incident (how and when identified and classified as more or less severe); (b) national or state drivers, such as polices, regulations, and schemes, designed to promote OD; and (c) the organisational context within which these these drivers are recieived and negotiated.

CONCLUSIONS

This is the first review to theorise how OD works, for whom, in what circumstances, and why. We identify and examine from the secondary data the five key mechanisms for successful OD and the three contextual factors that influence this. The next study stage will use interview and ethnographic data to test, deepen, or overturn our five hypothesised programme theories to explain what is required to strengthen OD in maternity services.

How transdisciplinary research teams learn to do knowledge translation (KT), and how KT in turn impacts transdisciplinary research: a realist evaluation and longitudinal case study

BACKGROUND

Transdisciplinary research and knowledge translation are increasingly regarded as key concepts underpinning applied research across the health and social sciences, due to their presumed potential in addressing complex, "wicked" problems and improving the use of research in practice and policy, respectively. Despite sharing an impact mandate, the relationship between transdisciplinary research collaboration and knowledge translation remains unclear. In response, we examined the relationship between transdisciplinary collaboration and knowledge translation to generate these understandings with a view towards maximizing the impact of collaborative efforts.

METHODS

We undertook a realist evaluation and longitudinal case study of a 5-year National Health and Medical Research Council-funded Centre of Research Excellence in Transdisciplinary Frailty Research. Data were collected between February 2017 and March 2020 over three rounds of theory development, refinement and testing using interviews, observation, document review and visual elicitation as data sources. The Human Research Ethics Committee of the University of Adelaide approved this study.

RESULTS

Iterative analysis of narrative interviews and visual data led to the development of three overarching programme theories explicating the reciprocal relationship between KT understandings and transdisciplinary team process. These programme theories revolve around the concept of a network, which we define in alignment with extant theoretical literature on network mechanisms and complex networks as graphically representable networks of agents/people (nodes) joined by social relationships (links). Our findings demonstrate that under the right contextual conditions, transdisciplinary team members respond through an improved ability to (1) navigate the network, (2) negotiate the network and (3) mobilize the network.

CONCLUSIONS

This research demonstrates the reciprocity and mutually supportive relationship between transdisciplinary research and knowledge translation. Our findings suggest that embedding a collaborative knowledge translation framework and providing resources such as facilitation and distributed leadership within a transdisciplinary team can improve collaboration and support transdisciplinary research objectives.

'For Want of a Nail': developing a transparent approach to retroduction and early initial programme theory development in a realist evaluation of community end of life care services

A crucial part of theory-driven realist thinking is retroduction, the process of looking backwards for explanation of how and why things may be. Conducted early in the realist evaluation process, it provides a foundation for evidenced 'theory gleaning'. Despite retroduction being an inherent part of the realist process, it is often 'hidden' in realist reports. This paper explains the thinking behind, alongside an example of, a framework created by the authors to make transparent the retroductive process as used in a realist evaluation of two community End of Life Care services. The approach makes visible the application of the 'sociological imagination' and lends robustness to hypotheses by establishing how the authors utilised: wide-ranging potential generative causation; stakeholder and Patient and Public Involvement feedback; literature scoping; and substantive theories at the middle range, specifically Transitions Theory. These stages led to the development of Initial Programme Theories, with a clear history of genesis.

A realist evaluation of the feasibility of a randomised controlled trial of a digital music and movement intervention for older people living in care homes

BACKGROUND

Low physical activity in care home residents brings about negative mental health consequences, such as higher levels of depression and loneliness. With advancements in communication technology, particularly during the COVID-19 pandemic, the feasibility and effectiveness of a randomised controlled trial (RCT) of a digital Physical Activity (PA) resource in care homes deserve more research attention. A realist evaluation was used to uncover influencing factors of a feasibility study implementation to inform how a digital music and movement programme would work and under what circumstances this would be most effective.

METHODS

Participants were 49 older adults (aged 65 years +) recruited across ten care homes in Scotland. Surveys were administered at baseline and post-intervention comprising psychometric questionnaires on multidimensional health markers validated in older adults with possible cognitive impairment. The intervention comprised 12 weeks of four prescribed digitally delivered movement (n = 3) and music-only (n = 1) sessions per week. An activity coordinator delivered these online resources in the care home. Post-intervention focus groups with staff and interviews with a sub-sample of participants were conducted to gain qualitative data on the acceptability of the intervention.

RESULTS

Thirty three care home residents started the intervention, but only 18 residents (84% female) completed both pre- and post-intervention assessments. Activity coordinators (AC) offered 57% of the prescribed sessions, with an average residents' adherence of 60%. Delivery of the intervention did not go as planned due to Covid restrictions in care homes and delivery challenges, including (1) motivation and engagement, (2) changes in cognitive impairment and disabilities of the participants, (3) death or hospitalisation of the participants and (4) limited staffing and technology resources to deliver the programme as intended. Despite this, group participation and encouragement of residents supported the delivery and acceptance of the intervention, with ACs and residents reporting improved mood, physical health, job satisfaction and social support. Improvements with large effect sizes were found for anxiety, depression, loneliness, perceived stress and sleep satisfaction, but no changes in fear of falling, domains of general health or appetite.

CONCLUSION

This realist evaluation suggested that this digitally delivered movement and music intervention is feasible. From the findings, the initial programme theory was refined for future implementation of an RCT in other care homes but future research exploring how to tailor the intervention to those with cognitive impairment and/or lacking capacity to consent is needed.

TRIAL REGISTRATION

Retrospectively registered at ClinicalTrials.gov NCT05559203.

Theory-based evaluation and programme theories in nursing: A discussion on the occasion of the updated Medical Research Council (MRC) Framework

Developing and evaluating health interventions for the benefit of patients is notoriously difficult. This also applies to the discipline of nursing, owing to the complexity of nursing interventions. Following significant revision, the updated guidance of the Medical Research Council (MRC) adopts a pluralistic view to intervention development and evaluation, including a theory-based perspective. This perspective promotes the use of program theory, aiming to understand how and under what circumstances interventions lead to change. In this discussion paper, we reflect the recommended use of program theory in the context of evaluation studies addressing complex nursing interventions. First, we review the literature by investigating the question whether and how evaluation studies targeting complex interventions used theory and to what extent program theories may contribute to enhance the theoretical foundations of intervention studies in nursing. Second, we illustrate the nature of theory-based evaluation and program theories. Third, we argue how this may impact theory building in nursing in general. We finish by discussing which resources, skills and competencies are necessary to fulfill the demanding task of undertaking theory-based evaluations. We caution against an oversimplified interpretation of the updated MRC guidance regarding the theory-based perspective, e.g. by using simple linear logic models, rather than articulating program theories. Instead, we encourage researchers to embrace the corresponding methodology, i.e. theory-based evaluation. With the prevailing perspective of knowledge production in crisis, we might be on the verge of a paradigm shift in health intervention research. Viewed through this lens, the updated MRC guidance could lead to a renewed understanding of what constitutes useful knowledge in nursing. This may facilitate knowledge production and, thereby, contribute to improve nursing practice for the benefit of the patient.

How to collaborate for health throughout the project timeline - a longitudinal study reflecting on implemented strategies in three projects for a healthy living environment

BACKGROUND

When improving the health of local and regional populations, cross-sector collaboration between different policy domains, non-governmental organisations and citizens themselves is needed. Previously, enabling factors and strategies have been identified to improve cross-sector collaboration for health. However, few longitudinal studies have been conducted to understand how the implementation of strategies for cross-sector collaboration changes throughout the collaboration process. The aim of this study is therefore to learn more about the different strategies that were implemented throughout three cross-sector collaboration projects for a healthy living environment.

METHODS

The realist evaluation approach was used to understand how the implemented strategies worked, in which context, why and with what outcomes. Project partners were asked to reflect on their implemented strategies at two different moments in the project timelines, and quarterly updates with project leaders were held. In addition two reference panels were organised for data triangulation.

RESULTS

Three key insights for successful cross-sector collaboration throughout projects for a healthy living environment were identified, namely 1. Investing in trust among the partners and faith in the project has a positive influence on continuing the collaboration throughout the project; 2. Making stakeholders actively participate throughout the project requires additional strategies after the onset of the project, and 3. Defining roles, tasks, and other prerequisites at the start of the project helps in pursuing the project over time, but needs re-examination throughout the project. These key insights were based on multiple examples of implemented strategies, linked to context, mechanisms and outcomes.

CONCLUSIONS

This study shows the different strategies that can be employed as the collaboration in projects for a healthy living environment progresses. We found that 'trust' does not merely include the relationships built between the partners, but at the onset of projects can also be based on faith in the project itself. In addition, as it can be difficult to foresee the right investments and strategies at the onset of the project, frequent reflection moments to choose fitting strategies might benefit regional partners in their cross-sector collaboration for health.

Realist analysis of whether emergency departments with primary care services generate 'provider-induced demand'

Background

It is not known whether emergency departments (EDs) with primary care services influence demand for non-urgent care (‘provider-induced demand’). We proposed that distinct primary care services in EDs encourages primary care demand, whereas primary care integrated within EDs may be less likely to cause additional demand. We aimed to explore this and explain contexts (C), mechanisms (M) and outcomes (O) influencing demand.

Methods

We used realist evaluation methodology and observed ED service delivery. Twenty-four patients and 106 staff members (including Clinical Directors and General Practitioners) were interviewed at 13 EDs in England and Wales (240 hours of observations across 30 days). Field notes from observations and interviews were analysed by creating ‘CMO’ configurations to develop and refine theories relating to drivers of demand.

Results

EDs with distinct primary care services were perceived to attract demand for primary care because services were visible, known or enabled direct access to health care services. Other influencing factors included patients’ experiences of accessing primary care, community care capacity, service design and population characteristics.

Conclusions

Patient, local-system and wider-system factors can contribute to additional demand at EDs that include primary care services. Our findings can inform service providers and policymakers in developing strategies to limit the effect of potential influences on additional demand when demand exceeds capacity.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12873-022-00709-2.

Rivalries for attention: insights from a realist evaluation of a postgraduate competency-based medical education implementation in Canada

BACKGROUND

Implementing competency-based medical education (CBME) in post-graduate medical education (PGME) is a complex process that requires multiple systemic changes in a complex system that is simultaneously engaged in multiple initiatives. These initiatives often compete for attention during the implementation of CBME and produce unintended and unanticipated consequences. Understanding the impact of this context is necessary for evaluating the effectiveness of CBME. The purpose of the study was to identify factors, such as contexts and processes, that contribute to the implementation of CBME.

METHODS

We conducted a realist evaluation using data collected from 15 programs through focus groups with residents (2 groups, n = 16) and faculty (one group, n = 8), and semi-structured interviews with program directors (n = 18), and program administrators (n = 12) from 2018 to 2021. Data were analyzed using a template analysis based on a coding framework that was developed from a sample of transcripts, the context-mechanism-outcomes framework for realist evaluations, and the core components of CBME.

RESULTS

The findings demonstrate that simultaneous initiatives in the academic health sciences system creates a key context for CBME implementation - rivalries for attention - and specifically, the introduction of curricular management systems (CMS) concurrent to, but separate from, the implementation of CBME. This context influenced participants' participation, communication, and adaptation during CBME implementation, which led to change fatigue and unmet expectations for the collection and use of assessment data.

CONCLUSIONS

Rival initiatives, such as the concurrent implementation of a new CMS, can have an impact on how programs implement CBME and greatly affect the outcomes of CBME. Mitigating the effects of rivals for attention with flexibility, clear communication, and training can facilitate effective implementation of CBME.

How can community-based (re)engagement initiatives meet the needs of 'NEET' young people? Findings from the theory gleaning phase of a realist evaluation in Sweden

Objective

There has been a lack of systematic and theoretically underpinned evaluations, internationally and in Sweden, of local multi-component initiatives delivered outside public employment services and formal education systems to young people who are not in employment, education or training (‘NEETs’). To bridge this knowledge gap, the objective of this study was to present findings from the theory gleaning phase of a realist evaluation aimed at assessing how Swedish community-based initiatives may work to (re)engage vulnerable ‘NEET’ young people in education or employment, under what conditions and why.

Results

Based on insights gleaned and synthesised from various sources, three candidate programme theories were elicited drawing attention to the importance of community-based initiatives in Sweden adopting a ‘caring approach’, a ‘capability approach’ and a ‘collaborative approach’ to (re)engage ‘NEET’ young people in education or employment. While limited to the initial phase of theory gleaning, the study provides valuable insights into the potential functioning of (re)engagement initiatives directed towards vulnerable ‘NEETs’ in addition to increasing the transparency of a highly iterative research project.

Making implementation science more real

Implementation science in healthcare aims to understand how to get evidence into practice. Once this is achieved in one setting, it becomes increasingly difficult to replicate elsewhere. The problem is often attributed to differences in context that influence how and whether implementation strategies work. We argue that realist research paradigms provide a useful framework to express the effect of contextual factors within implementation strategy causal processes. Realist studies are theory-driven evaluations that focus on understanding how and why interventions work under different circumstances. They consider the interaction between contextual circumstances, theoretical mechanisms of change and the outcomes they produce, to arrive at explanations of conditional causality (i.e., what tends to work, for whom, under what circumstances). This Commentary provides example applications using preliminary findings from a large realist implementation study of system-wide value-based healthcare initiatives in New South Wales, Australia. If applied judiciously, realist implementation studies may represent a sound approach to help optimise delivery of the right care in the right setting and at the right time.

Contributing to collaborative health governance in Africa: a realist evaluation of the Universal Health Coverage Partnership

Background

Policy dialogue, a collaborative governance mechanism, has raised interest among international stakeholders. They see it as a means to strengthen health systems governance and to participate in the development of health policies that support universal health coverage. In this context, WHO has set up the Universal Health Coverage Partnership. This Partnership aims to support health ministries in establishing inclusive, participatory, and evidence-informed policy dialogue. The general purpose of our study is to understand how and in what contexts the Partnership may support policy dialogue and with what outcomes. More specifically, our study aims to answer two questions: 1) How and in what contexts may the Partnership initiate and nurture policy dialogue? 2) How do collaboration dynamics unfold within policy dialogue supported by the Partnership? 

Methods

We conducted a multiple-case study realist evaluation based on Emerson’s integrative framework for collaborative governance to investigate the role of the Partnership in policy dialogue on three policy issues in six sub-Saharan African countries: health financing (Burkina Faso and Democratic Republic of Congo), health planning (Cabo Verde, Niger, and Togo), and aid coordination for health (Liberia). We interviewed 121 key informants, analyzed policy documents, and observed policy dialogue events.

Results

The Partnership may facilitate the initiation of policy dialogue when: 1) stakeholders feel uncertain about health sector issues and acknowledge their interdependence in responding to such issues, and 2) policy dialogue coincides with their needs and interests. In this context, policy dialogue enables stakeholders to build a shared understanding of issues and of the need for action and encourages collective leadership. However, ministries’ weak ownership of policy dialogue and stakeholders’ lack of confidence in their capacity for joint action hinder their engagement and curb the institutionalization of policy dialogue.

Conclusions

Development aid actors wishing to support policy dialogue must do so over the long term so that collaborative governance becomes routine and a culture of collaboration has time to grow. Public administrations should develop collaborative governance mechanisms that are transparent and intelligible in order to facilitate stakeholder engagement.

The workings of an action learning program for building mental health promotion capacity - A realist evaluation

Action learning is a promising approach for building mental health promotion (MHP) capacity. The aim of this study is to explore how action learning processes can strengthen MHP capacity within and across organizations in a community setting. We applied an embedded case study design and a realist evaluation framework to explore key combinations of mechanisms and contextual factors that generated the emergent MHP capacity outcomes of an action learning program, i.e. context-mechanism-outcome-configurations (CMO-configurations). Data consisted of 18 semi-structured face-to-face interviews, 10 telephone interviews, two group interviews, observations, and documents. Interviewees (n = 21) were participants and management employees. Our analytical provision of CMO-configurations provides insights into how contextual factors, such as participant motivation, organizational support, and existing task descriptions, in combination with certain program mechanisms, such as legitimization of specific agendas, learning-by-doing, and collaborations across organizational boundaries, explain the identified outcomes. Outcomes ranged across implementation of MHP initiatives, personal development among participants, and relational and collaborative development. Taken together, our results strengthen the notion that mechanisms of action learning hold the potential to build MHP capacity on an individual, organizational, and community level. This study, also, illustrates that realist evaluation offers a relevant methodology for investigating the underlying workings of capacity building programs.

Realist inquiry: a new way to think about overcrowding interventions

OBJECTIVES

Physician-focused throughput initiatives are intended to mitigate the effects of emergency department (ED) overcrowding. Our tertiary care academic hospital recently piloted an emergency physician leader role intended to improve throughput. Although a separate experimental evaluation of this initiative was undertaken, it was expected that such an evaluation could not alone provide the necessary information to inform 'how' the emergency physician leader intervention worked. The objective of this study was to conduct a realist evaluation of the emergency physician leader. Realist inquiry utilizes Context Mechanism Outcome configurations to determine the impacts of interventions that are often missed by quantitative analysis. Using a realist perspective, this study aimed to evaluate the emergency physician leader initiative's effects on throughput with the goal of building transferable lessons to the implementation of future interventions.

METHODS

Semi-structured interviews with key stakeholders in the intervention were conducted and analyzed using thematic and template techniques specifically aimed at identifying Context Mechanism Outcomes.

RESULTS

13 interviews were conducted with physicians and nurses who identified contexts and mechanisms which promoted or impeded ED throughput. For example, in situations where there was a clear indication for imaging or management that could not be initiated within the scope of a nursing protocol, the emergency physician leader initiating these orders was felt to promote ED throughput. Conversely, in contexts where there was no nurse available to fulfill early orders, the emergency physician leader's initiation of orders was perceived to impede throughput.

CONCLUSION

This evaluation provides insights into the reasoning and behaviour of individuals involved in the emergency physician leader initiative and provides a systematic approach to unraveling its complex causal pathway. Knowledge of context-mechanism-outcome relationships may help implementers design and measure the impact of future physician-focused throughput interventions.

Contextual factors of external inspections and mechanisms for improvement in healthcare organizations: A realist evaluation

External inspections constitute a key element of healthcare regulation. Improved quality of care is one of the important goals of inspections but the mechanisms of how inspections might contribute to quality improvement are poorly understood. Drawing on interviews with healthcare professionals and managers and health record data from inspected organizations, we used a realist evaluation approach to explore how twelve inspections of healthcare providers in x= Norway influenced quality improvement. We found that for inspections to contribute to quality improvement, there must be contextual structures present supporting accountability and engaging staff in improvement work. When such structures are present, inspections can contribute to improvement by creating awareness of gaps between desired and current practices, which leads to readiness for change and stimulates intra-organizational reasoning around quality improvement. We discuss our findings using the theory of de- and recoupling, noting how regulators can identify decoupling between intended goals, management systems, practices, and patient outcomes. We further argue that regulators can contribute to a recoupling between these levels by having the capacity to track the providers' clinical performance over time. This will hold the organization accountable for implementing improvement measures and evaluate the effects of the measures on quality of care.

At the interface between the formal and informal, the actual and the real: a realist study protocol for governance and accountability practices in urban settings focusing on adolescent sexual and reproductive health and rights

BACKGROUND

This paper presents the protocol of a study that aims at exploring how different multi-level governance arrangements impact on sexual and reproductive health of adolescents living in informal settlements. The overall objective of this study is to contribute to a better understanding of the causal chains underlying accountability in sexual and reproductive health for adolescent girls and young women living in urban informal settlements in low-and middle-income countries.

METHODS

The overarching methodology is realist evaluation. The study adopts a case study design, through which governance and accountability practices in Mumbai, Delhi, Cotonou and Kampala will be examined. Different social science methods to develop and test a programme theory will be used. Heuristic tools for the analysis of the accountability ecosystem and mapping of governance arrangements, drawing from contextual political analysis and critical realism, will be developed in order to identify the intervention-context-actor-mechanism-outcome configurations.

DISCUSSION

The methodological approach is geared towards building robust case-based explanation with due attention to context and the roles of different actors. The combination of different social science methods will lead us to a better grasp of the inherently political nature of social accountability.

What helps older people persevere with yoga classes? A realist process evaluation of a COVID-19-affected yoga program for fall prevention

Background

Falls among older people are a major global health concern. This process evaluation investigates the experience of participants aged 60+ in a yoga program aimed at preventing falls which transitioned from studio-based classes to online classes in response to COVID-19 restrictions. We sought to understand how the Successful AGEing (SAGE) yoga program functioned in both settings and as a hybrid program, and to explain why it worked well for most participants.

Methods

Realist process evaluation was used to explore the factors that facilitated a successful transition for most participants, and to consider why it did not work for a minority. This approach develops program theories that describe which mechanisms an intervention is (or is not) activating, and how this is mediated by context to generate process outcomes. Data included interviews with participants (n = 21) and yoga instructors (n = 3), self-report feedback forms (n = 46), observation of classes and routine process measures.

Results

Factors that facilitated a successful transition for most participants included the quality of yoga instruction, the program format and inherent characteristics of yoga. Gains in transitioning online included continuity and greater convenience. Losses included perceived reduction in the effectiveness of yoga instruction. There were greater challenges for people struggling with pain and in disadvantageous home environments. We identified six program theories configured around 16 mechanisms: 1. It’s worth the effort and 2. In expert hands (these had the same mechanisms: value expectancy, therapeutic alliance and achievement/mastery), 3. A communal experience (these mechanisms were shared experience, social connection, social comparison and peer checking), 4. Putting yoga within reach (accessibility, convenience, gratitude), 5. Building yoga habits (purposeful structure, momentum, accountability and continuity), and 6. Yoga’s special properties (embodiment and mindfulness).

Conclusions

This study showed that online delivery of a yoga program for people aged 60+ retained much of the value of a face-to-face program for the majority of participants, and increased the value for some. The structured, communal nature of an organised group program delivered by a skilled instructor, together with yoga’s intrinsic focus on mindfulness, facilitated continued engagement and perceived health benefits, despite the change in delivery mode.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-022-12818-5.

Do the implementation processes of a school-based daily physical activity (DPA) program vary according to the socioeconomic context of the schools? a realist evaluation of the Active at school program

Background

Less than half of Canadian children meet the Canadian Physical Activity (PA) Guidelines, and the proportion is even lower among children living in underprivileged neighbourhoods. Regular PA supports physical, cognitive, and psychological/social health among school-aged children. Successful implementation of school-based daily physical activity (DPA) programs is therefore important for all children and crucial for children who attend schools in lower socioeconomic settings. The purpose of this study is to uncover what worked, for whom, how, and why during the three-year implementation period of a new “flexible” DPA program, while paying particular attention to the socioeconomic setting of the participating schools.

Methods

This study is a realist evaluation using mixed methods for data generation. Longitudinal data were collected in 415 schools once a year during the three-year implementation period of the program using questionnaires. Data analysis was completed in three steps and included qualitative thematic analysis using a mixed inductive and deductive method and chi-square tests to test and refine context-mechanism-outcome (CMO) configurations.

Results

Giving the school teams autonomy in the choice of strategies appropriate to their context have allowed schools to take ownership of program implementation by activating a community empowerment process, which resulted in a cultural shift towards a sustainable DPA provision in most settings. In rural underprivileged settings, the mobilization of local resources seems to have successfully created the conditions necessary for implementing and maintaining changes in practice. In disadvantaged urban settings, implementing local leadership structures (leader, committee, and meetings) provided pivotal assistance to members of the school teams in providing new DPA opportunities. However, without continued external funding, those schools seem unable to support local leadership structures on their own, jeopardizing the sustainability of the program for children living in disadvantaged urban areas.

Conclusion

By exploring CMO configurations, we have been able to better understand what worked, for whom, how and why during the three-year implementation period of the Active at School! program. When implementing DPA policies, decision makers should consider adjusting resource allocations to meet the actual needs of schools from different backgrounds to promote equal PA opportunities for all children.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-022-12797-7.

The effectiveness of extended-duration supervision training for nurses and allied health professionals: A realist evaluation

BACKGROUND

Training helps maintain high-quality supervision and its associated benefits (e.g. reduced burnout, improved care). While studies have previously evaluated extended-duration supervision training programmes, none have treated these as complex interventions so have not employed realist approaches.

OBJECTIVES

Building on a previous realist synthesis, this evaluation tests and develops programme theory for extended-duration supervision training to answer the question: to what extent does the supervision training programme work, for whom, under what circumstances and why?

DESIGN

We conducted a realist evaluation of a novel state-wide Victorian 3-month supervision training programme including one or two 3.5-h workshops followed by weekly reflexive longitudinal audio diaries (LADs) for up to 12 weeks.

METHODS

Realist evaluation data comprised 25 entrance interviews with nurses and allied health professionals, 176 LADs (and 29 written diaries), and 23 exit interviews. We employed team-based realist analysis to identify context-mechanism-outcome configurations (CMOCs) to test and develop programme theory.

RESULTS

We refined four recurring CMOCs from the realist synthesis programme theory, found insufficient evidence for two, and established five new recurring CMOCs. We identified multiple positive outcomes from our extended-duration supervision training programme (e.g. improved supervisor practices) through various mechanisms relating to pedagogy (e.g. weekly reflexivity), supervisors (e.g. engagement), and workplaces (e.g. enabling supervision cultures). Some negative outcomes were reported (e.g. decreased engagement) through various mechanisms (e.g. suboptimal training design). Such mechanisms were thought to come about by diverse contexts including supervisors (e.g. inexperienced/experienced), professions (nursing/allied health), and organisations (supervision-enabled/non-enabled cultures).

CONCLUSIONS

Our findings extend the realist synthesis programme theory by highlighting various contexts triggering outcome-generating mechanisms. Programme outcomes are maximised through ongoing supervisor reflexivity paying attention to facilitator-supervisor relationships, as well as protected time for supervisors to translate learning into practice.

How do supported employment programs work? Answers from a systematic literature review

Many studies have found that supported employment (SE) has effectively helped people with severe mental illness obtain and maintain competitive employment. However, most SE studies have asked "What works?" rather than discerning what works for whom, in what circumstances, in what respects and how. It is important to understand the outcomes of SE and identify the impact factors (contexts and mechanisms) that can trigger them. Four literature databases were searched for studies that analyzed counseling settings. Overall, 104 publications met the inclusion criteria. The review showed that most of the research on SE programs were one-dimensional, looking at either the effects of SE programs, the client or the professional, or the relationship between clients and professionals. The model reveals that impact factors are interconnected and can have a cumulative impact on the client, professionals, and the environment.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10775-022-09533-3.

Are realist randomised controlled trials possible? A reflection on the INCLUSIVE evaluation of a whole-school, bullying-prevention intervention

We previously proposed that realist randomised controlled trials could be used to evaluate how, for whom and under what conditions complex interventions can be used to activate mechanisms to improve health. While this idea was accepted by some, it was also met with resistance, particularly from some realist evaluators who believe that trials are inextricably positivist and dependent on constant conjunctions to understand causation, and that realist trials are unfeasible because participants and contexts will be insufficiently diverse to enable the testing of context-mechanism-outcome configurations. In this paper, we reflect on analyses of qualitative and quantitative data from the Initiating Change Locally in Bullying and Aggression through the School Environment (INCLSUIVE) trial, and whether these are useful and aligned with realism. We summarise the concerns expressed by realists and reflect on the philosophical and practical challenges that we encountered and whether or not they are related to the trial's design. Finally, we reflect on the trial's weaknesses and highlight areas that future researchers might consider when running realist trials. We conclude that realist randomised controlled trials are philosophically coherent, practically feasible, and can produce nuanced findings.

Role of trust in sustaining provision and uptake of maternal and child healthcare: Evidence from a national programme in Nigeria

Despite increasing attention to implementation research in global health, evidence from low- and middle-income countries (LMICs) using realist evaluations, in understanding how complex health programmes work remains limited. This paper contributes to bridging this knowledge gap by reporting how, why and in what circumstances, the implementation and subsequent termination of a maternal and child health programme affected the trust of service users and healthcare providers in Nigeria. Key documents were reviewed, and initial programme theories of how context triggers mechanisms to produce intended and unintended outcomes were developed. These were tested, consolidated and refined through iterative cycles of data collection and analysis. Testing and validation of the trust theory utilized eight in-depth interviews with health workers, four focus group discussions with service users and a household survey of 713 pregnant women and analysed retroductively. The conceptual framework adopted Hurley's perspective on 'decision to trust' and Straten et al.'s framework on public trust and social capital theory. Incentives offered by the programme triggered confidence and satisfaction among service users, contributing to their trust in healthcare providers, increased service uptake, motivated healthcare providers to have a positive attitude to work, and facilitated their trust in the health system. Termination of the programme led to most service users' dissatisfaction, and distrust reflected in the reduction in utilization of MCH services, increased staff workloads leading to their decreased performance although residual trust remained. Understanding the role of trust in a programme's short and long-term outcomes can help policymakers and other key actors in the planning and implementation of sustainable and effective health programmes. We call for more theory-driven approaches such as realist evaluation to advance understanding of the implementation of health programmes in LMICs.

Contextual factors and mechanisms that influence sustainability: a realist evaluation of two scaled, multi-component interventions

Background

In 2012, Alberta Health Services created Strategic Clinical Networks^TM (SCNs) to develop and implement evidence-informed, clinician-led and team-delivered health system improvement in Alberta, Canada. SCNs have had several provincial successes in improving health outcomes. Little research has been done on the sustainability of these evidence-based implementation efforts.

Methods

We conducted a qualitative realist evaluation using a case study approach to identify and explain the contextual factors and mechanisms perceived to influence the sustainability of two provincial SCN evidence-based interventions, a delirium intervention for Critical Care and an Appropriate Use of Antipsychotics (AUA) intervention for Senior’s Health. The context (C) + mechanism (M) = outcome (O) configurations (CMOcs) heuristic guided our research.

Results

We conducted thirty realist interviews in two cases and found four important strategies that facilitated sustainability: Learning collaboratives, audit & feedback, the informal leadership role, and patient stories. These strategies triggered certain mechanisms such as sense-making, understanding value and impact of the intervention, empowerment, and motivation that increased the likelihood of sustainability. For example, informal leaders were often hands-on and influential to front-line staff. Learning collaboratives broke down professional and organizational silos and encouraged collective sharing and learning, motivating participants to continue with the intervention. Continual audit-feedback interventions motivated participants to want to perform and improve on a long-term basis, increasing the likelihood of sustainability of the two multi-component interventions. Patient stories demonstrated the interventions’ impact on patient outcomes, motivating staff to want to continue doing the intervention, and increasing the likelihood of its sustainability.

Conclusions

This research contributes to the field of implementation science, providing evidence on key strategies for sustainability and the underlying causal mechanisms of these strategies that increases the likelihood of sustainability. Identifying causal mechanisms provides evidence on the processes by which implementation strategies operate and lead to sustainability. Future work is needed to evaluate the impact of informal leadership, learning collaboratives, audit-feedback, and patient stories as strategies for sustainability, to generate better guidance on planning sustainable improvements with long term impact.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07214-5.

A realist evaluation of the continuum of HIV services for men who have sex with men

Background

Men who have Sex with Men (MSM) represent the risk group that are disproportionately most affected by the human immunodeficiency virus (HIV) and continue to drop-off from the steps of the continuum of HIV services that have been adopted to overcome poor engagement and retention in care. This realist evaluation aimed at: (1) describing the evaluation carried out in Haiti aiming to ascertain why, how and under which circumstances MSM are linked and retained along the continuum, (2) assessing the outcomes of this approach and (3) exploring the motivators and facilitators for the HIV continuum of services through mechanisms and pathways.

Methods

Guided by a realist approach, first, an initial program theory (IPT) was developed based on literature and frameworks review, participant observations and discussions with stakeholders. Then, the IPT was tested using a mixed method explanatory study: a quantitative phase to build the continuum from a cross-sectional analysis, and a qualitative phase to explore the motivators and facilitators related to proper linkages along the continuum. Finally, the IPT was refined by eliciting the mechanisms and pathways for outcomes improvement.

Results

The results showed that the current service delivery model is suboptimal in identifying, engaging, linking and retaining MSM, resulting in loss to follow-up at every step of the continuum and failure to fully realize the health and prevention benefits of antiretroviral. However, the mechanisms through which linkages across the continuum can be improved are: self-acceptance, sense of community support and sense of comprehensive and tailored HIV services. These mechanisms are based on 10 different pathways: self-esteem, awareness and pride, perception of HIV risk, pcceptance and HIV status, addressing community stigma, strengthening of MSM organizations and community networks, societal acceptation and tolerance, stigma reduction training for healthcare providers, engagement of peers as educators and navigators and, adapted services delivery through drug dispensing points and mobile technology and financial assistance.

Conclusions

The study findings show that engagement, adherence and retention to the continuum of HIV service for MSM are affected by a multi-layer of factors, thus highlighting the importance of taking a comprehensive approach to improve the program.

Understanding the mechanisms generating outcomes in a Danish peer support intervention for socially vulnerable people with type 2-diabetes: a realist evaluation

BACKGROUND

Despite an increasing use and positive effects of peer support interventions, little is known about how the outcomes are produced. Thus, it is essential not only to measure outcomes, but also to identify the mechanisms by which they are generated. Using a realist evaluation approach, we aimed to identify the mechanisms generating outcomes in a Danish peer support intervention for socially vulnerable people with type 2-diabetes (peers). By investigating the participating peers' interactions, we furthermore examined how their individual contextual factors either facilitated or hindered the mechanisms in operation.

METHODS

We used a multi-method case-study design (n = 9). Data included semi-structured interviews with four key groups of informants (peer, peer supporter, project manager, and a diabetes nurse) for each case (n = 25). Furthermore, we collected survey data from peers both before and after participation (n = 9). The interview data were analysed using a systematic text condensation, and the Intervention-context-actor-mechanism-outcome framework was used to structure the analysis.

RESULTS

We identified 2 groups of mechanisms that improved diabetes self-management and the use of healthcare services (outcomes): 'perceived needs and readiness' and 'encouragement and energy'. However, the mechanisms only generated the intended outcomes among peers with a stable occupation and financial situation, a relatively good health condition, and sufficient energy (all defined as contextual factors). Independent of these contextual factors, 'experience of social and emotional support' was identified as a mechanism within all peers that increased self-care awareness (defined as output). Dependent on whether the contextual factors facilitated or hindered the mechanisms to generate outcomes, we categorised the peers into those who achieved outcomes and those who did not.

CONCLUSIONS

We identified two groups of mechanisms that improved the peers' diabetes self-management and use of healthcare services. The mechanisms only generated the intended outcomes if peers' individual contextual factors facilitated an active interaction with the elements of the intervention. However, independent of these contextual factors, a third group of mechanisms increased self-care awareness among all peers. We highlight the importance of contextual awareness of the target groups in the design and evaluation of peer support interventions for socially vulnerable people with type 2-diabetes.

TRIAL REGISTRATION

ClinicalTrials.gov, Retrospective Registration (20 Jan 2021), registration number NCT04722289 .

Fundamental challenges in assessing the impact of research infrastructure

Clinical research infrastructure is one of the unsung heroes of the scientific response to the current COVID-19 pandemic. The extensive, long-term funding into research support structures, skilled people, and technology allowed the United Kingdom research response to move off the starting blocks at pace by utilizing pre-existing platforms. The increasing focus from funders on evaluating the outcomes and impact of research infrastructure investment requires both a reframing and progression of the current models in order to address the contribution of the underlying support infrastructure. The majority of current evaluation/outcome models focus on a "pipeline" approach using a methodology which follows the traditional research funding route with the addition of quantitative metrics. These models fail to embrace the complexity caused by the interplay of previous investment, the coalescing of project outputs from different funders, the underlying infrastructure investment, and the parallel development across different parts of the system. Research infrastructure is the underpinning foundation of a project-driven research system and requires long-term, sustained funding and capital investment to maintain scientific and technological expertise. Therefore, the short-term focus on quantitative metrics that are easy to collect and interpret and that can be assessed in a roughly 5-year funding cycle needs to be addressed. The significant level of investment in research infrastructure necessitates investment to develop bespoke methodologies that develop fit-for-purpose, longer-term/continual approach(es) to evaluation. Real-world research should reflect real-world evaluation and allow for the accrual of a narrative of value indicators that build a picture of the contribution of infrastructure to research outcomes. The linear approach is not fit for purpose, the research endeavour is a complex, twisted road, and the evaluation approach needs to embrace this complexity through the development of realist approaches and the rapidly evolving data ecosystem. This paper sets out methodological challenges and considers the need to develop bespoke methodological approaches to allow a richer assessment of impact, contribution, attribution, and evaluation of research infrastructure. This paper is the beginning of a conversation that invites the community to "take up the mantle" and tackle the complexity of real-world research translation and evaluation.