Using a realist approach to evaluate smoking cessation interventions targeting pregnant women and young people

Scaling-up digital follow-up care services: collaborative development and implementation of Remote Patient Monitoring pilot initiatives to increase access to follow-up care

Background

COVID-19 increased the demand for Remote Patient Monitoring (RPM) services as a rapid solution for safe patient follow-up in a lockdown context. Time and resource constraints resulted in unplanned scaled-up RPM pilot initiatives posing risks to the access and quality of care. Scalability and rapid implementation of RPM services require social change and active collaboration between stakeholders. Therefore, a participatory action research (PAR) approach is needed to support the collaborative development of the technological component while simultaneously implementing and evaluating the RPM service through critical action-reflection cycles.

Objective

This study aims to demonstrate how PAR can be used to guide the scalability design of RPM pilot initiatives and the implementation of RPM-based follow-up services.

Methods

Using a case study strategy, we described the PAR team's (nurses, physicians, developers, and researchers) activities within and across the four phases of the research process (problem definition, planning, action, and reflection). Team meetings were analyzed through content analysis and descriptive statistics. The PAR team selected ex-ante pilot initiatives to reflect upon features feedback and participatory level assessment. Pilot initiatives were investigated using semi-structured interviews transcribed and coded into themes following the principles of grounded theory and pilot meetings minutes and reports through content analysis. The PAR team used the MoSCoW prioritization method to define the set of features and descriptive statistics to reflect on the performance of the PAR approach.

Results

The approach involved two action-reflection cycles. From the 15 features identified, the team classified 11 as must-haves in the scaled-up version. The participation was similar among researchers (52.9%), developers (47.5%), and physicians (46.7%), who focused on suggesting and planning actions. Nurses with the lowest participation (5.8%) focused on knowledge sharing and generation. The top three meeting outcomes were: improved research and development system (35.0%), socio-technical-economic constraints characterization (25.2%), and understanding of end-user technology utilization (22.0%).

Conclusion

The scalability and implementation of RPM services must consider contextual factors, such as individuals' and organizations' interests and needs. The PAR approach supports simultaneously designing, developing, testing, and evaluating the RPM technological features, in a real-world context, with the participation of healthcare professionals, developers, and researchers.

Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

BACKGROUND

The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods.

METHODS

There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments.

RESULTS

Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six 'core elements' that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout.

LIMITATIONS

The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development.

CONCLUSIONS

This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as 'events in systems' that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers.

FUTURE WORK

We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources.

FUNDING

This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).

The Music Engagement Program for people with Alzheimer's disease and dementia: Pilot feasibility trial outcomes

Alzheimer's disease and dementia are common, highly disabling conditions frequently requiring residential care. This exploratory proof-of-concept study aimed to determine if the specialised Music Engagement Program (MEP) was sustainable, acceptable, and effective in improving quality of life, emotional wellbeing, and depression symptoms in this population. Sixteen residents, six staff members, and three family and community members took part in the evaluation of the MEP for people living with dementia in a residential aged-care nursing home in Canberra, Australia. Multiple methods were used. Quantitative evaluation assessed residents' depression symptoms (Cornell scale) at pre- and post-intervention, and emotional wellbeing pre- and post-session. Qualitative interviews with staff, and family and community members addressed the MEP's acceptability and potential sustainability. Results showed residents' mean depression scores were reduced from pre- to post-intervention (p = .039; d_z = 0.72). Interviews established multiple benefits for residents including improved mood, calmness, and reduced aggression. However, staff did not believe it was feasible to continue the MEP sessions beyond the trial period without an external facilitator, citing potential difficulties in adhering to internal activities due to time constraints. This pilot study provides encouraging preliminary evidence for the MEP's acceptability and potential effectiveness for improving depression and wellbeing in this group.

Explaining Mechanisms That Influence Smoke-Free Implementation at the Local Level: A Realist Review of Smoking Bans

INTRODUCTION

While studies have been undertaken to understand the adoption of outdoor and indoor smoking bans, not much is known about why implementation of smoke-free (SF) environments differs at local levels. As most European countries remain at the level of indoor bans, we aim to translate existing evidence into practical recommendations on how to improve SF (outdoor) implementation within European municipalities.

METHODS

We applied six methodological steps of a realist review consistent with the RAMESES publication standards for realist syntheses. Literature search was conducted in PubMed/MEDLINE and Web of Science. In total, 3829 references were screened, of which 43 were synthesized. Studies dating from 2004 to 2015 with rigor evidence of SF implementation at the local level were selected. Implementation outcomes were SF enforcement, monitoring, nonsmoking compliance, and public support in cities.

RESULTS

The explanatory realist framework links four innovation stages with three context-mechanism-outcome (CMO)configurations. We identified "triggering trust," "increasing priorities," and "limiting opposing interests" as underlying mechanisms, when (1) establishing, (2) developing, (3) contesting, and (4) implementing local smoking bans. The CMO propositions (CMOs) support practical recommendations, such as (1) providing authorities with local data when establishing and developing bans, (2) developing long-term strategies and implementing state-funded SF programs to prioritize sustained enforcement, and (3) limiting opposing interests through the use of the child protection frame.

CONCLUSIONS

This is the first realist review on the implementation of SF enviroments at the local level. The process-oriented theory explains how and why CMOs determine SF development in cities and municipalities from planning until implementation.

IMPLICATIONS

In 2015, only 16% of the world's population lived under the jurisdiction of comprehensive SF laws. The findings of this realist review are useful to implement WHO goals of the Framework Convention on Tobacco Control (FCTC) and specifically SF environments at more local levels and to adjust them to specific contextual circumstances. This paper unpacks three mechanisms that could be triggered by SF strategies developed at local levels and that can result in improved policy implementation. Such evidence is needed to enhance SF strategies at the level of cities and municipalities and to achieve WHO "Healthy Cities Network" objectives.

Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the "black box" to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.

Enhancing implementation of smoke-free places: A comparative qualitative study across seven European cities

BACKGROUND

Advocacy, resources and intersubjective reasonable arguments are known as factors that contribute to smoke-free (SF) adoption and implementation in Chinese and Anglo-Saxon places. Less is known about how the implementation of smoking bans differs across European places. The aim of this qualitative comparative study is to identify and classify the SF policy implementation processes and types undertaken at the local level in seven European cities according to the views of local bureaucrats and sub-national stakeholders.

METHOD

Semi-structured expert interviews (n = 56) with local decision makers and stakeholders were conducted as qualitative part of the comparative SILNE-R project in Belgium (Namur), Finland (Tampere), Germany (Hanover), the Republic of Ireland (Dublin), the Netherlands (Amersfoort), Italy (Latina), and Portugal (Coimbra). Qualitative interviews were analyzed using the framework analysis.

RESULTS

Implementation of SF environments predominantly focuses on indoor bans or youth-related settings. Progressive-hungry (Dublin), moderate-rational (Tampere), upper-saturated (Hanover, Amersfoort), and lower saturated (Namur, Coimbra, Latina) implementation types can be distinguished. These four types differ with regards to their engagement in enhancing SF places as well as along their level of perceived tobacco de-normalization and public smoking visibility. In all municipalities SF environments are adopted at national levels, but are differently implemented at the local level due national policy environments, enforcement strategies and the level of collaboration. Major mechanisms to expand SF regulations were found to be scientific evidence, public support, and the child protection frame. However, counter-mechanisms of closure occur if data on declining prevalence and new youth addiction trends trigger low prioritization.

CONCLUSIONS

This study found four SF implementation types two mechanisms of progressive expansion and defensive closure. Development and enhancement of smoking bans requires a suitable national policy environment and indirect national-level support of self-governed local initiatives. Future SF policies can be enhanced by laws pertaining to places frequented by minors.

Using Theories of Change to inform implementation of health systems research and innovation: experiences of Future Health Systems consortium partners in Bangladesh, India and Uganda

Background

The Theory of Change (ToC) is a management and evaluation tool supporting critical thinking in the design, implementation and evaluation of development programmes. We document the experience of Future Health Systems (FHS) Consortium research teams in Bangladesh, India and Uganda with using ToC. We seek to understand how and why ToCs were applied and to clarify how they facilitate the implementation of iterative intervention designs and stakeholder engagement in health systems research and strengthening.

Methods

This paper combines literature on ToC, with a summary of reflections by FHS research members on the motivation, development, revision and use of the ToC, as well as on the benefits and challenges of the process. We describe three FHS teams’ experiences along four potential uses of ToCs, namely planning, communication, learning and accountability.

Results

The three teams developed ToCs for planning and evaluation purposes as required for their initial plans for FHS in 2011 and revised them half-way through the project, based on assumptions informed by and adjusted through the teams’ experiences during the previous 2 years of implementation. All teams found that the revised ToCs and their accompanying narratives recognised greater feedback among intervention components and among key stakeholders. The ToC development and revision fostered channels for both internal and external communication, among research team members and with key stakeholders, respectively. The process of revising the ToCs challenged the teams’ initial assumptions based on new evidence and experience. In contrast, the ToCs were only minimally used for accountability purposes.

Conclusions

The ToC development and revision process helped FHS research teams, and occasionally key local stakeholders, to reflect on and make their assumptions and mental models about their respective interventions explicit. Other projects using the ToC should allow time for revising and reflecting upon the ToCs, to recognise and document the adaptive nature of health systems, and to foster the time, space and flexibility that health systems strengthening programmes must have to learn from implementation and stakeholder engagement.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-017-0272-y) contains supplementary material, which is available to authorized users.

Framework for Selecting Best Practices in Public Health: A Systematic Literature Review

Evidence-based public health has commonly relied on findings from empirical studies, or research-based evidence. However, this paper advocates that practice-based evidence derived from programmes implemented in real-life settings is likely to be a more suitable source of evidence for inspiring and guiding public health programmes. Selection of best practices from the array of implemented programmes is one way of generating such practice-based evidence. Yet the lack of consensus on the definition and criteria for practice-based evidence and best practices has limited their application in public health so far. To address the gap in literature on practice-based evidence, this paper hence proposes measures of success for public health interventions by developing an evaluation framework for selection of best practices. The proposed framework was synthesised from a systematic literature review of peer-reviewed and grey literature on existing evaluation frameworks for public health programmes as well as processes employed by health-related organisations when selecting best practices. A best practice is firstly defined as an intervention that has shown evidence of effectiveness in a particular setting and is likely to be replicable to other situations. Regardless of the area of public health, interventions should be evaluated by their context, process and outcomes. A best practice should hence meet most, if not all, of eight identified evaluation criteria: relevance, community participation, stakeholder collaboration, ethical soundness, replicability, effectiveness, efficiency and sustainability. Ultimately, a standardised framework for selection of best practices will improve the usefulness and credibility of practice-based evidence in informing evidence-based public health interventions. Significance for public healthBest practices are a valuable source of practice-based evidence on effective public health interventions implemented in real-life settings. Yet, despite the frequent branding of interventions as best practices or good practices, there is no consensus on the definition and desirable characteristics of such best practices. Hence, this is likely to be the first systematic review on the topic of best practices in public health. Having a single widely accepted framework for selecting best practices will ensure that the selection processes by different agencies are fair and comparable, as well as enable public health workers to better appreciate and adopt best practices in different settings. Ultimately, standardisation will improve the credibility and usefulness of practice-based evidence to that of research-based evidence.

Using mixed methods effectively in prevention science: designs, procedures, and examples

There is growing interest in using a combination of quantitative and qualitative methods to generate evidence about the effectiveness of health prevention, services, and intervention programs. With the emerging importance of mixed methods research across the social and health sciences, there has been an increased recognition of the value of using mixed methods for addressing research questions in different disciplines. We illustrate the mixed methods approach in prevention research, showing design procedures used in several published research articles. In this paper, we focused on two commonly used mixed methods designs: concurrent and sequential mixed methods designs. We discuss the types of mixed methods designs, the reasons for, and advantages of using a particular type of design, and the procedures of qualitative and quantitative data collection and integration. The studies reviewed in this paper show that the essence of qualitative research is to explore complex dynamic phenomena in prevention science, and the advantage of using mixed methods is that quantitative data can yield generalizable results and qualitative data can provide extensive insights. However, the emphasis of methodological rigor in a mixed methods application also requires considerable expertise in both qualitative and quantitative methods. Besides the necessary skills and effective interdisciplinary collaboration, this combined approach also requires an open-mindedness and reflection from the involved researchers.

A realist analysis of hospital patient safety in Wales: applied learning for alternative contexts from a multisite case study

Background

Hospital patient safety is a major social problem. In the UK, policy responses focus on the introduction of improvement programmes that seek to implement evidence-based clinical practices using the Model for Improvement, Plan-Do-Study-Act cycle. Empirical evidence that the outcomes of such programmes vary across hospitals demonstrates that the context of their implementation matters. However, the relationships between features of context and the implementation of safety programmes are both undertheorised and poorly understood in empirical terms.

Objectives

This study is designed to address gaps in conceptual, methodological and empirical knowledge about the influence of context on the local implementation of patient safety programmes.

Design

We used concepts from critical realism and institutional analysis to conduct a qualitative comparative-intensive case study involving 21 hospitals across all seven Welsh health boards. We focused on the local implementation of three focal interventions from the 1000 Lives+patient safety programme: Improving Leadership for Quality Improvement, Reducing Surgical Complications and Reducing Health-care Associated Infection. Our main sources of data were 160 semistructured interviews, observation and 1700 health policy and organisational documents. These data were analysed using the realist approaches of abstraction, abduction and retroduction.

Setting

Welsh Government and NHS Wales.

Participants

Interviews were conducted with 160 participants including government policy leads, health managers and professionals, partner agencies with strategic oversight of patient safety, advocacy groups and academics with expertise in patient safety.

Main outcome measures

Identification of the contextual factors pertinent to the local implementation of the 1000 Lives+patient safety programme in Welsh NHS hospitals.

Results

An innovative conceptual framework harnessing realist social theory and institutional theory was produced to address challenges identified within previous applications of realist inquiry in patient safety research. This involved the development and use of an explanatory intervention–context–mechanism–agency–outcome (I-CMAO) configuration to illustrate the processes behind implementation of a change programme. Our findings, illustrated by multiple nested I-CMAO configurations, show how local implementation of patient safety interventions are impacted and modified by particular aspects of context: specifically, isomorphism, by which an intervention becomes adapted to the environment in which it is implemented; institutional logics, the beliefs and values underpinning the intervention and its source, and their perceived legitimacy among different groups of health-care professionals; and the relational structure and power dynamics of the functional group, that is, those tasked with implementing the initiative. This dynamic interplay shapes and guides actions leading to the normalisation or the rejection of the patient safety programme.

Conclusions

Heightened awareness of the influence of context on the local implementation of patient safety programmes is required to inform the design of such interventions and to ensure their effective implementation and operationalisation in the day-to-day practice of health-care teams. Future work is required to elaborate our conceptual model and findings in similar settings where different interventions are introduced, and in different settings where similar innovations are implemented.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Practitioner opinions on health promotion interventions that work: opening the 'black box' of a linear evidence-based approach

While attempts are being made to improve health promotion by following a linear Evidence-Based (EB) approach, the actors involved are aware that the quality of health promotion is not just a matter of supplying 'evidence-based' interventions to local practitioners, but the result of a situated coproduction process that depends on many factors. This paper explores what constitutes an intervention that works from the perspective of health promotion professionals (HPP), and how, according to them, the development and implementation of interventions should be improved. We interviewed 81 HPPs about the use of 10 health promotion interventions at 30 Municipality Health Services in The Netherlands. The HPPs described an intervention that works as something that produces its intended effects after being realized in a local situation. Interventions are realized by combining elements of a supplied intervention (e.g. a theory, artefacts) with elements that are situated in the local context (e.g. funding, local network). Interventions that are transferred contain implicit assumptions about local contexts, but it is often unclear what precisely constitutes an intervention and what is assumed of local contexts. An intervention that works is a situated configuration of aligned elements. A linear EB approach depends on the realization of the local circumstances in which 'evidence based' interventions can work. Various strategies are possible for approximating such circumstances, but the core assumption that the configuration that is realized in practice is similar to the 'evidence based' intervention seems unrealistic for most health promotion in The Netherlands. Under such circumstances, attention should shift from central quality assurance to the system of actors and the distributed actions and heterogeneous learning processes that together add up to interventions that work.

How to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa

Background

This paper presents the development of a study design built on the principles of theory-driven evaluation. The theory-driven evaluation approach was used to evaluate an adolescent sexual and reproductive health intervention in Mali, Burkina Faso and Cameroon to improve continuity of care through the creation of networks of social and health care providers.

Methods/design

Based on our experience and the existing literature, we developed a six-step framework for the design of theory-driven evaluations, which we applied in the ex-post evaluation of the networking component of the intervention. The protocol was drafted with the input of the intervention designer. The programme theory, the central element of theory-driven evaluation, was constructed on the basis of semi-structured interviews with designers, implementers and beneficiaries and an analysis of the intervention's logical framework.

Discussion

The six-step framework proved useful as it allowed for a systematic development of the protocol. We describe the challenges at each step. We found that there is little practical guidance in the existing literature, and also a mix up of terminology of theory-driven evaluation approaches. There is a need for empirical methodological development in order to refine the tools to be used in theory driven evaluation. We conclude that ex-post evaluations of programmes can be based on such an approach if the required information on context and mechanisms is collected during the programme.