First Things First: How to Elicit the Initial Program Theory for a Realist Evaluation of Complex Integrated Care Programs

Alternative payment models in Dutch hospital care: what works, how, why and under what circumstances? Protocol for a realist evaluation study

INTRODUCTION

The predominant provider payment models in healthcare, particularly fee-for-service, hinder the delivery of high-value care and can encourage healthcare providers to prioritise the volume of care over the value of care. To address these issues, healthcare providers, payers and policymakers are increasingly experimenting with alternative payment models (APMs), such as shared savings (SS) and bundled payment (BP). Despite a growing body of literature on APMs, there is still limited insight into what works in developing and implementing successful APMs, as well as how, why and under what circumstances. This paper presents the protocol for a study that aims to (1) identify these circumstances and reveal the underlying mechanisms through which outcomes are achieved and (2) identify transferrable lessons for successful APMs in practice.

METHODS AND ANALYSIS

Drawing on realist evaluation principles, this study will employ an iterative three-step approach to elicit a programme theory that describes the relationship between context, mechanisms and outcomes of APMs. The first step involves a literature review to identify the initial programme theory. The second step entails empirical testing of this theory via a multiple case study design including seven SS and BP initiatives in Dutch hospital care. We will use various qualitative and quantitative methods, including interviews with involved stakeholders, document analysis and difference-in-differences analyses. In the final step, these data and the applicable formal theories will be combined to test and refine the (I)PT and address the research objectives.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the Research Ethics Review Committee of Erasmus School of Health Policy and Management (Project ID ETH2122-0170). Where necessary, informed consent will be obtained from study participants. Among other means, study results will be disseminated through a publicly available manual for stakeholders (eg, healthcare providers and payers), publications in peer-reviewed scientific journals and (inter)national conference presentations.

A realist evaluation of policy interventions to reduce public subsidies of private dental care in Finland

OBJECTIVES

The Finnish dental care market operates as a dual system, divided between a regulated, affordable public sector and a less regulated, more expensive private sector that receives public subsidies. In 2015 and 2016, two policy interventions were introduced to reduce these subsidies for private dental services. The aim of this study was to evaluate the impact of these policy changes on the dental care market.

METHODS

This study was a realist evaluation. Context-Intervention-Mechanism-Outcome-configurations were applied to elicit an initial program theory (IPT) for the policy interventions. The IPT allowed a complicated system to be reduced to the main components, allowing for better understanding of the underlying mechanisms and the chain of events started by the interventions. The resulting hypotheses about the chain of events and outcomes were tested against a dataset collected from the Social Insurance Institution of Finland (SII) registries on public and private dental visits in the cities of Espoo, Helsinki and Oulu during the years 2010-2016. The used dataset consisted of N = 17 111 625 dental procedures or N = 8 139 990 individual visits (which can include several procedures) at a public (n = 9 097 407 procedures, n = 4 083 475 visits) or a private (n = 8 014 218 procedures or n = 4 056 515 visits) dental clinic. The system was studied during three time periods related to the two interventions in 2015 and in 2016. Changes were evaluated by statistically analysing changes in several key metrics: mean subsidy, mean out-of-pocket price, mean (non-subsidized) price, number of patients treated, number of professionals, procedures per professional, Case-Mix adjusted procedures per professional, patient-to-professional ratio, total procedures.

RESULTS

The 2015 and 2016 reductions to the subsidization of private dental care reduced the average subsidies paid to the private dental sector by 49% [-49.1, -38.8]. A 26% [25.2, 26.7] increase in the out-of-pocket price paid in the private sector was observed. Over the 2 years, 12.2% of patients left the private sector and an increase of 13% was observed in the number of patients treated in the public sector. The public sector increased its number of dental care professionals by 2.3% and the patient-to-professional ratio increased by 9.9% over the 2 years, while the private sector lost 4.6% of its dental care professionals and increased its prices by 4.0% [3.5, 4.5].

CONCLUSIONS

The policy changes had tangible effects on both the private and public sectors of the Finnish dental care market. By reducing subsidies, the private sector became more expensive for patients, causing many to transition to the public sector for their dental needs. While the public sector increased its capacity to accommodate the rise in patients, the demand still outpaced the growth in professionals, hinting at capacity or resource constraints in the public sector. The results also show initial evidence that contrary to the objectives, the policy changes increased the cost to the public sector as subsidized patient cared for in the private sector costs less to the government than treating that same patient in the public sector.

A Realist Review Protocol into the Contexts and Mechanisms That Enable the Inclusion of Environmental Sustainability Outcomes in the Design of Lean Healthcare Improvement Interventions

Healthcare makes a significant contribution to the social, economic and environmental benefits of communities. It is correspondingly a significant employer and consumer of both energy and consumables, often at high costs. Lean, a quality improvement methodology focuses on the elimination of non-value add (NVA) activities (steps that do not add value from the perspective of the customer) to improve the flow of people, information or goods. Increasingly, Lean thinking is evolving from its initial focus on eliminating NVA to a more holistic approach that encompasses sustainability. However, little work has been undertaken intentionally, including environmental sustainability outcomes in Lean healthcare interventions. Realist review methodology facilitates an understanding of the extent to which an intervention works, for whom, in what context, why and how, and has proven useful in research relating to Lean interventions in healthcare settings. This protocol provides details for a realist review that will enable an understanding of the specific contexts in which certain mechanisms are activated that enable the inclusion of environmental sustainability outcomes in the design of Lean healthcare improvement interventions.

Effective factors in planning, implementation, and management of educational program evaluation in medical sciences: A practical guide

BACKGROUND

Educational program evaluation is a complex issue, and it is essential to have knowledge of the potential challenges and solutions during the whole process. The present study aimed to identify the influential components in planning, implementation, and evaluation management of educational programs in medical sciences and then provide an applied guide to guarantee the best possible evaluation by evaluators of educational programs.

MATERIALS AND METHODS

This descriptive study was conducted in three steps. First, the effective components in planning, implementation, and evaluation management of educational programs in medical sciences were reviewed. Second, experts' opinion was asked through a focus group discussion regarding the mentioned components. Third, regarding the complied applied guide, the opinions of 40 medical educationist and program evaluation experts were investigated using a checklist.

RESULTS

An applied guide for planning, implementation, and evaluation management of educational programs in medical sciences consists of eight stages: determining the evaluation questions and standards, determining the type of information required, determining resources to collect information, determining methods and tools to collect information, determining data analysis methods, determining the timing and frequency of reporting, determining the appropriate ways of reporting, and determining strategies to maintain the cooperation of data sources.

CONCLUSION

The spread of educational programs in medical sciences universities leads to an increasing need for program evaluation to provide evidence of their effectiveness and improvement. The present research provided an applied guide to make the evaluation of educational programs feasible by using a set of concepts, principles, methods, theories, and models.

What works for whom, how and why in mental health education for undergraduate health profession students? A realist synthesis protocol

INTRODUCTION

It has been shown that mental health education can support positive attitudes of health profession students towards people with mental health challenges, which supports them to provide optimal healthcare to this group. There are many different approaches to designing and delivering mental health education to health profession students. Each has their own advantages and disadvantages, and often mental health education programmes incorporate a multimodal approach in order to reap the benefits of a variety of teaching and learning approaches. The aim of this study is to understand the current landscape of teaching and learning approaches to mental health education for undergraduate health profession students. We will examine the features of successful outcomes for health profession students for:Learning environment.Knowledge development and retention.Confidence.

MOTIVATION

Preparedness for professional practice.

METHODS

For this, a realist synthesis has been chosen in order to review the literature. Realist synthesis lends itself to the review of complex interventions such as mental health education for undergraduate health profession curricula because it seeks to uncover the range of different mechanisms and context configurations that produce different outcomes. Health profession education and education practice, in general, is complex. A patient and public involvement (PPI) group is involved throughout this study and includes undergraduate health profession students, and members of the St John of Gods Hospital Consumers and Carers Council who are involved at every stage of the research. This study will engage with a stakeholder group who will support the refining of the programme theory.

ETHICS AND DISSEMINATION

Ethical approval has been sought and approved by Royal College of Surgeons, Ireland Ethical Committee (REC number: 212622783). We will aim to write up and publish the full synthesis as a journal article. We will also discuss ways of dissemination outside of academia with our PPI group.

A realist synthesis of interprofessional patient safety activities and healthcare student attitudes towards patient safety

Interprofessional education (IPE) has been used for instilling a positive safety culture within healthcare, yet what interventions work to change healthcare student attitudes and how improved patient safety outcomes are best achieved with this intervention, is unclear and challenging to evaluate. A realist synthesis was undertaken to ascertain how, why, and in what circumstances IPE activities result in a positive change to student attitudes to patient safety. Database searches of CINAHL, MEDLINE, Scopus, and Eric were undertaken in April 2022 to identify relevant studies. Synthesis with a realist framework of analysis, coupled with the development of a program theory was conducted to identify interactions among contexts, intervention, mechanisms, and outcomes (CIMO). Twenty-three articles eligible for inclusion articulated environments in which varied contexts, interventions, and mechanisms were activated to influence student attitudes to patient safety. Findings from this realist synthesis informs awareness into the methods of delivering and evaluating IPE activities and offers new perspectives for educators in planning and evaluating future IPE from a collaborative and positive safety culture perspective.

The role of provincial health administration in supporting district health management teams in the Democratic Republic of Congo: eliciting an initial programme theory of a realist evaluation

BACKGROUND

In 2006, the Ministry of Health in the Democratic Republic of Congo designed a strategy to strengthen the health system by developing health districts. This strategy included a reform of the provincial health administration to provide effective technical support to district health management teams in terms of leadership and management. The provincial health teams were set up in 2014, but few studies have been done on how, for whom, and under what circumstances their support to the districts works. We report on the development of an initial programme theory that is the first step of a realist evaluation seeking to address this knowledge gap.

METHODS

To inform the initial programme theory, we collected data through a scoping review of primary studies on leadership or management capacity building of district health managers in sub-Saharan Africa, a review of policy documents and interviews with the programme designers. We then conducted a two-step data analysis: first, identification of intervention features, context, actors, mechanisms and outcomes through thematic content analysis, and second, formulation of intervention-context-actor-mechanism-outcome (ICAMO) configurations using a retroductive approach.

RESULTS

We identified six ICAMO configurations explaining how effective technical support (i.e. personalised, problem-solving centred and reflection-stimulating) may improve the competencies of the members of district health management teams by activating a series of mechanisms (including positive perceived relevance of the support, positive perceived credibility of provincial health administration staff, trust in provincial health administration staff, psychological safety, reflexivity, self-efficacy and perceived autonomy) under specific contextual conditions (including enabling learning environment, integration of vertical programmes, competent public health administration staff, optimal decision space, supportive work conditions, availability of resources and absence of negative political influences).

CONCLUSIONS

We identified initial ICAMO configurations that explain how provincial health administration technical support for district health management teams is expected to work, for whom and under what conditions. These ICAMO configurations will be tested in subsequent empirical studies.

Enhanced or hindered research benefits? A realist review of community engagement and participatory research practices for non-communicable disease prevention in low- and middle-income countries

INTRODUCTION

Community engagement and participatory research are widely used and considered important for ethical health research and interventions. Based on calls to unpack their complexity and observed biases in their favour, we conducted a realist review with a focus on non-communicable disease prevention. The aim was to generate an understanding of how and why engagement or participatory practices enhance or hinder the benefits of non-communicable disease research and interventions in low- and middle-income countries.

METHODS

We retroductively formulated theories based on existing literature and realist interviews. After initial searches, preliminary theories and a search strategy were developed. We searched three databases and screened records with a focus on theoretical and empirical relevance. Insights about contexts, strategies, mechanisms and outcomes were extracted and synthesised into six theories. Five realist interviews were conducted to complement literature-based theorising. The final synthesis included 17 quality-appraised articles describing 15 studies.

RESULTS

We developed six theories explaining how community engagement or participatory research practices either enhance or hinder the benefits of non-communicable disease research or interventions. Benefit-enhancing mechanisms include community members' agency being realised, a shared understanding of the benefits of health promotion, communities feeling empowered, and community members feeling solidarity and unity. Benefit-hindering mechanisms include community members' agency remaining unrealised and participation being driven by financial motives or reputational expectations.

CONCLUSION

Our review challenges assumptions about community engagement and participatory research being solely beneficial in the context of non-communicable disease prevention in low- and middle-income countries. We present both helpful and harmful pathways through which health and research outcomes are affected. Our practical recommendations relate to maximising benefits and minimising harm by addressing institutional inflexibility and researcher capabilities, managing expectations on research, promoting solidarity in solving public health challenges and sharing decision-making power.

Supporting professionals to implement integrated, person-centered care for people with chronic conditions: the TARGET pilot study

OBJECTIVE

The TARGET program for integrated, person-centered care for people with chronic conditions offers primary care (PC) professionals a set of tools and trainings to actively engage in population segmentation and person-centered needs assessments (PCNAs). A pilot study was conducted to gain insight into the program's feasibility and acceptability, and identify preconditions for successful implementation.

DESIGN AND SETTING

Seven Dutch PC practices participated in a half-year pilot study starting in August 2020. We performed a review of the population segmentation tool, observed four training sessions and 15 PCNAs, and interviewed 15 professionals and 12 patients.

RESULTS

Regarding feasibility and acceptability, we found that the tools and trainings provided professionals with skills to use the segmentation tool and take a more coaching role in the well-appreciated PCNAs. Concerning implementation preconditions, we found that team commitment and network connections need improvement, although work pleasure increased and professionals generally wanted the program to continue.

CONCLUSIONS

While the content of the TARGET program is supported by its users, the implementation process, for instance team commitment to the program, needs more attention in future upscaling efforts.

Development of a Person-Centred Integrated Care Approach for Chronic Disease Management in Dutch Primary Care: A Mixed-Method Study

To reduce the burden of chronic diseases on society and individuals, European countries implemented chronic Disease Management Programs (DMPs) that focus on the management of a single chronic disease. However, due to the fact that the scientific evidence that DMPs reduce the burden of chronic diseases is not convincing, patients with multimorbidity may receive overlapping or conflicting treatment advice, and a single disease approach may be conflicting with the core competencies of primary care. In addition, in the Netherlands, care is shifting from DMPs to person-centred integrated care (PC-IC) approaches. This paper describes a mixed-method development of a PC-IC approach for the management of patients with one or more chronic diseases in Dutch primary care, executed from March 2019 to July 2020. In Phase 1, we conducted a scoping review and document analysis to identify key elements to construct a conceptual model for delivering PC-IC care. In Phase 2, national experts on Diabetes Mellitus type 2, cardiovascular diseases, and chronic obstructive pulmonary disease and local healthcare providers (HCP) commented on the conceptual model using online qualitative surveys. In Phase 3, patients with chronic conditions commented on the conceptual model in individual interviews, and in Phase 4 the conceptual model was presented to the local primary care cooperatives and finalized after processing their comments. Based on the scientific literature, current practice guidelines, and input from a variety of stakeholders, we developed a holistic, person-centred, integrated approach for the management of patients with (multiple) chronic diseases in primary care. Future evaluation of the PC-IC approach will show if this approach leads to more favourable outcomes and should replace the current single-disease approach in the management of chronic conditions and multimorbidity in Dutch primary care.

Conducting focus groups in realist evaluation

Focus groups are valuable tools for evaluators to help stakeholders to clarify programme theories. In 1987, R.K. Merton, often attributed with the birth of focus groups, wrote about how these were 'being mercilessly misused'. In the 1940s, his team had conceived focus groups as tools for developing middle-range theory, but through their astonishing success focus groups have metamorphosed and are often an 'unchallenged' choice in many evaluation approaches, while their practice seems to provide a philosophically diverse picture. This article examines what knowledge focus group data generate, and how they support theory development. It starts with an overview of the history of focus groups, establishing a relationship between their emergence as a data collection method and the evaluation profession. Practical lessons for conducting groups in realist evaluation are suggested, while exploring how qualitative data can support programme and middle-range theory development using the example of realist evaluation.

Unpacking the design, implementation and uptake of community-integrated health care services: a critical realist synthesis

Introduction

Community-integrated care initiatives are increasingly being used for social and health service delivery and show promising outcomes. Nevertheless, it is unclear what structures and underlining causal agents (generative mechanisms) are responsible for explaining how and why they work or not.

Methods and analysis

Critical realist synthesis, a theory-driven approach to reviewing and synthesising literature based on the critical realist philosophy of science, underpinned the study. Two lenses guided our evidence synthesis, the community health system and the patient-focused perspective of integrated care. The realist synthesis was conducted through the following steps: (1) concept mining and framework formulation, (2) searching for and scrutinising the evidence, (3) extracting and synthesising the evidence (4) developing the narratives from causal explanatory theories, and (5) disseminate, implement and evaluate.

Results

Three programme theories, each aligning with three groups of stakeholders, were unearthed. At the systems level, three bundles of mechanisms were identified, that is, (1) commitment and motivation, (2) willingness to address integrated health concerns and (3) shared vision and goals. At the provider level, five bundles of mechanisms critical to the successful implementation of integrated care initiatives were abstracted, that is, (1) shared vision and buy-in, (2) shared learning and empowerment, (3) perceived usefulness, (4) trust and perceived support and (5) perceived role recognition and appreciation. At the user level, five bundles of mechanisms were identified, that is, (1) motivation, (2) perceived interpersonal trust, (3) user-empowerment, (4) perceived accessibility to required services and (5) self-efficacy and self-determination.

Conclusion

We systematically captured mechanism-based explanatory models to inform practice communities on how and why community-integrated models work and under what health systems conditions.

PROSPERO registration number

CRD42020210442.

Supporting care home residents in the last year of life through 'Needs Rounds': Development of a pre-implementation programme theory through a rapid collaborative online approach

Background

Realist evaluation aims to address the knowledge to practice gap by explaining how an intervention is expected to work, as well as what is likely to impact upon the success of its implementation, by developing programme theories that link contexts, mechanisms and outcomes. Co-production approaches to the development of programme theories offer substantial benefits in addressing power relations, including and valuing different types of knowledge, and promoting buy-in from stakeholders while navigating the complex social systems in which innovations are embedded. This paper describes the co-production of an initial programme theory of how an evidence based intervention developed in Australia - called 'Palliative Care Needs Rounds' - might work in England and Scotland to support care home residents approaching their end of life.

Methods

Using realist evaluation and iPARIHS (integrated Promoting Action on Research Implementation in Health Services) we sought to determine how contexts and mechanisms of change might shape implementation outcomes. Pre-intervention online interviews (n = 28) were conducted (February-April 2021), followed by four co-design online workshops with 43 participants (April-June 2021). The online interviews and workshops included a range of stakeholders, including care home staff, specialist palliative care staff, paramedics, general practitioners, and relatives of people living in care homes.

Results

This methodology paper reports developments in realist evaluation and co-production methodologies, and how they were used to develop context, mechanisms, outcomes (CMOs) configurations, and chains of inference. The initial (pre-intervention) programme theory is used to illustrate this process. Two developments to iPARIHS are described. First, involving stakeholders in the collaborative co-design workshops created opportunities to commence facilitation. Second, we describe developing iPARIHS' innovation component, to include novel stakeholder interpretations, perceptions and anticipated use of the intervention as they participated in workshop discussions.

Conclusions

This rapid and robust co-production methodology draws on interactive collaborative research practices (interviews, workshop discussions of data, illustrative vignettes and visual methods). These innovative and engaging methods can be packaged for online processes to develop, describe and interrogate the CMOs in order to co-produce a programme theory. These approaches also commence facilitation and innovation, and can be adopted in other implementation science and realist studies.

The Patient Centered Assessment Method (PCAM) for Action-Based Biopsychosocial Evaluation of Patient Needs: Validation and Perceived Value of the Dutch Translation

The Patient Centered Assessment Method (PCAM) is an action-based tool that supports professionals to engage in a biopsychosocial assessment with patients and measure their needs. It is a promising tool for person-centered care. As the Netherlands lacks such a tool, a Dutch version was developed. Furthermore, we aimed to contribute to the relatively limited insights into the psychometric properties and value of the tool when used as part of a needs assessment in primary care. Confirmatory factor analysis was used to study construct validity and Cronbach’s alpha was computed to assess reliability. Furthermore, we interviewed 15 primary care professionals who used the PCAM. It was confirmed that each PCAM domain measures a separate construct, informed by the biopsychosocial model. The tool showed adequate reliability (Cronbach’s alpha = 0.83). Despite face validity concerns, the tool was mainly valued for measurement of patient needs and to facilitate action planning. Criticism of the PCAM pertained to a limited focus on the patient perspective, which is one of the crucial aspects of person-centered care. These rich, mixed-method insights can help to improve the value of the PCAM, as one of the few multifunctional tools to support professionals in holistic assessments.