Realist evaluation: an immanent critique

The Delphi of ORACLE: An Expert Consensus Survey for the Development of the Observational Risk Assessment of Contractures (Longitudinal Evaluation)

OBJECTIVE

Despite rising prevalence rates, no standard tool is available to identify individuals at risk of developing contractures. This study aimed to gain expert consensus on items for the development of the Observational Risk Assessment Tool for Contractures: Longitudinal Evaluation (ORACLE) for care home residents.

DESIGN

A two-round, online modified Delphi study.

PARTICIPANTS

Panellists were qualified healthcare professionals with a background in physiotherapy, occupational therapy, nursing, and rehabilitation medicine.

MAIN OUTCOME MEASURES

In the first round, the experts were asked to rate the predesigned list of items on a Likert scale while in the second round, consensus was sought in the areas of disagreement identified in the previous round.

RESULTS

The two rounds of the Delphi survey included 30 and 25 panellists, respectively. The average clinical and academic experience of the panellists was 22.2 years and 10.5 years, respectively. The panel demonstrated a high level of consensus regarding the clinical factors (10 out of 15 items); preventive care approaches (9 out of 10 items), and contextual factors (12 out of 13 items) ranging from 70% to 100%.

CONCLUSION

This Delphi study determined expert consensus on items to be included in a contracture risk assessment tool (ORACLE). The items were related to factors associated with joint contractures, appropriate preventive care interventions, and potentially relevant contextual factors associated with care home settings. The promise of a risk assessment tool that includes these items has the capacity to reduce the risk of contracture development or progression and to trigger timely and appropriate referrals to help prevent further loss of function and independence.

Sharing is Caring: A Realist Evaluation of a Social Support Group for Individuals Who Have Been Bereaved by Suicide

AIM

To understand the experiences and perceived impact on the wellbeing of individuals attending a suicide bereavement social support group.

DESIGN

A qualitative study guided by a realist evaluation framework.

METHODS

Data were collected from May-July 2020 using online semistructured individual interviews with participants (N = 6), from the North West of England recruited from a suicide bereavement support group's social media. Data were analysed using thematic analysis informed by the realist framework.

RESULTS

Effective social support includes the prioritisation of building meaningful connections with like-minded individuals, providing a safe space for authentic self-expression aiding personal relationship maintenance. Contextual factors included: Societal and cultural stigma of suicide, self-stigma and gender norms. Mechanisms influencing support seeking include: Not wanting to burden loved ones due to judgement, and a lack of understanding.

IMPACT

Policymakers can reduce demand on healthcare systems by developing tailored support groups to suit individual needs.

A protocol for a critical realist synthesis of school mindfulness interventions designed to promote pupils' mental wellbeing

Introduction

The review described in this protocol will be the first critical realist review of the literature reporting on the impact of school-based mindfulness interventions on the mental wellbeing of pupils. Mindfulness interventions are increasingly being introduced into schools to promote children's (and teachers') wellbeing. Findings from impact evaluations, including systematic reviews and metanalysis, suggest that school-based mindfulness interventions promote pupils' wellbeing. However, there is a need for further evidence on the underlying causal mechanisms and contexts that explain program outcomes, to provide insight into how mindfulness programs can be successfully implemented in other contexts.

Methods and analysis

A critical realist review methodology will be used to provide a causal interdisciplinary understanding of how school-based mindfulness interventions promote the mental wellbeing of pupils. This will be done through a systematic literature review and extrapolating context, agency, intervention, mechanisms, and outcome configurations. This will enable an understanding of how, in certain contexts, pupils can use the resources offered by a mindfulness intervention knowingly or unknowingly to trigger mechanisms that promote their mental wellbeing and what mechanisms in the context support, restrict or prevent change. We will then use retrodiction and retroduction to develop the most plausible interdisciplinary middle-range theory to explain the findings.

Discussion

The review findings will inform a critical realist evaluation of a mindfulness intervention in schools. The findings from the review will also enable us to inform policymakers and other stakeholders about what conditions need to be in place for mindfulness interventions to promote pupils' mental wellbeing. We will publish the findings from the review in academic and professional publications, policy briefs, workshops, conferences, and social media.PROSPERO registration number: CRD42023410484.

Common health assets protocol: a mixed-methods, realist evaluation and economic appraisal of how community-led organisations (CLOs) impact on the health and well-being of people living in deprived areas

INTRODUCTION

This research investigates how community-led organisations' (CLOs') use of assets-based approaches improves health and well-being, and how that might be different in different contexts. Assets-based approaches involve 'doing with' rather than 'doing to' and bring people in communities together to achieve positive change using their own knowledge, skills and experience. Some studies have shown that such approaches can have a positive effect on health and well-being. However, research is limited, and we know little about which approaches lead to which outcomes and how different contexts might affect success.

METHODS AND ANALYSIS

Using a realist approach, we will work with 15 CLOs based in disadvantaged communities in England, Scotland and Northern Ireland. A realist synthesis of review papers, and a policy analysis in different contexts, precedes qualitative interviews and workshops with stakeholders, to find out how CLOs' programmes work and identify existing data. We will explore participants' experiences through: a Q methodology study; participatory photography workshops; qualitative interviews and measure outcomes using a longitudinal survey, with 225 CLO participants, to assess impact for people who connect with the CLOs. An economic analysis will estimate costs and benefits to participants, for different contexts and mechanisms. A 'Lived Experience Panel' of people connected with our CLOs as participants or volunteers, will ensure the appropriateness of the research, interpretation and reporting of findings.

ETHICS AND DISSEMINATION

This project, research tools and consent processes have been approved by the Glasgow Caledonian University School of Health and Life Sciences Ethics Committee, and affirmed by Ethics Committees at Bournemouth University, Queen's University Belfast and the University of East London. Common Health Assets does not involve any National Health Service sites, staff or patients.Findings will be presented through social media, project website, blogs, policy briefings, journal articles, conferences and visually in short digital stories, and photographic exhibitions.

Combining realist evaluation and transformative evaluation to advance research in palliative care: The case of end of life companionship

BACKGROUND

Palliative care requires innovative methods to understand what works, for whom, in what circumstances and why. Realist evaluation has become one prominent approach due to its preoccupation with building, and testing, causal theories to explain the influence of contextual factors on outcomes. Undertaking realist evaluation is not without challenges and may amplify issues of underrepresentation, disempower those working in palliative care, and produce results with poor ecological validity. Complementary approaches are needed which mitigate these challenges, whilst producing credible findings that advances knowledge.

PURPOSE

In this article it is outlined how realist evaluation provides a toolkit to advance research to explain, and empirically test, the complex contours of palliative care. Moreover, it is proposed that transformative evaluation can provide a catalyst to engage and empower those within palliative care, create the opportunity for care transformation, and produce more informed and authentic theories.

DISCUSSION

Contemporary issues in palliative care pertain to the complexity of palliative care, the insufficiency of experimental designs alone, and the challenges of achieving inclusive research participation. In this article it is argued that theory led, participatory, opportunistic and naturalistic approaches can provide an antidote to the issues in the literature. The combination also mitigates many methodological critiques of the individual approaches, by increasing the transformative potential of realist evaluation, and explanatory potential of transformative evaluation.

Integrated Child and Family Hub models for detecting and responding to family adversity: protocol for a mixed-methods evaluation in two sites

INTRODUCTION

Integrated community healthcare Hubs may offer a 'one stop shop' for service users with complex health and social needs, and more efficiently use service resources. Various policy imperatives exist to implement Hub models of care, however, there is a dearth of research specifically evaluating Hubs targeted at families experiencing adversity. To contribute to building this evidence, we propose to co-design, test and evaluate integrated Hub models of care in two Australian community health services in low socioeconomic areas that serve families experiencing adversity: Wyndham Vale in Victoria and Marrickville in New South Wales.

METHODS AND ANALYSIS

This multisite convergent mixed-methods study will run over three phases to (1) develop the initial Hub programme theory through formative research; (2) test and, then, (3) refine the Hub theory using empirical data. Phase 1 involves co-design of each Hub with caregivers, community members and practitioners. Phase 2 uses caregiver and Hub practitioner surveys at baseline, and 6 and 12 months after Hub implementation, and in-depth interviews at 12 months. Two stakeholder groups will be recruited: caregivers (n=100-200 per site) and Hub practitioners (n=20-30 per site). The intervention is a co-located Hub providing health, social, legal and community services with no comparator. The primary outcomes are caregiver-reported: (i) identification of, (ii) interventions received and/or (iii) referrals received for adversity from Hub practitioners. The study also assesses child, caregiver, practitioner and system outcomes including mental health, parenting, quality of life, care experience and service linkages. Primary and secondary outcomes will be assessed by examining change in proportions/means from baseline to 6 months, from 6 to 12 months and from baseline to 12 months. Service linkages will be analysed using social network analysis. Costs of Hub implementation and a health economics analysis of unmet need will be conducted. Thematic analysis will be employed to analyse qualitative data.

ETHICS AND DISSEMINATION

Royal Children's Hospital and Sydney Local Health District ethics committees have approved the study (HREC/62866/RCHM-2020). Participants and stakeholders will receive results through meetings, presentations and publications.

TRIAL REGISTRATION NUMBER

ISRCTN55495932.

A critical realist evaluation of advance care planning in care homes

AIMS

To evaluate care planning in advance of end-of-life care in care homes.

DESIGN

A qualitative study.

METHODS

Qualitative data were collected from January 2018-July 2019 (using focus groups and semi-structured interviews) from three care homes in the South West of England. The data were analysed using thematic analysis followed by Critical Realist Evaluation.

RESULTS

Participants comprised of registered nurses (N = 4), care assistants (N = 8), bereaved relatives (N = 7), and domiciliary staff (N = 3). Although the importance of advance care planning was well recognized, the emotional labour of frequently engaging in discussions about death and dying was highlighted as a problem by some care home staff. It was evident that in some cases care home staff's unmet emotional needs led them to rushing and avoiding discussions about death and dying with residents and relatives. A sparsity of mechanisms to support care home staff's emotional needs was noted across all three care homes. Furthermore, a lack of training and knowledge appeared to inhibit care home staff's ability to engage in meaningful care planning conversations with specific groups of residents such as those living with dementia. The lack of training was principally evident amongst non-registered care home staff and those with non-formal caring roles such as housekeeping.

CONCLUSION

There is a need for more focused education to support registered and non-registered care home staff to effectively engage in sensitive discussions about death and dying with residents. Furthermore, greater emotional support is necessary to help build workforce resilience and sustain change.

IMPACT

Knowledge generated from this study can be used to inform the design and development of future advance care planning interventions capable of supporting the delivery of high-quality end-of-life care in care homes.

The role of the advanced clinical practitioner in breast diagnosis: A systematic review of the literature

INTRODUCTION

Increasing prevalence in breast cancers, workforce shortages and technological advancements have increased the need to further develop advanced practice in breast diagnosis. The Advanced Clinical Practitioner training programme has been introduced to support this need. The aim of this work was to systematically review studies that explore advanced practice in mammography to assess the potential impact of the introduction of a specific Advanced Clinical Practitioner training programme in breast diagnosis within the UK.

METHODS

A systematic PRISMA review of the literature published between 1999 and January 2020 was carried out. A total of 17 studies were included in the review.

RESULTS

Four themes were identified in the literature in relation to advanced practice in breast imaging: multidisciplinary practice; roles and responsibilities associated with advanced practice; development and progression; embedding and sustaining advanced practice. It was evident across all themes that advanced practice is vital in supporting better care for patients attending breast imaging in light of workforce shortages. Although advanced practice and its benefits are well established in breast imaging, persistent barriers were acknowledged such as role ambiguity, recruitment issues, lack of support from some radiologists and poor funding.

CONCLUSION

Findings suggest that introducing a more formalised pathway to advanced practice into breast imaging through the implementation of a specific Advanced Clinical Practitioner apprenticeship training programme may overcome many of the challenges evidenced in this review.

IMPLICATIONS FOR PRACTICE

The findings of this review will help inform the development of the Advanced Clinical Practitioner apprenticeship programme specific to breast diagnosis.

What works, for whom and under what circumstances? Using realist methodology to evaluate complex interventions in nursing: A scoping review

BACKGROUND

The Medical Research Council's (MRC) framework recommends basing every evaluation of a complex intervention on a theory that explains WHY an intervention works instead of focusing exclusively on the identification IF an intervention works. Theory-based evaluation approaches such as realist evaluations are promising in this respect. The aim of this scoping review is to examine current scientific literature with regard to the use of realist methodology in studies that evaluate complex nursing interventions. We also seek to answer the question whether realist evaluation studies of complex nursing interventions are embedded in the Medical Research Council's framework.

METHODS

A systematic database search in MEDLINE and CINAHL supplemented with snowballing techniques and a hand search in relevant nursing journals were performed between January and February 2018 and updated in July 2019. Both authors independently screened the title / abstract of studies that were initially identified and appraised the full texts if inclusion criteria were fulfilled. Inclusion criteria were the explicit use of realist methodology and a focus on the development or evaluation of complex nursing interventions. Results were summarized narratively.

RESULTS

A total of 28 articles from 22 studies, published between 2010 and 2019, were included. The majority of the studies included originate in the UK. Realist methodology was used by a variety of nursing disciplines, including midwifery, community / public health, critical care, palliative care, gerontological nursing, psychiatric nursing / mental health, and infection control. Study designs included case studies and realist reviews; two studies were conducted alongside a randomized controlled trial. All studies included used more than one method to develop their results; the majority of methods were qualitative by nature. Thirteen of the studies included aimed to develop a theory about what works, for whom and under what circumstances regarding the investigated complex nursing intervention. One study was clearly embedded in the Medical Research Council's framework.

CONCLUSION

Realist evaluation broadens the methodologies that have traditionally been used in nursing. Conceptual clarity on its principal tenets is still needed, as well as the methods used to assess them. Nursing scientists should be more critical when using novel methodology and contribute to academic debates about limitations. Embedding realist evaluations into the Medical Research Council's framework is in its infancy in nursing and needs more contributions in order to develop a common academic position within the profession. Tweetable abstract Realist methodology in nursing - review of the use, methods, limitations and embedding into the Medical Research Council's framework.

Systematic critical realist review of interventions designed to improve end-of-life care in care homes

The demand for high-quality end-of-life care is rising. Frequently evidenced concerns about the provision of end-of-life in care homes relate to inter-disciplinary communication and engagement in advance care planning. A number of interventions employing different mechanisms have been designed to address these issues. Therefore, the aim of this systematic critical realist review was to describe and explain the effectiveness of interventions designed to improve end-of-life care in care homes. Electronic searches were conducted in ScienceDirect, MEDLINE, PubMed, PsychINFO, and CINAHL from January 2000 to August 2018. Forty one studies were included in the review. While most of the evidence identified in this review was not strong, there was evidence to suggest that education and inter-professional collaboration can be effective intervention mechanisms for improving end-of-life care in care homes. High staff turnover was a significant contextual mechanism impacting on the sustainability of interventions. In terms of human agency, it is important to note a consistent finding related to the dedication and enthusiasm of care home staff who deliver end-of-life care.

Developing programme theories as part of a realist evaluation of a healthcare quality improvement programme

The application of realist approaches to health services research and evaluation has continued to grow over the past two decades. However, difficulties in defining and operationalising key realist concepts of contexts, mechanisms and outcomes in healthcare settings continue to be recognised within the realist evaluation literature. Reflecting on an ongoing realist evaluation of a healthcare quality improvement programme across an inter-organisational context, this article explores some of the methodological challenges encountered by the author in the early stages of programme theory development. An individualised operationalisation and application of realist concepts are presented to demonstrate how initial programme theories can be developed despite the methodological difficulties presented.

A critical realist evaluation of a music therapy intervention in palliative care

BACKGROUND

Music therapy is increasingly used as an adjunct therapy to support symptom management in palliative care. However, studies to date have paid little attention to the processes that lead to changes in patient outcomes. To fill this gap, we examined the processes and experiences involved in the introduction of music therapy as an adjunct complementary therapy to palliative care in a hospice setting in the United Kingdom (UK).

METHODS

Using a realistic evaluation approach, we conducted a qualitative study using a variety of approaches. These consisted of open text answers from patients (n = 16) on how music therapy helped meet their needs within one hospice in Northern Ireland, UK. We also conducted three focus groups with a range of palliative care practitioners (seven physicians, seven nursing staff, two social workers and three allied health professionals) to help understand their perspectives on music therapy's impact on their work setting, and what influences its successful implementation. This was supplemented with an interview with the music therapist delivering the intervention.

RESULTS

Music therapy contains multiple mechanisms that can provide physical, psychological, emotional, expressive, existential and social support. There is also evidence that the hospice context, animated by a holistic approach to healthcare, is an important facilitator of the effects of music therapy. Examination of patients' responses helped identify specific benefits for different types of patients.

CONCLUSIONS

There is a synergy between the therapeutic aims of music therapy and those of palliative care, which appealed to a significant proportion of participants, who perceived it as effective.

Co-located welfare advice in general practice: A realist qualitative study

General practitioners (GPs) engage with patients about a variety of social issues distinct from direct clinical work ("non-health" issues), such as health-related benefits and debt. Co-located welfare advice services could provide support to practices but have usually been considered in terms of patient rather than practice outcomes. We aimed to develop an initial programme theory for how the provision of co-located advice supports specific practice outcomes, and to identify salient barriers and enabling factors. Twenty-four semi-structured interviews with general practice staff, advice staff and service funders in two UK urban localities were conducted between January and July 2016. Data were thematically analysed and a modified Realist Evaluation approach informed the topic guide, thematic analysis and interpretation. Two outcomes are described linked to participant accounts of the impact of such non-health work on practices: reduction of GP consultations linked to non-health issues and reduced practice time spent on non-health issues. We found that individual responses and actions influencing service awareness were key facilitators to each of the practice outcomes, including proactive engagement, communication, regular reminders and feedback between advice staff, practice managers and funders. Facilitating implementation factors were: not limiting access to GP referral, and offering booked appointments and advice on a broader range of issues responsive to local need. Key barriers included pre-existing sociocultural and organisational rules and norms largely outside of the control of service implementers, which maintained perceptions of the GP as the "go-to-location". We conclude that co-location of welfare advice services alone is unlikely to enable positive outcomes for practices and suggest several factors amenable to intervention that could enhance the potential for co-location to meet desired objectives.

The possibility of critical realist randomised controlled trials

Background

Some realists have criticised randomised controlled trials for their inability to explain the causal relations that they identify; to take into account the influence of the social context of the interventions they evaluate; and to account for individual difference. However, among realists, there is controversy over whether it is possible to improve trials by making them realist, or whether realism and the philosophical assumptions underlying trials are incompatible. This paper contributes to the debate in Trials on this issue. The debate thus far has concentrated on the possibility of combining trial methodology with that of realist evaluation.

Main body

We concur with the contention that it is not feasible to combine randomised controlled trial design with the realist evaluation approach. However, we argue that a different variant of realism, critical realism, provides a more appropriate theoretical grounding for realist trials.

In contrast to realist evaluation, which regards social mechanisms as an amalgam of social resources and people’s reasoning, critical realism insists on their distinction. It does so on the basis of its assertion of the need to distinguish between social structures (in which resources lie) and human agency (which is at least partly guided by reasoning). From this perspective, conceiving of social mechanisms as external to participants can be seen as a valid methodological strategy for supplementing the exclusive concentration of trials on outcomes.

While accepting realist evaluation’s insistence that causality in open systems involves a configuration of multiple generative mechanisms, we adopt the critical realist interpretation of the experimental method, which sees it as creating artificial closure in order to identify the effects of specific causal mechanisms. If randomised controlled trials can be regarded as epidemiological proxies that substitute probabilistic controls over extraneous factors for closed experiments, their examination of the powers of discrete mechanisms through observation of the variation of outcomes is appropriate.

Conclusion

While there are still issues to be resolved, critical realist randomised controlled trials are possible and have the potential to overcome some of the difficulties faced by traditional trial designs in accounting for the influence of social context and individual interpretation.

Great Expectations: The Implementation of Integrated Care and Its Contribution to Improved Outcomes for People with Chronic Conditions

There are great expectations regarding the potential contribution of integrated care interventions to improved outcomes, but so far the evidence is mixed. In this dissertation, we focussed on why, when and how some integrated care interventions contribute to improved outcomes, while others do not. To this purpose, we developed the COMIC Model for studying the Context, Outcomes and Mechanisms of Integrated Care interventions. Evaluations that make use of the COMIC Model take into account the context in which an intervention is implemented and can thereby provide insights into why an intervention does (not) work and how the intervention and/or the context can be changed to achieve improved outcomes.

Music therapy for palliative care: A realist review

OBJECTIVE

Music therapy has experienced a rising demand as an adjunct therapy for symptom management among palliative care patients. We conducted a realist review of the literature to develop a greater understanding of how music therapy might benefit palliative care patients and the contextual mechanisms that promote or inhibit its successful implementation.

METHOD

We searched electronic databases (CINAHL, Embase, Medline, and PsychINFO) for literature containing information on music therapy for palliative care. In keeping with the realist approach, we examined all relevant literature to develop theories that could explain how music therapy works.

RESULTS

A total of 51 articles were included in the review. Music therapy was found to have a therapeutic effect on the physical, psychological, emotional, and spiritual suffering of palliative care patients. We also identified program mechanisms that help explain music therapy's therapeutic effects, along with facilitating contexts for implementation.

SIGNIFICANCE OF RESULTS

Music therapy may be an effective nonpharmacological approach to managing distressing symptoms in palliative care patients. The findings also suggest that group music therapy may be a cost-efficient and effective way to support staff caring for palliative care patients. We encourage others to continue developing the evidence base in order to expand our understanding of how music therapy works, with the aim of informing and improving the provision of music therapy for palliative care patients.

Development of the COMIC Model for the comprehensive evaluation of integrated care interventions

Objective

To develop a model for the comprehensive evaluation of integrated care interventions that provides insights into when, why and how successful outcomes can be achieved.

Methods

A preliminary model was developed based on the Context + Mechanism + Outcome Model and further developed based on its application to a literature review, two case studies and an expert questionnaire. The COMIC Model for studying the Context, Outcomes and Mechanisms of Integrated Care interventions interventions assumes that an intervention is introduced using certain mechanisms (categorised according to the Chronic Care Model), which are met with particular context factors (described by barriers and facilitators and categorised according to the Implementation Model), which combined, contribute to specific outcomes (categorised by the WHO dimensions of quality of care).

Results

Application of the COMIC model to the literature review and expert questionnaires did not allow for statements to be made about the relationships between mechanisms, context and outcomes. Application to the two case studies made it possible to (1) comprehensively analyse the mechanisms, context and outcomes of the specific case, (2) to make the relationships between the mechanisms, context and outcomes within each case visible, and (3) to compare the two cases to each other in a systematic way that added value to the analysis.

Discussion

Using the COMIC Model makes it possible to comprehensively study the interplay of the mechanisms, context and outcomes of integrated care interventions and thereby provides insights into when, why and how integrated care contributes to improved outcomes.