Reporting guidelines for realist evaluations seek to improve clarity and transparency

An increasing number of realist evaluations are being conducted from a wide range of disciplinary perspectives and with diverse, fit-for-purpose methods. This commentary discusses the recent BMC Medicine publication of RAMESES II reporting guidelines for realist evaluations. Knowledge users such as program implementers and decision-makers will benefit from the increased transparency of reporting and interpretation in light of the totality of evidence encouraged by this guidance. It is hoped that these reporting guidelines will eventually lead to improved knowledge synthesis and contribute to the cumulative science regarding realist evaluation.Please see related article: https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-016-0643-1.

Can "realist" randomised controlled trials be genuinely realist?

In this paper, we respond to a paper by Jamal and colleagues published in Trials in October 2015 and take an opportunity to continue the much-needed debate about what applied scientific realism is. The paper by Jamal et al. is useful because it exposes the challenges of combining a realist evaluation approach (as developed by Pawson and Tilley) with the randomised controlled trial (RCT) design.

We identified three fundamental differences that are related to paradigmatic differences in the treatment of causation between post-positivist and realist logic: (1) the construct of mechanism, (2) the relation between mediators and moderators on one hand and mechanisms and contexts on the other hand, and (3) the variable-oriented approach to analysis of causation versus the configurational approach.

We show how Jamal et al. consider mechanisms as observable, external treatments and how their approach reduces complex causal processes to variables. We argue that their proposed RCT design cannot provide a truly realist understanding. Not only does the proposed realist RCT design not deal with the RCT’s inherent inability to “unpack” complex interventions, it also does not enable the identification of the dynamic interplay among the intervention, actors, context, mechanisms and outcomes, which is at the core of realist research. As a result, the proposed realist RCT design is not, as we understand it, genuinely realist in nature.

Realism and resources: Towards more explanatory economic evaluation

To be successfully and sustainably adopted, policy-makers, service managers and practitioners want public programmes to be affordable and cost-effective, as well as effective. While the realist evaluation question is often summarised as what works for whom, under what circumstances, we believe the approach can be as salient to answering questions about resource use, costs and cost-effectiveness - the traditional domain of economic evaluation methods. This paper first describes the key similarities and differences between economic evaluation and realist evaluation. It summarises what health economists see as the challenges of evaluating complex interventions, and their suggested solutions. We then use examples of programme theory from a recent realist review of shared care for chronic conditions to illustrate two ways in which realist evaluations might better capture the resource requirements and resource consequences of programmes, and thereby produce explanations of how they are linked to outcomes (i.e. explanations of cost-effectiveness).

RAMESES II reporting standards for realist evaluations

BACKGROUND

Realist evaluation is increasingly used in health services and other fields of research and evaluation. No previous standards exist for reporting realist evaluations. This standard was developed as part of the RAMESES II project. The project's aim is to produce initial reporting standards for realist evaluations.

METHODS

We purposively recruited a maximum variety sample of an international group of experts in realist evaluation to our online Delphi panel. Panel members came from a variety of disciplines, sectors and policy fields. We prepared the briefing materials for our Delphi panel by summarising the most recent literature on realist evaluations to identify how and why rigour had been demonstrated and where gaps in expertise and rigour were evident. We also drew on our collective experience as realist evaluators, in training and supporting realist evaluations, and on the RAMESES email list to help us develop the briefing materials. Through discussion within the project team, we developed a list of issues related to quality that needed to be addressed when carrying out realist evaluations. These were then shared with the panel members and their feedback was sought. Once the panel members had provided their feedback on our briefing materials, we constructed a set of items for potential inclusion in the reporting standards and circulated these online to panel members. Panel members were asked to rank each potential item twice on a 7-point Likert scale, once for relevance and once for validity. They were also encouraged to provide free text comments.

RESULTS

We recruited 35 panel members from 27 organisations across six countries from nine different disciplines. Within three rounds our Delphi panel was able to reach consensus on 20 items that should be included in the reporting standards for realist evaluations. The overall response rates for all items for rounds 1, 2 and 3 were 94 %, 76 % and 80 %, respectively.

CONCLUSION

These reporting standards for realist evaluations have been developed by drawing on a range of sources. We hope that these standards will lead to greater consistency and rigour of reporting and make realist evaluation reports more accessible, usable and helpful to different stakeholders.

Study protocol: realist evaluation of effectiveness and sustainability of a community health workers programme in improving maternal and child health in Nigeria

BACKGROUND

Achievement of improved maternal and child health (MCH) outcomes continues to be an issue of international priority, particularly for sub-Saharan African countries such as Nigeria. Evidence suggests that the use of Community Health Workers (CHWs) can be effective in broadening access to, and coverage of, health services and improving MCH outcomes in such countries.

METHODS/DESIGN

In this paper, we report the methodology for a 5-year study which aims to evaluate the context, processes, outcomes and longer-term sustainability of a Nigerian CHW scheme. Evaluation of complex interventions requires a comprehensive understanding of intervention context, mechanisms and outcomes. The multidisciplinary and mixed-method realist approach will facilitate such evaluation. A favourable policy environment within which the study is conducted will ensure the successful uptake of results into policy and practice. A realist evaluation provides an overall methodological framework for this multidisciplinary and mixed methods research, which will be undertaken in Anambra state. The study will draw upon health economics, social sciences and statistics. The study comprises three steps: (1) initial theory development; (2) theory validation and (3) theory refinement and development of lessons learned. Specific methods for data collection will include in-depth interviews and focus group discussions with purposefully identified key stakeholders (managers, service providers and service users), document reviews, analyses of quantitative data from the CHW programme and health information system, and a small-scale survey. The impact of the programme on key output and outcome indicators will be assessed through an interrupted time-series analysis (ITS) of monthly quantitative data from health information system and programme reports. Ethics approvals for this study were obtained from the University of Leeds and the University of Nigeria.

DISCUSSION

This study will provide a timely and important contribution to health systems strengthening specifically within Anambra state in southeast Nigeria but also more widely across Nigeria. This paper should be of interest to researchers who are interested in adapting and applying robust methodologies for assessing complex health system interventions. The paper will also be useful to policymakers and practitioners who are interested in commissioning and engaging in such complex evaluations to inform policies and practices.

What does theory-driven evaluation add to the analysis of self-reported outcomes of diabetes education? A comparative realist evaluation of a participatory patient education approach

OBJECTIVE

To explore the effects of the Next Education (NEED) patient education approach in diabetes education.

METHODS

We tested the use of the NEED approach at eight intervention sites (n=193). Six additional sites served as controls (n=58). Data were collected through questionnaires, interviews and observations. We analysed data using descriptive statistics, logistic regression and systematic text condensation.

RESULTS

Results from logistic regression demonstrated better overall assessment of education program experiences and enhanced self-reported improvements in maintaining medications correctly among patients from intervention sites, as compared to control sites. Interviews and observations suggested that improvements in health behavior could be explained by mechanisms related to the education setting, including using person-centeredness and dialogue. However, similar mechanisms were observed at control sites. Observations suggested that the quality of group dynamics, patients' motivation and educators' ability to facilitate participation in education, supported by the NEED approach, contributed to better results at intervention sites.

CONCLUSION

The use of participatory approaches and, in particular, the NEED patient education approach in group-based diabetes education improved self-management skills and health behavior outcomes among individuals with diabetes.

PRACTICE IMPLICATIONS

The use of dialogue tools in diabetes education is advised for educators.

Understanding the impact of area-based interventions on area safety in deprived areas: realist evaluation of a neighbour nuisance intervention in Arnhem, the Netherlands

BACKGROUND

Area-based health inequalities may partly be explained by higher levels of area disorder in deprived areas. Area disorder may cause safety concerns and hence impair health. This study assessed how, for whom and in what conditions the intervention Meeting for Care and Nuisance (MCN) had an impact on neighbour nuisance and area safety in four deprived districts in Arnhem, the Netherlands.

METHODS

Realist evaluation methodology was applied to uncover how, for whom, and under what conditions MCN was expected to and actually produced change. Expected change was based on action plans and scientific theories. Actual change was based on progress reports, media articles, interviews with district managers, and quantitative surveys.

RESULTS

Three levels of impact were distinguished. At the organisational level, partly as expected, MCN's coordinated partnership strategy enabled role alignment, communication, and leadership. This resulted in a more efficient approach of nuisance households. At the level of nuisance households, as expected, MCN's joint assistance and enforcement strategy removed many of the underlying reasons for nuisance. This resulted in less neighbour nuisance. At the district level, perceptions of social control and area safety improved only in one district. Key conditions for change included a wider safety approach, dense population, and central location of the district within the city.

CONCLUSIONS

This realist evaluation provided insight into the mechanisms by which a complex area-based intervention was able to reduce neighbour nuisance in deprived areas. Depending on wider conditions, such a reduction in neighbour nuisance may or may not lead to improved perceptions of area safety at the district level.

Explaining how unexploded ordnance clearance enhances livelihoods in the Lao PDR

BACKGROUND

Following violent conflict, the continued presence of landmines and unexploded ordnance pose a barrier to rebuilding livelihoods. Mine action removes these explosive remnants of conflict to enable communities to safely return contaminated land to productive use. There is limited understanding, however, of how, why, in what context and in what respects mine action contributes to livelihoods. Yet, such information is required for effective resource allocation, checking underlying program assumptions, understanding benefits and potential harms.

METHODS

The evaluation was undertaken in the Lao People's Democratic Republic. It used an interpretive case study design and applied the principles of realist evaluation. Program staff and local government authorities were interviewed (N=37) and program beneficiaries. In total, 38 individual interviews with program beneficiaries were conducted and eighteen focus group interviews (9 with males, 9 with females), each with 6-9 participants.

RESULTS

The evaluation identified two main mechanisms through which the program 'worked': (1) communication pre- and post-clearance and (2) the delivery of the product (cleared land).

CONCLUSION

The realist approach helped to refine the program theory, highlighted the role of self- and task-efficacy and community communication, assisted in identifying contextual factors that influence outcomes and suggested a revision of expected outcomes.

How shall we examine and learn about public-private partnerships (PPPs) in the health sector? Realist evaluation of PPPs in Hong Kong

The World Health Organization advocates the goal of universal coverage of health systems to ensure that everyone can avail the services they need and are protected from the associated financial risks. Governments are increasingly engaging and interacting with the private sector in initiatives collectively referred to as public-private partnerships (PPPs) to enhance the capacity of health systems to meet this objective. Understanding the values that motivate partners and demonstrating commitment for building relationships were found to be key lessons in building effective PPPs; however there, remain many research gaps. This study focusses on the practice of PPPs at the inter-organisational (meso) level and interpersonal (micro) level in Hong Kong Special Administrative Region (HKSAR). The influence of the structural components of different PPPs on stakeholder interpretation and actions, as well as the eventual outcomes of the PPPs, is examined, in terms of a realist evaluation, which applies a context-mechanism-outcome configuration as the research methodology. Seven key factors initiating commitment in a partnership, critical for sustainable PPPs, were identified as follows: (1) building of trust; (2) clearly defined objectives and roles; (3) time commitment; (4) transparency and candid information, particularly in relation to risk and benefit; (5) contract flexibility; (6) technical assistance or financial incentive behind procedural arrangements; and (7) the awareness and acceptability of structural changes related to responsibility and decisions (power and authority).

A realist evaluation of an antenatal programme to change drinking behaviour of pregnant women

OBJECTIVE

to use realist evaluation to describe and explain how and in what circumstances screening and alcohol brief interventions work in routine antenatal care.

DESIGN

a realist evaluation incorporating systematic reviews and qualitative data.

SETTING

NHS Lothian, which is one of the 14 Scottish health boards.

PARTICIPANTS

participants were recruited from two maternity units. In phase one, interviews were conducted with four participants responsible for policy implementation. These data were supported by two systematic reviews. In phase two, 17 pregnant women and 15 midwives participated in interviews, with a further six midwifery team leaders involved in a focus group.

FINDINGS

training and resources provided to midwives as part of the programme acted as facilitating mechanisms that improved their skills and confidence to screen and deliver alcohol brief interventions. The programme elicited positive change in attitudes to drinking in pregnancy and possibly stimulated drinking behaviour change amongst pregnant women. However, the small numbers of pregnant women being identified for alcohol brief interventions meant delivery was infrequent and resulted in the programme not working as anticipated. The findings also revealed contextual issues around midwife-pregnant woman relationship and the challenges of negotiating the timing of screening and alcohol brief interventions delivery.

CONCLUSIONS

Drinking in pregnancy is an emotive issue, therefore delivering alcohol brief interventions at the first antenatal appointment when they are more likely to achieve the most benefits poses challenges. When training midwives to screen and deliver alcohol brief interventions, special attention is needed to improve person-centred communication skills to overcome barriers associated with discussing sensitive prenatal alcohol use and enhance early identification and delivery of alcohol brief interventions at the first antenatal appointment.

'Oh no, no, no, we haven׳t got time to be doing that': Challenges encountered introducing a breast-feeding support intervention on a postnatal ward

OBJECTIVE

to identify elements in the environment of a postnatal ward which impacted on the introduction of a breast-feeding support intervention.

DESIGN

a concurrent, realist evaluation including practice observations and semi-structured interviews.

SETTING

a typical British maternity ward.

PARTICIPANTS

five midwives and two maternity support workers were observed. Seven midwives and three maternity support workers were interviewed. Informed consent was obtained from all participants. Ethical approval was granted by the relevant authorities.

FINDINGS

a high level of non-compliance with the intervention was driven by a lack of time and staff, and the ward staffs׳ lack of control of the organisation of their time and space. This was compounded by a propensity towards task orientation, workload reduction and resistance to change - all of which supported the existing medical approach to care. Limited support for the intervention was underpinned by staff willingness to reconsider their views and a widespread frustration with current ways of working.

KEY CONCLUSIONS

this small, local study suggests that the environment and working conditions on a typical British postnatal ward present significant barriers to the introduction of breast-feeding support interventions requiring a relational approach to care.

IMPLICATIONS FOR PRACTICE

midwives and maternity support workers need to be able to control their time and space, and feel able to provide the relational care they perceive that women need, before breast-feeding support interventions can be successfully implemented in practice. Frustration with current ways of working, and a willingness to consider other approaches, could be harnessed to initiate change that would benefit health professionals and the women and families in their care. However, without appropriate leadership or facilitation for change, this could alternatively encourage learned helplessness and passive resistance.

General practitioners' management of the long-term sick role

In this paper, we use qualitative research techniques to examine the role of general practitioners in the management of the long-term sickness absence. In order to uncover the perspectives of all the main agents affected by the actions of general practitioners, a case study approach focussing on one particular employment sector, the public health service, is adopted. The role of family physicians is viewed from the perspectives of health service managers, occupational health physicians, employees/patients, and general practitioners. Our argument is theoretically framed by Talcott Parsons's model of the medical contribution to the sick role, along with subsequent conceptualisations of the social role and position of physicians. Sixty one semi-structured interviews and three focus group interviews were conducted in three Health and Social Care Trusts in Northern Ireland between 2010 and 2012. There was a consensus among respondents that general practitioners put far more weight on the preferences and needs of their patients than they did on the requirements of employing organisations. This was explained by respondents in terms of the propinquity and longevity of relationships between doctors and their patients, and by the ideology of holistic care and patient advocacy that general practitioners viewed as providing the foundations of their approach to patients. The approach of general practitioners was viewed negatively by managers and occupational health physicians, and more positively by general practitioners and patients. However, there is some evidence that general practitioners would be prepared to forfeit their role as validators of sick leave. Given the imperatives of both state and capital to reduce the financial burden of long-term sickness, this preparedness puts into doubt the continued role of general practitioners as gatekeepers to legitimate long-term sickness absence.

Empowering change: realist evaluation of a Scottish Government programme to support normal birth

BACKGROUND

midwife-led care has consistently been found to be safe and effective in reducing routine childbirth interventions and improving women's experience of care. Despite consistent UK policy support for maximising the role of the midwife as the lead care provider for women with healthy pregnancies, implementation has been inconsistent and the persistent use of routine interventions in labour has given rise to concern. In response the Scottish Government initiated Keeping Childbirth Natural and Dynamic (KCND), a maternity care programme that aimed to support normal birth by implementing multiprofessional care pathways and making midwife-led care for healthy pregnant women the national norm.

AIM

the evaluation was informed by realist evaluation. It aimed to explore and explain the ways in which the KCND programme worked or did not work in different maternity care contexts.

METHODS

the evaluation was conducted in three phases. In phase one semi-structured interviews and focus groups were conducted with key informants to elicit the programme theory. At phase two, this theory was tested using a multiple case study approach. Semi-structured interviews and focus groups were conducted and a case record audit was undertaken. In the final phase the programme theory was refined through analyses and interpretation of the data.

SETTING AND PARTICIPANTS

the setting for the evaluation was NHS Scotland. In phase one, 12 national programme stakeholders and 13 consultant midwives participated. In phase two case studies were undertaken in three health boards; overall 73 participants took part in interviews or focus groups. A case record audit was undertaken of all births in Scotland during one week in two consecutive years before and after pathway implementation.

FINDINGS

government and health board level commitment to, and support of, the programme signalled its importance and facilitated change. Consultant midwives tailored change strategies, using different approaches in response to the culture of care and inter-professional relationships within contexts. In contexts where practice was already changing KCND was seen as validating and facilitating. In areas where a more medical culture existed there was strong resistance to change from midwives and medical staff and robust implementation strategies were required. Overall the pathways appeared to enable midwives to achieve change.

KEY CONCLUSIONS

our study highlighted the importance of those involved in a change programme working across levels of hierarchy within an organisation and from the macro-context of national policy and institutions to the meso-context of regional health service delivery and the micro-context of practitioner's experiences of providing care. The assumptions and propositions that inform programmes of change, which are often left at a tacit level and unexamined by those charged with implementing them, were made explicit. This examination illuminated the roles of the three key change mechanisms adopted in the KCND programme - appointment of consultant midwives as programme champions, multidisciplinary care pathways, and midwife-led care. It revealed the role of the commitment mechanism, which built on the appointment of the local change champions. The analysis indicated that the process of change, despite these clear mechanisms, needed to be adapted to local contexts and responses to the implementation of KCND.

IMPLICATIONS FOR PRACTICE

initial formative evaluation should be conducted prior to development of complex healthcare programmes to ensure that (1) the interventions will address the changes required, (2) key stakeholders who may support or resist change are identified, and (3) appropriate facilitation strategies are developed tailored to context.