Realist evaluation of UK medical education quality assurance

Service Providers' Perspectives of an Integrated Community Mental Health Service in the UK

This qualitative service evaluation aims to provide in-depth insights of service providers' perspectives of a new integrated community mental health service piloted in one NHS Integrated Care Board locality in South West England, UK, considering to what extent the service is meeting the mental health support needs of adults who are in between primary and secondary care services. In total, 21 semi-structured remote interviews were carried out in June-August 2023 with service providers and lived experience representatives. The evaluation was carried out through a researcher in residence -placement. Qualitative analysis drew on framework and thematic analysis, which was completed using Nvivo 20. Thematic analysis drew on a framework of context, mechanisms and intended or unintended consequences. These showed that service providers' positive expectations of the service were undermined by insufficient integration, which was evident from the imbalances in information flow and presence of pre-existing provider specific practices. The evaluation found several improvement opportunities: a need for deepening integration beyond the initial service development phase; aligning working practices with service delivery aims and ensuring that new services are not rolled out prematurely before service delivery practices have been fully developed.

Accreditation Approaches for Professional Education Programs: Toward Best Practice

Accreditation has a central role in the quality assurance of professional education programs, but research on the effectiveness of different models of accreditation is limited. The purpose of this study was to rapidly appraise the evidence for the effectiveness, impact, and feasibility of different accreditation approaches, in order to inform best practices for the accreditation of professional education programs. The study focused on accreditation for programs that produce practice-ready graduates, including for veterinary programs. The authors searched several databases for articles published from 2000 to 2020, using search terms identified during a scoping phase, and applied a "rapid review" methodology in line with contextual, time, and resource requirements. Relevant articles that were classed as empirical or conceptual were included in the study, while papers appraised as solely commentaries or descriptive were excluded from the evidence base. The full-text review included 32 articles. We identified a clear transition in the literature from input- and process-based models (pre- and early 2000s) to outcomes-based models (in the 2000s and early 2010s). Continuous quality improvement and targeted models (including risk-based and thematic) represent more recent approaches in accreditation practice. However, as noted by other scholars, we identified limited empirical evidence for the relative effectiveness of different accreditation approaches in professional education, although evidence for the more recent accreditation approaches is emerging. In terms of best practice in view of the current lack of definitive evidence for the adoption of any specific model of accreditation, we argue that accrediting authorities adopt a contextual approach to accreditation that includes clearly articulating the purpose and focus of their regulatory activities, and selecting and implementing accreditation methods that are consistent with their underlying principles.

What Really Matters for Supervision Training Workshops? A Realist Evaluation

PURPOSE

Supervision training supports health care supervisors to perform their essential functions. Realist evaluations are increasingly popular for evaluating complex educational interventions, but no such evaluations exist appraising supervision workshops. Building on an earlier realist synthesis of supervision training, the authors evaluated whether supervision workshops work, for whom and under what circumstances, and why.

METHOD

The authors conducted a 2-stage realist evaluation during 2018-2019 to refine and develop program theory. The intervention involved half-day, face-to-face supervision workshops as part of an Australian state-wide government-funded program for health care and human services supervisors. Data collection involved realist interviews with 10 workshop developers (stage 1) and 43 supervisors (stage 2). The authors employed team-based data analysis using realist logic to refine and develop program theory by identifying contexts, mechanisms, outcomes, and context-mechanism-outcome configurations.

RESULTS

Despite their brevity, the supervision workshops had many reported benefits for supervisors (e.g., improved satisfaction) through various perceived mechanisms pertaining to pedagogy (e.g., mixed pedagogies), workshops (e.g., optimal duration), and individuals (e.g., supervisor engagement). However, they also yielded negative reported outcomes (e.g., suboptimal knowledge gains) brought about by assorted perceived mechanisms related to pedagogy (e.g., suboptimal peer learning), workshops (e.g., content irrelevance), and individuals (e.g., suboptimal facilitator competence). Such mechanisms were thought to be triggered by diverse contexts including supervisors' levels of experience, sector, and workplace supervision cultures.

CONCLUSIONS

While the findings partly support the realist synthesis of supervision training and previous realist evaluations of faculty development, this realist evaluation extends this literature considerably. Health care educators should employ mixed pedagogies (e.g., didactic teaching, peer learning), relevant content, optimal workshop duration, and competent/engaging facilitators. Educators also need to tailor workshops according to supervisors' contexts including the sectors and supervision cultures in which supervision is practiced, and supervisors' levels of experience (e.g., experienced supervisors appreciated workshop brevity).

Understanding impacts of accreditation on medical teachers and students: A systematic review and meta-ethnography

PURPOSE

Accreditation is widely used by medical schools around the word to evaluate their curricula and educational processes, although its impacts on those involved in the 'frontline' of medical education receive little attention. This study systematically identified and synthesised qualitative studies that have explored medical teachers' and students' experiences of accreditation.

METHODS

Four databases (Pubmed, EMBASE, ERIC, and PsychINFO) were searched for relevant published articles. Synthesis was performed using meta-ethnography.

RESULTS

Eighteen articles were included in the final synthesis with 1017 individual participants from 10 countries. Findings were categorised into four domains, including navigating power differentials, evaluating credibility, influencing medical programmes, and culture and behaviour. The synthesis demonstrates divergent views on the value of accreditation in medical schools from students and staff including both positive and negative impacts on medical education programmes and stakeholders.

CONCLUSIONS

Although accreditation is perceived to have many benefits, it also has a number of unintended consequences, including on staff morale, student-teacher relationships, and teacher workloads. Medical teachers also have a number of concerns about the credibility of accreditation standards, assessors, and processes. Regulators and policymakers should consider the views of teachers and students as they seek to improve current accreditation practices.

Recent research into healthcare professions regulation: a rapid evidence assessment

BACKGROUND AND AIMS

Over the last decade, regulators have taken significant steps towards tackling perceptions that regulatory systems are burdensome. There has been much international research activity in the regulation of health and care professionals. This article reports a review of studies on health professions regulation between January 2011 and March 2020. Its chief object was to provide robust and up-to-date evidence to assist regulators in policy development and implementation. The main objectives of this study were to: 1. Identify and retrieve research in the field of health and care professions regulation in English since 2011; 2. Evaluate the published research, exploring its utility to regulators and practitioners, and drawing out any key messages; 3. Draw conclusions concerning the scope and limitations of the research literature and identify areas for further research.

METHODS

We undertook a rapid evidence assessment (REA) of the international literature on health and care professions regulation, including reviewing ten UK regulators' websites to identify issues of concern and strategic priorities. We retrieved 3833 references, using a four-stage screening process to select the 81 most relevant.

RESULTS

Results are reported within six key themes: harm prevention and patient safety; fitness to practise; quality assurance of education and training; registration including maintenance of registers; guidelines and standards and relations with regulatory bodies.

CONCLUSIONS

Regulation of professionals in health and care is comparatively undeveloped as a field of academic study. Consequently, the published evidence is diffuse and small-scale. Most work presents relatively weak data of low relevance to regulators, mainly reporting or describing the current position. Few studies are able to show the impact of regulation or demonstrate a causal link between regulation and its effects. To inform their research and policy agendas health and social care regulators need to commission, interpret and apply the scholarly literature more effectively; academics need to engage with regulators to ensure that their research provides high-quality evidence with practical relevance to the regulators' agendas. Further study is needed to explore how effective academic collaborations between regulators and researchers may be created and sustained.

Discharge communication study: a realist evaluation of discharge communication experiences of patients, general practitioners and hospital practitioners, alongside a corresponding discharge letter sample

OBJECTIVES

To develop a programme theory for the intervention of patients receiving discharge letters.

DESIGN

We used a realist evaluation approach and captured multiple perspectives of hospital discharge to refine our previously developed programme theory. General practitioner (GP), patient and hospital clinician views of a single discharge event in which they were all involved were collected using semi-structured interviews and surveys. These were then triangulated to match the corresponding discharge letter. Data were qualitatively synthesised and compared in meta-matrices before interrogation with realist logic of analysis to develop the programme theory that maps out how patients receiving discharge letters works in specific contexts.

SETTING

14 GP practices and four hospital trusts in West Midlands, UK.

PARTICIPANTS

10 complete matched cases (GP, patient and hospital practitioner), and a further 26 cases in which a letter was matched with two out of the three participants.

RESULTS

We identified seven context mechanism outcome configurations not found through literature searching. These related to the broad concepts of: patient preference for receiving letters, patient comprehension of letters, patient-directed letters, patient harm and clinician views on patients receiving letters. 'Patient choice' was important to the success (or not) of the intervention. Other important contexts for positive effects included: letters written in plain English, lay explanations for jargon, verbal information also provided, no new information in letter and patient choice acknowledged. Three key findings were: patient understanding is perhaps greater than clinicians perceive, clinician attitudes are a barrier to patients receiving letters and that, negative outcomes more commonly manifested when patients had not received letters, rather than when they had.

CONCLUSIONS

We suggest how patients receiving discharge letters could be improved to enhance patient outcomes. Our programme theory has potential for use in different healthcare contexts and as a framework for policy development relating to patient discharge.

Postgraduate medical education quality metrics panels can be enhanced by including learner outcomes

Postgraduate medical education (PME) quality assurance at Health Education England (HEE) currently relies upon survey data. As no one metric can reflect all aspects of training, and each has its limitations, additional metrics should be explored. At HEE (West Midlands), we explored the use of learner outcomes, speciality examination pass rates and Annual Review of Competence Progression (ARCP) outcomes, as quality metrics. Feedback received from our local Quality Forum of 40 senior educators frames the discussion through this paper. Overall, learner outcomes are useful quality metrics that add to survey data to provide a more comprehensive picture of PME quality. However, the utility of ARCP outcomes as quality metrics is currently limited by concerns regarding variations in ARCP practice between regions. To address these concerns, ARCPs need the same processes, rigour, scrutiny and investment as other high-stakes assessments. This will improve the reliability and validity of the ARCP as an assessment and improve the usefulness of ARCP outcomes as quality metrics. Research is required to determine the optimal combination of metrics to use in PME quality assurance and to appraise the validity and reliability of the ARCP as an assessment.