Realist methods in medical education research: what are they and what can they contribute?

Role of institutional entrepreneurship in building adaptive capacity in community-based healthcare organisations: realist review protocol

INTRODUCTION

Over the past 3 decades, there has been a substantial shift to the marketisation of government-funded health services. For organisations traditionally buffered from the competitive pressures of for-profit enterprises, such as community-based organisations, this means developing the capacity to adapt to competitive tendering processes, shifting client expectations, and increasing demands for greater accountability. Drawing on ideas of institutional entrepreneurship, we believe that attempts to build adaptive capacity require the transformation of existing institutional arrangements. Key in this may be identifying and fostering institutional entrepreneurs--actors who take the lead in being the impetus for, and giving direction to, structural change. This study focuses on the strategies used by institutional entrepreneurs to build adaptive capacity in the community-based healthcare sector.

METHODS AND ANALYSIS

The research will use an adapted rapid realist review. The review will find underlying theories that explain the circumstances surrounding the implementation of capacity-building strategies that shape organisational response and generate outcomes by activating causal mechanisms. An early scoping of the literature, and consultations with key stakeholders, will be undertaken to identify an initial programme theory. We will search for relevant journal articles and grey literature. Data will be extracted based on contextual factors, mechanisms and outcomes, and their configurations. The analysis will seek patterns and regularities in these configurations and will focus on confirming, refuting or refining our programme theory.

ETHICS AND DISSEMINATION

The study does not involve primary research and, therefore, does not require formal ethical approval. However, ethical standards of utility, usefulness, feasibility, propriety, accuracy and accountability will be followed. The results will be written up according to the Realist and Meta-Review Evidence Synthesis: Evolving Standards guidelines. Once completed, findings will be published in a peer-reviewed journal.

TRIAL REGISTRATION NUMBER

CRD42015026487.

An exploration of contextual dimensions impacting goals of care conversations in postgraduate medical education

Background

Postgraduate medical trainees are not well prepared difficult conversations about goals of care with patients and families in the acute care clinical setting. While contextual nuances within the workplace can impact communication, research to date has largely focused on individual communication skills. Our objective was to explore contextual factors that influence conversations between trainees and patients/families about goals of care in the acute care setting.

Methods

We conducted an exploratory qualitative study involving five focus groups with Internal Medicine trainees (n = 20) and a series of interviews with clinical faculty (n = 11) within a single Canadian centre. Thematic framework analysis was applied to categorize the data and identify themes and subthemes.

Results

Challenges and factors enabling goals of care conversations emerged within individual, interpersonal and system dimensions. Challenges included inadequate preparation for these conversations, disconnection between trainees, faculty and patients, policies around documentation, the structure of postgraduate medical education, and resource limitations; these challenges led to missed opportunities, uncertainty and emotional distress. Enabling factors were awareness of the importance of goals of care conversations, support in these discussions, collaboration with colleagues, and educational initiatives enabling skill development; these factors have resulted in learning, appreciation, and an established foundation for future educational initiatives.

Conclusions

Contextual factors impact how postgraduate medical trainees communicate with patients/families about goals of care. Attention to individual, interpersonal and system-related factors will be important in designing educational programs that help trainees develop the capacities needed for challenging conversations.

Electronic supplementary material

The online version of this article (doi:10.1186/s12904-016-0107-6) contains supplementary material, which is available to authorized users.

Rethinking research in the medical humanities: a scoping review and narrative synthesis of quantitative outcome studies

OBJECTIVES

The rise of medical humanities teaching in medical education has introduced pressure to prove efficacy and utility. Review articles on the available evidence have been criticised for poor methodology and unwarranted conclusions. To support a more nuanced discussion of how the medical humanities work, we conducted a scoping review of quantitative studies of medical humanities teaching.

METHODS

Using a search strategy involving MEDLINE, EMBASE and ERIC, and hand searching, our scoping review located 11 045 articles that referred to the use of medical humanities teaching in medical education. Of these, 62 studies using quantitative evaluation methods were selected for review. Three iterations of analysis were performed: descriptive, conceptual, and discursive.

RESULTS

Descriptive analysis revealed that the medical humanities as a whole cannot be easily systematised based on simple descriptive categories. Conceptual analysis supported the development of a conceptual framework in which the foci of the arts and humanities in medical education can be mapped alongside their related epistemic functions for teaching and learning. Within the framework, art functioned as expertise, as dialogue or as a means of expression and transformation. In the discursive analysis, we found three main ways in which the relationship between the arts and humanities and medicine was constructed as, respectively, intrinsic, additive and curative.

CONCLUSIONS

This review offers a nuanced framework of how different types of medical humanities work. The epistemological assumptions and discursive positioning of medical humanities teaching frame the forms of outcomes research that are considered relevant to curriculum decision making, and shed light on why dominant review methodologies make some functions of medical humanities teaching visible and render others invisible. We recommend the use of this framework to improve the rigor and relevance of future explorations of the efficacy and utility of medical humanities teaching.

Using Win-Win Strategies to Implement Health in All Policies: A Cross-Case Analysis

Background

In spite of increasing research into intersections of public policy and health, little evidence shows how policy processes impact the implementation of Health in All Policies (HiAP) initiatives. Our research sought to understand how and why strategies for engaging partners from diverse policy sectors in the implementation of HiAP succeed or fail in order to uncover the underlying social mechanisms contributing to sustainable implementation of HiAP.

Methods

In this explanatory multiple case study, we analyzed grey and peer-review literature and key informant interviews to identify mechanisms leading to implementation successes and failures in relation to different strategies for engagement across three case studies (Sweden, Quebec and South Australia), after accounting for the role of different contextual conditions.

Findings

Our results yielded no support for the use of awareness-raising or directive strategies as standalone approaches for engaging partners to implement HiAP. However, we found strong evidence that mechanisms related to “win-win” strategies facilitated implementation by increasing perceived acceptability (or buy-in) and feasibility of HiAP implementation across sectors. Win-win strategies were facilitated by mechanisms related to several activities, including: the development of a shared language to facilitate communication between actors from different sectors; integrating health into other policy agendas (eg., sustainability) and use of dual outcomes to appeal to the interests of diverse policy sectors; use of scientific evidence to demonstrate the effectiveness of HiAP; and using health impact assessment to make policy coordination for public health outcomes more feasible and to give credibility to policies being developed by diverse policy sectors.

Conclusion

Our findings enrich theoretical understanding in an under-unexplored area of intersectoral action. They also provide policy makers with examples of HiAP across wealthy welfare regimes, and improve understanding of successful HiAP implementation practices, including the win-win approach.

How do medical educators design a curriculum that facilitates student learning about professionalism?

OBJECTIVE

This study analyses the ways in which curriculum reform facilitated student learning about professionalism.

METHODS

Design-based research provided the structure for an iterative approach to curriculum change which we undertook over a 3 year period. The learning environment of the Personal and Professional Development Theme (PPD) was analysed through the sociocultural lens of Activity Theory. Lave and Wenger's and Mezirow's learning theories informed curriculum reform to support student development of a patient-centred and critically reflective professional identity. The renewed pedagogical outcomes were aligned with curriculum content, learning and teaching processes and assessment, and intense staff education was undertaken. We analysed qualitative data from tutor interviews and free-response student surveys to evaluate the impact of curriculum reform.

RESULTS

Students' and tutors' reflections on learning in PPD converged on two principle themes--'Developing a philosophy of medicine' and 'Becoming an ethical doctor'--which corresponded to the overarching PPD theme aims of communicative learning. Students and tutors emphasised the importance of the unique learning environment of PPD tutorials for nurturing personal development and the positive impact of the renewed assessment programme on learning.

CONCLUSIONS

A theory-led approach to curriculum reform resulted in student engagement in the PPD curriculum and facilitated a change in student perspective about the epistemological foundation of medicine.

A critical hybrid realist-outcomes systematic review of relationships between medical education programmes and communities: BEME Guide No. 35

BACKGROUND

The relationships between medical schools and communities have long inspired and troubled medical education programmes. Successive models of community-oriented, community-based and community-engaged medical education have promised much and delivered to varying degrees. A two-armed realist systematic review was undertaken to explore and synthesize the evidence on medical school-community relationships.

METHOD

One arm used standard outcomes criteria (Kirkpatrick levels), the other a realist approach seeking out the underlying contexts, mechanisms and outcomes. 38 reviewers completed 489 realist reviews and 271 outcomes reviews; 334 articles were reviewed in the realist arm and 181 in the outcomes arm. Analyses were based on: descriptive statistics on both articles and reviews; the outcomes involved; the quality of the evidence presented; realist contexts, mechanisms, and outcomes; and an analysis of underlying discursive themes.

FINDINGS

The literature on medical school-community relationships is heterogeneous and largely idiographic, with no common standards for what a community is, who represents communities, what a relationship is based on, or whose needs are or should be being addressed or considered.

CONCLUSIONS

Community relationships can benefit medical education, even if it is not always clear why or how. There is much opportunity to improve the quality and precision of scholarship in this area.

Assessment of preclinical students' academic motivation before and after a three-day academic affair program

Background

Medical students’ motivation is an important driving factor for academic performance, and therefore medical teachers and educators are often highly interested in this topic. This study evaluated the impact of an academic affair program upon preclinical year medical students’ motivation to study.

Design and methods

An intervention study was conducted using a pretest-posttest study design. A total of 296 preclinical year medical students who had just passed their first year and were about to attend their second year at the Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand, participated in the study. The intervention comprised of dialogues for personality development, pictorial expression in groups, as well as small group lectures delivered by senior students giving information on how to prepare for the forthcoming classes. Students’ academic motivation was measured before and after the intervention program, applying the transculturally translated Academic Motivation Scale (AMS). Cronbach’s alpha of Thai version AMS was 0.8992. The average scores in seven scales of AMS were compared between the pre- and posttest results, using the Wilcoxon signed-rank test. The differences were confirmed by using the multivariate analysis of variance.

Results

Students’ academic motivation increased after participation in the three-day academic program. There was also a significant increase in introjected extrinsic motivation, which can enhance the students’ self-esteem and feeling of self-worth (P<0.001). Moreover, intrinsic motivation toward accomplishment increased significantly (P<0.001). This is related to the enjoyment of passing academic milestones, and a step ahead of autonomous motivation. Amotivation level declined significantly (P<0.001). The change of academic motivational constructs before and after the intervention was altogether significant (P=0.036, multivariate analysis of variance).

Conclusion

After experiencing a three-day intervention, the new students’ motivation advanced along the continuum of self-determination toward autonomous motivation. Therefore, it is considered to be worthwhile conducting an academic intervention to catalyze the evolution of preclinical year medical students’ academic motivation. Moreover, educators and faculties should evaluate the impact of interventions in evidence-based approaches to secure both controlled and autonomous types of motivation.

A realist review of family-based interventions for children of substance abusing parents

BACKGROUND

Millions of children across North America and Europe live in families with alcohol or drug abusing parents. These children are at risk for a number of negative social, emotional and developmental outcomes, including an increased likelihood of developing a substance use disorder later in life. Family-based intervention programs for children with substance abusing parents can yield positive outcomes. This study is a realist review of evaluations of family-based interventions aimed at improving psychosocial outcomes for children of substance abusing parents (COSAPs). The primary objectives were to uncover patterns of contextual factors and mechanisms that generate program outcomes, and advance program theory in this field.

METHODS

Realist review methodology was chosen as the most appropriate method of systematic review because it is a theory-driven approach that seeks to explore mechanisms underlying program effectiveness (or lack thereof). A systematic and comprehensive search of academic and grey literature uncovered 32 documents spanning 7 different intervention programs. Data was extracted from the included documents using abstraction templates designed to code for contexts, mechanisms and outcomes of each program. Two candidate program theories of family addiction were used to guide data analysis: the family disease model and the family prevention model. Data analysis was undertaken by a research team using an iterative process of comparison and checking with original documents to determine patterns within the data.

RESULTS

Programs originating in both the family disease model and the family prevention model were uncovered, along with hybrid programs that successfully included components from each candidate program theory. Four demi-regularities were found to account for the effectiveness of programs included in this review: (1) opportunities for positive parent-child interactions, (2) supportive peer-to-peer relationships, (3) the power of knowledge, and (4) engaging hard to reach families using strategies that are responsive to socio-economic needs and matching services to client lived experience.

CONCLUSIONS

This review yielded new findings that had not otherwise been explored in COSAP program research and are discussed in order to help expand program theory. Implications for practice and evaluation are further discussed.

Interventions that support the creation of dementia friendly environments in health care: protocol for a realist review

BACKGROUND

Improving health-care outcomes for people living with dementia when they are admitted to hospital is a policy priority. Dementia friendly interventions in health care promote inclusion of patients and carers in decision-making and adapt practices and environments to be appropriate to the needs of people with cognitive impairment. While there has been a wealth of activity, the number of studies evaluating interventions is limited, and the majority focuses on reporting staff and organisational outcomes. By focusing on patient and carer outcomes, this review will aim to develop an explanatory account of how and in what circumstances dementia friendly environments in health care work for people living with dementia and with what outcomes.

METHOD/DESIGN

Realist review is a theory-driven method which seeks to produce explanatory accounts of why interventions work and specifically, what combination of components are most effective in producing particular outcomes. Stakeholder interviews, a review of the literature, and an expert steering group workshop will be used to explore the assumptions behind interventions that are designed to enhance health care for people living with dementia to understand the underlying programme theories. The review will focus on studies that report patient and carer outcomes, including involvement in decision-making, length of stay and referral to long-term care, adverse incidents (e.g. patient distress, delirium falls, nutrition and hydration and infection), antipsychotic medication prescribing, evidence of patient-centred care and patient and carer satisfaction.

DISCUSSION

The review will provide an explanatory model about how dementia friendly interventions in hospital settings improve outcomes for people living with dementia and their family carers and in what circumstances for future testing and evaluation of future dementia friendly initiatives.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42015017562.

Collective action for knowledge mobilisation: a realist evaluation of the Collaborations for Leadership in Applied Health Research and Care

Background

The establishment of the Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) was the culmination of a number of policy initiatives to bridge the gap between evidence and practice. CLAHRCs were created and funded to facilitate development of partnerships and connect the worlds of academia and practice in an effort to improve patient outcomes through the conduct and application of applied health research.

Objectives

Our starting point was to test the theory that bringing higher education institutions and health-care organisations closer together catalyses knowledge mobilisation. The overall purpose was to develop explanatory theory regarding implementation through CLAHRCs and answer the question ‘what works, for whom, why and in what circumstances?’. The study objectives focused on identifying and tracking implementation mechanisms and processes over time; determining what influences whether or not and how research is used in CLAHRCs; investigating the role played by boundary objects in the success or failure of implementation; and determining whether or not and how CLAHRCs develop and sustain interactions and communities of practice.

Methods

This study was a longitudinal realist evaluation using multiple qualitative case studies, incorporating stakeholder engagement and formative feedback. Three CLAHRCs were studied in depth over four rounds of data collection through a process of hypothesis generation, refining, testing and programme theory specification. Data collection included interviews, observation, documents, feedback sessions and an interpretive forum.

Findings

Knowledge mobilisation in CLAHRCs was a function of a number of interconnected issues that provided more or less conducive conditions for collective action. The potential of CLAHRCs to close the metaphorical ‘know–do’ gap was dependent on historical regional relationships, their approach to engaging different communities, their architectures, what priorities were set and how, and providing additional resources for implementation, including investment in roles and activities to bridge and broker boundaries. Additionally, we observed a balance towards conducting research rather than implementing it. Key mechanisms of interpretations of collaborative action, opportunities for connectivity, facilitation, motivation, review and reflection, and unlocking barriers/releasing potential were important to the processes and outcomes of CLAHRCs. These mechanisms operated in different contexts including stakeholders’ positioning, or ‘where they were coming from’, governance arrangements, availability of resources, competing drivers, receptiveness to learning and evaluation, and alignment of structures, positions and resources. Preceding conditions influenced the course and journey of the CLAHRCs in a path-dependent way. We observed them evolving over time and their development led to the accumulation of different types of impacts, from those that were conceptual to, later in their life cycle, those that were more direct.

Conclusions

Most studies of implementation focus on researching one-off projects, so a strength of this study was in researching a systems approach to knowledge mobilisation over time. Although CLAHRC-like approaches show promise, realising their full potential will require a longer and more sustained focus on relationship building, resource allocation and, in some cases, culture change. This reinforces the point that research implementation within a CLAHRC model is a long-term investment and one that is set within a life cycle of organisational collaboration.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

The concept of mechanism from a realist approach: a scoping review to facilitate its operationalization in public health program evaluation

BACKGROUND

Public health interventions are complex by nature, and their evaluation requires unpacking their intervention logic and their interactions with open social systems. By focusing on the interrelationships between context, mechanism, and outcome, Pawson and Tilley's realist approach appears a promising innovation for public health-related evaluation works. However, and as expected of any methodological innovation, this approach is being constructed gradually by answering the multiple challenges to its operationalization that fall in its path. One of these challenges, users of this approach agree on, is the necessity of clarifying its key concept of mechanism.

METHOD

We first collected the definitions of mechanism from published works of Pawson and colleagues. Secondly, a scoping review was conducted to identify the ones quoted by users of the realist approach for evaluating public health interventions (1997-2012). We then appraised the clarity and precision of this concept against the three dimensions defined by Daigneault and Jacobs "term, sense and referent."

RESULTS

Of the 2344 documents identified in the scoping review, 49 documents were included. Term: Users of the realist approach use adjectives qualifying the term mechanism that were not specifically endorsed by Pawson and colleagues. Sense: None of the attributes stated by Pawson and colleagues has been listed in all of the documents analyzed, and some contributions clarified its attributes. Referent: The concept of mechanism within a realist approach can be ascribed to theory-based evaluation, complex social interventions, and critical realism.

CONCLUSION

This review led us to reconsider the concept of mechanism within the realist approach by confronting the theoretical stance of its proponents to the practical one of its users. This resulted in a clearer, more precise definition of the concept of mechanism which may in turn trigger further improvements in the way the realist approach is applied in evaluative practice in public health and potentially beyond. A mechanism is hidden but real, is an element of reasoning and reactions of agents in regard to the resources available in a given context to bring about changes through the implementation of an intervention, and evolves within an open space-time and social system of relationships.

A realist analysis of hospital patient safety in Wales: applied learning for alternative contexts from a multisite case study

Background

Hospital patient safety is a major social problem. In the UK, policy responses focus on the introduction of improvement programmes that seek to implement evidence-based clinical practices using the Model for Improvement, Plan-Do-Study-Act cycle. Empirical evidence that the outcomes of such programmes vary across hospitals demonstrates that the context of their implementation matters. However, the relationships between features of context and the implementation of safety programmes are both undertheorised and poorly understood in empirical terms.

Objectives

This study is designed to address gaps in conceptual, methodological and empirical knowledge about the influence of context on the local implementation of patient safety programmes.

Design

We used concepts from critical realism and institutional analysis to conduct a qualitative comparative-intensive case study involving 21 hospitals across all seven Welsh health boards. We focused on the local implementation of three focal interventions from the 1000 Lives+patient safety programme: Improving Leadership for Quality Improvement, Reducing Surgical Complications and Reducing Health-care Associated Infection. Our main sources of data were 160 semistructured interviews, observation and 1700 health policy and organisational documents. These data were analysed using the realist approaches of abstraction, abduction and retroduction.

Setting

Welsh Government and NHS Wales.

Participants

Interviews were conducted with 160 participants including government policy leads, health managers and professionals, partner agencies with strategic oversight of patient safety, advocacy groups and academics with expertise in patient safety.

Main outcome measures

Identification of the contextual factors pertinent to the local implementation of the 1000 Lives+patient safety programme in Welsh NHS hospitals.

Results

An innovative conceptual framework harnessing realist social theory and institutional theory was produced to address challenges identified within previous applications of realist inquiry in patient safety research. This involved the development and use of an explanatory intervention–context–mechanism–agency–outcome (I-CMAO) configuration to illustrate the processes behind implementation of a change programme. Our findings, illustrated by multiple nested I-CMAO configurations, show how local implementation of patient safety interventions are impacted and modified by particular aspects of context: specifically, isomorphism, by which an intervention becomes adapted to the environment in which it is implemented; institutional logics, the beliefs and values underpinning the intervention and its source, and their perceived legitimacy among different groups of health-care professionals; and the relational structure and power dynamics of the functional group, that is, those tasked with implementing the initiative. This dynamic interplay shapes and guides actions leading to the normalisation or the rejection of the patient safety programme.

Conclusions

Heightened awareness of the influence of context on the local implementation of patient safety programmes is required to inform the design of such interventions and to ensure their effective implementation and operationalisation in the day-to-day practice of health-care teams. Future work is required to elaborate our conceptual model and findings in similar settings where different interventions are introduced, and in different settings where similar innovations are implemented.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Psychobiobehavioral Model for Preterm Birth in Pregnant Women in Low- and Middle-Income Countries

Preterm birth (PTB) is a final common outcome resulting from many interrelated etiological pathways; of particular interest is antenatal psychosocial distress (i.e., stress, anxiety, and depression). In LMI countries, both exposure to severe life stressors and rate of PTB are on average greater when compared with high-income countries. In LMI countries women are exposed to some of the most extreme psychosocial stress worldwide (e.g., absolute poverty, limited social resources). High prevalence of antenatal stress and depression have been observed in some studies from LMI countries. We propose a psychosocial, biological, and behavioral model for investigating the complex multisystem interactions in stress responses leading to PTB and explain the basis of this approach. We discuss ethical considerations for a psychosocial, biological, and behavioral screening tool to predict PTB from a LMI country perspective.

Evaluating an interprofessional education curriculum: A theory-informed approach

BACKGROUND

This paper retrospectively reports on an evaluation framework applied to a local interprofessional education (IPE) curriculum design. The theoretically informed IPE curriculum spans the undergraduate health and social care programmes of over 10 professions as a curriculum theme. The teaching design and its impact were informed by psycho-social and learning theories.

AIMS

This meta-analysis is presented to share the importance of longitudinal IPE, whole curriculum evaluation for comparisons and to advance our understandings of what works and why.

METHOD

The meta-analysis used the Presage, Process and Product conceptual framework outlined by Biggs in 1993, and the Kirkpatrick in 1996, evaluation outcome model. Data are shared on the final overall learning from evaluating the teaching and the outcomes from students, teachers, practitioners, patients and carers.

RESULTS

The evaluation highlighted cyclical issues relating to students experiences, facilitators abilities and highlights the challenges of learning in practice which was highly praised by students. The problems and challenges were solved through the application of theory to illuminate our understandings.

CONCLUSION

We lament at missed opportunities for the application of theoretically informed research questions that still require to be addressed. However, we share this framework as having offered a complete and comprehensive evaluation process.

A realist review of brief interventions for alcohol misuse delivered in emergency departments

BACKGROUND

Brief interventions (BIs) involve screening for alcohol misuse and providing feedback to patients about their use, with the aim of reducing alcohol consumption and related consequences. BIs have been implemented in various healthcare settings, including emergency departments (ED), where they have been found to contribute mixed results in their ability to address alcohol misuse among adults. Mechanisms through which BIs work and contextual factors impacting BI effectiveness are not clear. The purpose of this review was to understand how, for whom, and under what circumstances BIs work for adults misusing alcohol and who have been admitted to an ED. A realist review was chosen to answer these questions as realist reviews create context-mechanism-outcome configurations, leading to the development of comprehensive and detailed theories; in this case explaining how and for whom BIs work.

METHODS

Databases including PsycINFO, Healthstar, CINAHL, Medline, and Nursing and Allied Health were searched for articles published until December 2013. The search strategy focused on studies examining BIs that targeted alcohol misuse among adults admitted into the ED. The search identified 145 relevant abstracts, of which 36 were included in the review. The literature was synthesized qualitatively (immersion/crystallization).

RESULTS

Four mechanisms were found within reviewed studies, including engagement in/retention of BI materials, resolving ambivalence, increased awareness/insight into consequences of drinking, and increased self-efficacy/empowerment to use skills for change. The following contexts were found to impact mechanisms: emotional state, injury attributed to alcohol use, severity of alcohol use, and baseline stage of change.

CONCLUSIONS

This realist review provides advances in theories regarding which mechanisms to target during a BI and which contexts create the most favorable conditions for these mechanisms to occur, ultimately leading to optimal BI outcomes. These results can inform future clinical decision-making when delivering BIs in ED settings. Future research should conduct quantitative examination to confirm these findings.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42013006549.

Fracture in the Elderly Multidisciplinary Rehabilitation (FEMuR): study protocol for a phase II randomised feasibility study of a multidisciplinary rehabilitation package following hip fracture [ISRCTN22464643]

BACKGROUND

Proximal femoral fracture is a common, major health problem in old age resulting in loss of functional independence and a high-cost burden on society, with estimated health and social care costs of £2.3 billion per year in the UK. Rehabilitation has the potential to maximise functional recovery and maintain independent living, but evidence of effectiveness is lacking. Usual rehabilitation care is delivered by a multi-disciplinary team in the hospital and in the community. An 'enhanced rehabilitation' intervention has been developed consisting of a workbook, goal-setting diary and extra therapy sessions, designed to improve self-efficacy and increase the amount and quality of the practice of physical exercise and activities of daily living.

METHODS/DESIGN

This paper describes the design of a phase II study comprising an anonymous cohort of all proximal femoral fracture patients admitted to the three acute hospitals in Betsi Cadwaladr University Health Board over a 6-month period with a randomised feasibility study comparing the enhanced rehabilitation intervention with usual care. These will assess the feasibility of a future definitive randomised controlled trial and concurrent economic evaluation in terms of recruitment, retention, outcome measure completion, compliance with the intervention and fidelity of delivery, health service use data, willingness to be randomised and effect size for a future sample size calculation. Focus groups will provide qualitative data to contribute to the assessment of the acceptability of the intervention amongst patients, carers and rehabilitation professionals and the feasibility of delivering the planned intervention. The primary outcome measure is function assessed by the Barthel Index. Secondary outcomes measure the ability to perform activities of daily living, anxiety and depression, potential mediators of outcomes such as hip pain, self-efficacy and fear of falling, health utility, health service use, objectively assessed physical function and adverse events. Participants' preference for rehabilitation services will be assessed in a discrete choice experiment.

DISCUSSION

Phase II studies are an opportunity to not only assess the feasibility of trial methods but also to compare different methods of outcome measurement and novel methods of obtaining health service use data from routinely collected patient information.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN22464643, UKCRN16677.

Factors influencing intercultural doctor-patient communication: a realist review

OBJECTIVE

Due to migration, doctors see patients from different ethnic backgrounds. This causes challenges for the communication. To develop training programs for doctors in intercultural communication (ICC), it is important to know which barriers and facilitators determine the quality of ICC. This study aimed to provide an overview of the literature and to explore how ICC works.

METHODS

A systematic search was performed to find literature published before October 2012. The search terms used were cultural, communication, healthcare worker. A realist synthesis allowed us to use an explanatory focus to understand the interplay of communication.

RESULTS

In total, 145 articles met the inclusion criteria. We found ICC challenges due to language, cultural and social differences, and doctors' assumptions. The mechanisms were described as factors influencing the process of ICC and divided into objectives, core skills and specific skills. The results were synthesized in a framework for the development of training.

CONCLUSION

The quality of ICC is influenced by the context and by the mechanisms. These mechanisms translate into practical points for training, which seem to have similarities with patient-centered communication.

PRACTICE IMPLICATIONS

Training for improving ICC can be developed as an extension of the existing training for patient-centered communication.

Conceptual and practical challenges in the assessment of physician competencies

Abstract The shift to using outcomes-based competency frameworks in medical education in many countries around the world requires educators to find ways to assess multiple competencies. Contemporary medical educators recognize that a competent trainee not only needs sound biomedical knowledge and technical skills, they also need to be able to communicate, collaborate and behave in a professional manner. This paper discusses methodological challenges of assessment with a particular focus on the CanMEDS Roles. The paper argues that the psychometric measures that have been the mainstay of assessment practices for the past half-century, while still valuable and necessary, are not sufficient for a competency-oriented assessment environment. New assessment approaches, particularly ones from the social sciences, are required to be able to assess non-Medical Expert (Intrinsic) roles that are situated and context-bound. Realist and ethnographic methods in particular afford ways to address the challenges of this new assessment. The paper considers the theoretical and practical bases for tools that can more effectively assess non-Medical Expert (Intrinsic) roles.

Twelve tips for improving the effectiveness of web-based multimedia instruction for clinical learners

Using educational technology does not necessarily make medical education more effective. There are many different kinds of technology available to the contemporary medical teacher and what constitutes effective use may depend on the technology, the learning situation and many other factors. Web-based multimedia instruction (WBMI) provides learners with self-directed independent learning opportunities based on didactic material enhanced with multimedia features such as video and animations. WBMI may be used to replace other didactic events (e.g. lectures) or it may be provided in addition to other learning opportunities. Clinical educators looking to use WBMI need to make sure that it will meet both their learners' needs and the program's needs, and it has to align to the contexts in which it is used. The following 12 tips have been developed to help guide faculty through some of the key features of the effective use of WBMI in clinical teaching programs. These tips are based on more than a decade developing, using and appraising WBMI in support of surgical clerkship education across the USA and beyond and they are intended both to inform individual uses of WBMI in clinical training and to guide the strategic use of WBMI in clinical clerkship curricula.

Grades in formative workplace-based assessment: a study of what works for whom and why

CONTEXT

Grades are commonly used in formative workplace-based assessment (WBA) in medical education and training, but may draw attention away from feedback about the task. A dilemma arises because the self-regulatory focus of a trainee must include self-awareness relative to agreed standards, which implies grading.

OBJECTIVES

In this study we aimed to understand the meaning which medical students construct from WBA feedback with and without grades, and what influences this.

METHODS

Year 3 students were invited to take part in a randomised crossover study in which each student served as his or her own control. Each student undertook one WBA with and one without grades, and then chose whether or not to be given grades in a third WBA. These preferences were explored in semi-structured interviews. A realist approach to analysis was used to gain understanding of student preferences and the impact of feedback with and without grades.

RESULTS

Of 83 students who were given feedback with and without grades, 65 (78%) then chose to have feedback with grades and 18 (22%) without grades in their third WBA. A total of 24 students were interviewed. Students described how grades locate their performance and calibrate their self-assessment. For some, low grades focused attention and effort. Satisfactory and high grades enhanced self-efficacy.

CONCLUSIONS

Grades are concrete, powerful and blunt, can be harmful and need to be explained to help students create helpful meaning from them. Low grades risk reducing self-efficacy in some and may encourage others to focus on proving their ability rather than on improvement. A metaphor of the semi-permeable membrane is introduced to elucidate how students reduced potential negative effects and enhanced the positive effects of feedback with grades by selective filtering and pumping. This study illuminates the complexity of the processing of feedback by its recipients, and informs the use of grading in the provision of more effective, tailored feedback.

Lung cancer staging: the value of PET depends on the clinical setting

BACKGROUND

Although positron emission tomography (PET) imaging is widely recommended in the evaluation of patients with lung cancer, randomized controlled trials (RCTs) assessing this have demonstrated inconsistent results. We asked whether differences in the clinical context and endpoints could explain these discrepancies.

METHODS

We used realist synthesis methods to analyze how contextual differences among RCTs affected the results. We focused on RCTs to minimize confounding yet permit evaluation of differences by comparing across studies.

RESULTS

This analysis suggests that the impact of PET depends on the clinical setting. PET is of greatest benefit in identifying M1 disease in patients with a high chance of such involvement and when little traditional imaging [e.g., abdominal/pelvis computed tomography (CT) and bone scan] is used. Identification of N2,3 involvement by PET prior to resection is seen primarily when there is at least a moderate probability of such and the rate of invasive staging is high. The rate of N2 disease not identified preoperatively appears to increase if PET is used to avoid invasive mediastinal staging in clinical settings in which the risk of N2,3 involvement is moderately high. There is both a potential benefit in avoiding stage-inappropriate resection as well as a risk of missed (stage-appropriate) resection if PET findings are not evaluated carefully.

CONCLUSIONS

A blanket recommendation for PET may be too simplistic without considering nuances of the clinical setting.

Understanding rural clinical learning spaces: Being and becoming a doctor

CONTEXT

Calls for health professions education that can foster transformative educational experiences have been voiced. Studies suggest that extended clinical training at rural sites potentially provides transformative learning spaces. This article explores 'being and becoming' as a construct for understanding the student experience at a rural clinical school (RCS).

METHODS

Sixty-two in-depth interviews were conducted over a three-year period with RCS students, graduates (as interns) and intern supervisors. Thematically analysed data were mapped according to the adapted Kirkpatrick model for appraising educational interventions. Drawing on realist perspectives, findings were further analysed to discern the mechanisms influencing the being and becoming of junior doctors.

RESULTS

Responses provided evidence of changed attitudes and behaviour, and the adoption of professional practice that was seen to influence patient outcomes. Analysis highlighted sharing of values through role modelling, engagement with preceptors, being respected as part of a team, and being trusted to assume responsibility for a patient as key mechanisms. The outcome was confident, competent and caring interns.

DISCUSSION

Rural clinical learning spaces influence the 'being and becoming' of a junior doctor. Understanding this process in the context of place (rural platform), participation (community of practice) and person can inform expanded agendas for students' clinical learning.

Toward a common understanding: supporting and promoting education scholarship for medical school faculty

CONTEXT

Education scholarship (ES) is integral to the transformation of medical education. Faculty members who engage in ES need encouragement and recognition of this work. Beginning with the definition of ES as 'an umbrella term which can encompass both research and innovation in health professions education', and which as such represents an activity that is separate and distinct from teaching and leadership, the purpose of our study was to explore how promotion policies and processes are used in Canadian medical schools to support and promote ES.

METHODS

We conducted an analysis of the promotion policies of 17 Canadian medical schools and interviews with a key informant at each institution. We drew on an interpretive approach to policy analysis to analyse the data and to understand explicit messages about how ES was represented and supported.

RESULTS

Of the 17 schools' promotion documents, only nine contained specific reference to ES. There was wide variation in focus and level of detail. All key informants indicated that ES is recognised and considered for academic promotion. Barriers to the support and recognition of ES included a lack of understanding of ES and its relationship to teaching and leadership. This was manifest in the variability in promotion policies and processes, support systems, and career planning and pathways for ES.

CONCLUSIONS

This lack of clarity may make it challenging for medical school faculty members to make sense of how they might successfully align ES within an academic career. There is a need therefore to better articulate ES in promotion policies and support systems. Creating a common understanding of ES, developing guidelines to assess the impact of all forms of ES, developing an informed leadership and system of mentors, and creating explicit role descriptions and guidelines are identified as potential strategies to ensure that ES is appropriately valued.

How design of places promotes or inhibits mobility of older adults: realist synthesis of 20 years of research

OBJECTIVE

The objective of this study was to determine the environmental features that best support aging in place.

METHOD

We conducted a realist synthesis, a theory-driven interpretive method of evidence synthesis, of 120+ articles (published 1991-2011) that attempts to explain how place may influence older adults' decisions about mobility (e.g., physical activity). We developed an initial program theory, reviewed the literature, identified outcomes, analyzed and synthesized patterns, and created a final program theory.

RESULTS

Safety was a central mechanism, serving as one of the bridges between environmental components (e.g., connectivity, aesthetics, retail and services) and decisions about mobility. Population density, sidewalk presence, and park proximity did not emerge as key factors.

DISCUSSION

Safety considerations are one of the most prominent influences of older adults' decisions about mobility. Street connectivity, pedestrian access and transit, and retail and services were also important. These factors are amenable to change and can help promote mobility for older adults.

Improving health Professional's knowledge of hepatitis B using cartoon based learning tools: a retrospective analysis of pre and post tests

BACKGROUND

Hepatitis B serology is complex and a lack of knowledge in interpretation contributes to the inadequate levels of screening and referral for highly effective hepatitis antiviral treatments. This knowledge gap needs to be addressed so that current and future healthcare professionals are more confident in the detection and assessment of hepatitis B to improve the uptake of treatment and reduce long-term complications from the disease. Cartoons have been used effectively as a teaching tool in other settings and were considered as a potentially useful teaching aid in explaining hepatitis B serology. This study examines the impact of cartoons in improving healthcare professionals' knowledge.

METHODS

A cartoon based learning tool designed to simplify the complexities of hepatitis B serology was developed as part of an online learning program for medical practitioners, nurses and students in these professions. A retrospective analysis was carried out of pre and post online test results.

RESULTS

An average improvement of 96% of correct answers to case study questions in hepatitis B serology was found across all ten questions following the use of an online cartoon based learning tool.

CONCLUSION

The data indicates a significant improvement of participants' knowledge of hepatitis B serology from pre-test to post-test immediately following an online cartoon based learning tool. However, further research is required to measure its long term impact.

The role of reading on the health and well-being of people with neurological conditions: a systematic review

OBJECTIVES

Little research has been conducted that investigates the benefits of reading for people with neurological conditions despite its age old use to improve well-being. The aim of this study was to identify and review the evidence of the effect of 'lone' reading, reading aloud and shared reading groups on the health and well-being of people with neurological conditions in clinical and long-term care settings.

METHODS

A literature search was conducted incorporating a systematic search of electronic databases, internet searching, 'snowballing' technique from references of relevant studies and consultation with clinicians and academics in the field.

RESULTS

Twelve studies (five quantitative, three qualitative and four mixed methods) met the criteria for inclusion in the review. No randomised controlled trials were identified. Significant heterogeneity in the results of the quantitative studies precluded statistical data synthesis. Thematic analysis and synthesis was applied to the three qualitative studies and the qualitative data of the mixed-method studies. All but one of the quantitative studies reported that the reading interventions had a positive effect. The evidence from the qualitative studies demonstrated multiple positive effects of shared reading groups.

CONCLUSIONS

The effect of 'lone' reading, reading aloud and shared reading groups on the health and well-being of people with neurological conditions is currently an under-researched area. Although this review reports encouraging results of positive effects, the results should be viewed with caution due to the lack of randomisation, the small numbers of participants involved, and the limited and heterogeneous evidence base.

Improving psychosocial health and employment outcomes for individuals receiving methadone treatment: a realist synthesis of what makes interventions work

BACKGROUND

For over 50 years, methadone has been prescribed to opioid-dependent individuals as a pharmacological approach for alleviating the symptoms of opioid withdrawal. However, individuals prescribed methadone sometimes require additional interventions (e.g., counseling) to further improve their health. This study undertook a realist synthesis of evaluations of interventions aimed at improving the psychosocial and employment outcomes of individuals on methadone treatment, to determine what interventions work (or not) and why.

METHODS

The realist synthesis method was utilized because it uncovers the processes (or mechanisms) that lead to particular outcomes, and the contexts within which this occurs. A comprehensive search process resulted in 31 articles for review. Data were extracted from the articles, and placed in four templates to assist with analysis. Data analysis was an iterative process and involved comparing and contrasting data within and across each template, and cross checking with original articles to determine key patterns in the data.

RESULTS

For individuals on methadone, engagement with an intervention appears to be important for improved psychosocial and/or employment outcomes. The engagement process involves attendance at interventions as well as an investment in what is offered. Three intervention contexts (often in some combination) support the engagement process: a) client-centered contexts (or those where clients' psychosocial and/or employment needs/issues/skills are recognized and/or addressed); b) contexts which address clients' socio-economic conditions and needs; and, c) contexts where there are positive client-counselor and/or peer relationships. There is some evidence that sometimes ongoing engagement is necessary to maintain positive outcomes. There is also some evidence that complete abstinence from drugs (e.g., cocaine, heroin) is not necessary for engagement.

CONCLUSIONS

It is important to consider how the contexts of interventions might elicit and/or support clients' engagement. Further research is needed to explore how an individual's background (e.g., involvement with different interventions over an extended period) may influence engagement. Long-term engagement may be necessary to sustain some positive outcomes although how long is unclear and requires further research. Engagement can occur without complete abstinence from such drugs as cocaine or heroin, but additional research is required as engagement may be influenced by the extent and type of drug use.

How much evidence does it take? A cumulative meta-analysis of outcomes of simulation-based education

CONTEXT

Studies that investigate research questions that have already been resolved represent a waste of resources. However, the failure to collect sufficient evidence to resolve a given question results in ambiguity.

OBJECTIVES

The present study was conducted to reanalyse the results of a meta-analysis of simulation-based education (SBE) to determine: (i) whether researchers continue to replicate research studies after the answer to a research question has become known, and (ii) whether researchers perform enough replications to definitively answer important questions.

METHODS

A systematic search of multiple databases to May 2011 was conducted to identify original research evaluating SBE for health professionals in comparison with no intervention or any active intervention, using skill outcomes. Data were extracted by reviewers working in duplicate. Data synthesis involved a cumulative meta-analysis to illuminate patterns of evidence by sequentially adding studies according to a variable of interest (e.g. publication year) and re-calculating the pooled effect size with each addition. Cumulative meta-analysis by publication year was applied to 592 comparative studies using several thresholds of 'sufficiency', including: statistical significance; stable effect size classification and magnitude (Hedges' g ± 0.1), and precise estimates (confidence intervals of less than ± 0.2).

RESULTS

Among studies that compared the outcomes of SBE with those of no intervention, evidence supporting a favourable effect of SBE on skills existed as early as 1973 (one publication) and further evidence confirmed a quantitatively large effect of SBE by 1997 (28 studies). Since then, a further 404 studies were published. Among studies comparing SBE with non-simulation instruction, the effect initially favoured non-simulation training, but the addition of a third study in 1997 brought the pooled effect to slightly favour simulation, and by 2004 (14 studies) this effect was statistically significant (p < 0.05) and the magnitude had stabilised (small effect). A further 37 studies were published after 2004. By contrast, evidence from studies evaluating repetition continued to show borderline statistical significance and wide confidence intervals in 2011.

CONCLUSIONS

Some replication is necessary to obtain stable estimates of effect and to explore different contexts, but the number of studies of SBE often exceeds the minimum number of replications required.

'No need to worry': an exploration of general practitioners' reassuring strategies

Background

In view of the paucity of evidence regarding effective ways of reassuring worried patients, this study explored reassuring strategies that are considered useful by general practitioners (GPs).

Methods

In a study using a qualitative observational design, we re-analysed an existing dataset of fifteen stimulated recall interviews in which GPs elaborated on their communication with patients in two videotaped consultations. Additionally we held stimulated recall interviews with twelve GPs about two consultations selected for a strong focus on reassurance.

Results

To reassure patients, GPs pursued multiple goals: 1. influencing patients’ emotions by promoting trust, safety and comfort, which is considered to be reassuring in itself and supportive of patients’ acceptance of reassuring information and 2. influencing patients’ cognitions by challenging patients’ belief that their symptoms are indicative of serious disease, often followed by promoting patients’ belief that their symptoms are benign. GPs described several actions to activate mechanisms to achieve these goals.

Conclusions

GPs described a wealth of reassuring strategies, which make a valuable contribution to the current literature on doctor-patient communication. This detailed description may provide practicing GPs with new tools and can inform future studies exploring the effectiveness of reassurance strategies.

A framework for outcome-level evaluation of in-service training of health care workers

BACKGROUND

In-service training is a key strategic approach to addressing the severe shortage of health care workers in many countries. However, there is a lack of evidence linking these health care worker trainings to improved health outcomes. In response, the United States President's Emergency Plan for AIDS Relief's Human Resources for Health Technical Working Group initiated a project to develop an outcome-focused training evaluation framework. This paper presents the methods and results of that project.

METHODS

A general inductive methodology was used for the conceptualization and development of the framework. Fifteen key informant interviews were conducted to explore contextual factors, perceived needs, barriers and facilitators affecting the evaluation of training outcomes. In addition, a thematic analysis of 70 published articles reporting health care worker training outcomes identified key themes and categories. These were integrated, synthesized and compared to several existing training evaluation models. This formed an overall typology which was used to draft a new framework. Finally, the framework was refined and validated through an iterative process of feedback, pilot testing and revision.

RESULTS

The inductive process resulted in identification of themes and categories, as well as relationships among several levels and types of outcomes. The resulting framework includes nine distinct types of outcomes that can be evaluated, which are organized within three nested levels: individual, organizational and health system/population. The outcome types are: (1) individual knowledge, attitudes and skills; (2) individual performance; (3) individual patient health; (4) organizational systems; (5) organizational performance; (6) organizational-level patient health; (7) health systems; (8) population-level performance; and (9) population-level health. The framework also addresses contextual factors which may influence the outcomes of training, as well as the ability of evaluators to determine training outcomes. In addition, a group of user-friendly resources, the Training Evaluation Framework and Tools (TEFT) were created to help evaluators and stakeholders understand and apply the framework.

CONCLUSIONS

Feedback from pilot users suggests that using the framework and accompanying tools may support outcome evaluation planning. Further assessment will assist in strengthening guidelines and tools for operationalization.

Glossary for the implementation of Health in All Policies (HiAP)

Health in All Policies (HiAP) is becoming increasingly popular as a governmental strategy to improve population health by coordinating action across health and non-health sectors. A variety of intersectoral initiatives may be used in HiAP that frame health determinants as the bridge between policies and health outcomes. The purpose of this glossary is to present concepts and terms useful in understanding the implementation of HiAP as a cross-sectoral policy. The concepts presented here were applied and elaborated over the course of case studies of HiAP in multiple jurisdictions, which used key informant interviews and the systematic review of literature to study the implementation of specific HiAP initiatives.

A review of longitudinal community and hospital placements in medical education: BEME Guide No. 26

BACKGROUND

Traditionally, clinical learning for medical students consists of short-term and opportunistic encounters with primarily acute-care patients, supervised by an array of clinician preceptors. In response to educational concerns, some medical schools have developed longitudinal placements rather than short-term rotations. Many of these longitudinal placements are also integrated across the core clinical disciplines, are commonly termed longitudinal integrated clerkships (LICs) and often situated in rural locations. This review aimed to explore, analyse and synthesise evidence relating to the effectiveness of longitudinal placements, for medical students in particular to determine which aspects are most critical to successful outcomes.

METHOD

Extensive search of the literature resulted in 1679 papers and abstracts being considered, with 53 papers ultimately being included for review. The review group coded these 53 papers according to standard BEME review guidelines. Specific information extracted included: data relating to effectiveness, the location of the study, number of students involved, format, length and description of placement, the learning outcomes, research design, the impact level for evaluation and the main evaluation methods and findings. We applied a realist approach to consider what works well for whom and under what circumstances.

FINDINGS

The early LICs were all community-based immersion programs, situated in general practice and predominantly in rural settings. More recent LIC innovations were situated in tertiary-level specialist ambulatory care in urban settings. Not all placements were integrated across medical disciplines but were longitudinal in relation to location, patient base and/or supervision. Twenty-four papers focussed on one of four programs from different viewpoints. Most evaluations were student opinion (survey, interview, focus group) and/or student assessment results. Placements varied from one half day per week for six months through to full time immersion for more than 12 months. The predominant mechanism relating to factors influencing effectiveness was continuity of one or more of: patient care, supervision and mentorship, peer group and location. The success of LICs and participation satisfaction depended on the preparation of both students and clinical supervisors, and the level of support each received from their academic institutions.

CONCLUSION

Longitudinal placements, including longitudinal integrated placements, are gaining in popularity as an alternative to traditional block rotations. Although relatively few established LICs currently exist, medical schools may look for ways to incorporate some of the principles of LICs more generally in their clinical education programmes. Further research is required to ascertain the optimum length of time for placements depending on the defined learning outcomes and timing within the programme, which students are most likely to benefit and the effects of context such as location and type of integration.

A systematic review of outcome and impact of master's in health and health care

BACKGROUND

The 'human resources for health' crisis has highlighted the need for more health (care) professionals and led to an increased interest in health professional education, including master's degree programmes. The number of these programmes in low- and middle-income countries (LMIC) is increasing, but questions have been raised regarding their relevance, outcome and impact. We conducted a systematic review to evaluate the outcomes and impact of health-related master's degree programmes.

METHODS

We searched the databases Scopus, Pubmed, Embase, CINAHL, ERIC, Psychinfo and Cochrane (1999 - November 2011) and selected websites. All papers describing outcomes and impact of health-related Master programmes were included. Three reviewers, two for each article, extracted data independently. The articles were categorised by type of programme, country, defined outcomes and impact, study methods used and level of evidence, and classified according to outcomes: competencies used in practice, graduates' career progression and impact on graduates' workplaces and sector/society.

RESULTS

Of the 33 articles included in the review, most originated from the US and the UK, and only one from a low-income country. The programmes studied were in public health (8), nursing (8), physiotherapy (5), family practice (4) and other topics (8). Outcomes were defined in less than one third of the articles, and impact was not defined at all. Outcomes and impact were measured by self-reported alumni surveys and qualitative methods. Most articles reported that competencies learned during the programme were applied in the workplace and alumni reported career progression or specific job changes. Some articles reported difficulties in using newly gained competencies in the workplace. There was limited evidence of impact on the workplace. Only two articles reported impact on the sector. Most studies described learning approaches, but very few described a mechanism to ensure outcome and impact of the programme.

CONCLUSIONS

Evidence suggests that graduates apply newly learned competencies in the field and that they progress in their career. There is a paucity of well-designed studies assessing the outcomes and impact of health-related master's degree programmes in low- and middle-income countries. Studies of such programmes should consider the context and define outcomes and impact.

RAMESES publication standards: realist syntheses

BACKGROUND

There is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas - for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for realist systematic reviews.

DESIGN

A mixed method study synthesising data between 2011-2012 from a literature review, online Delphi panel and feedback from training, workshops and email list.

METHODS

We: (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigour may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards.

RESULTS

We identified 35 published realist syntheses, provided real-time support to 9 ongoing syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within 3 rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%.

CONCLUSIONS

This project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses.

RAMESES publication standards: realist syntheses

BACKGROUND

There is growing interest in realist synthesis as an alternative systematic review method. This approach offers the potential to expand the knowledge base in policy-relevant areas - for example, by explaining the success, failure or mixed fortunes of complex interventions. No previous publication standards exist for reporting realist syntheses. This standard was developed as part of the RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) project. The project's aim is to produce preliminary publication standards for realist systematic reviews.

METHODS

We (a) collated and summarized existing literature on the principles of good practice in realist syntheses; (b) considered the extent to which these principles had been followed by published syntheses, thereby identifying how rigor may be lost and how existing methods could be improved; (c) used a three-round online Delphi method with an interdisciplinary panel of national and international experts in evidence synthesis, realist research, policy and/or publishing to produce and iteratively refine a draft set of methodological steps and publication standards; (d) provided real-time support to ongoing realist syntheses and the open-access RAMESES online discussion list so as to capture problems and questions as they arose; and (e) synthesized expert input, evidence syntheses and real-time problem analysis into a definitive set of standards.

RESULTS

We identified 35 published realist syntheses, provided real-time support to 9 on-going syntheses and captured questions raised in the RAMESES discussion list. Through analysis and discussion within the project team, we summarized the published literature and common questions and challenges into briefing materials for the Delphi panel, comprising 37 members. Within three rounds this panel had reached consensus on 19 key publication standards, with an overall response rate of 91%.

CONCLUSION

This project used multiple sources to develop and draw together evidence and expertise in realist synthesis. For each item we have included an explanation for why it is important and guidance on how it might be reported. Realist synthesis is a relatively new method for evidence synthesis and as experience and methodological developments occur, we anticipate that these standards will evolve to reflect further methodological developments. We hope that these standards will act as a resource that will contribute to improving the reporting of realist syntheses.

Manchester Clinical Placement Index (MCPI). Conditions for medical students' learning in hospital and community placements

The drive to quality-manage medical education has created a need for valid measurement instruments. Validity evidence includes the theoretical and contextual origin of items, choice of response processes, internal structure, and interrelationship of a measure's variables. This research set out to explore the validity and potential utility of an 11-item measurement instrument, whose theoretical and empirical origins were in an Experience Based Learning model of how medical students learn in communities of practice (COPs), and whose contextual origins were in a community-oriented, horizontally integrated, undergraduate medical programme. The objectives were to examine the psychometric properties of the scale in both hospital and community COPs and provide validity evidence to support using it to measure the quality of placements. The instrument was administered twice to students learning in both hospital and community placements and analysed using exploratory factor analysis and a generalizability analysis. 754 of a possible 902 questionnaires were returned (84% response rate), representing 168 placements. Eight items loaded onto two factors, which accounted for 78% of variance in the hospital data and 82% of variance in the community data. One factor was the placement learning environment, whose five constituent items were how learners were received at the start of the placement, people's supportiveness, and the quality of organisation, leadership, and facilities. The other factor represented the quality of training-instruction in skills, observing students performing skills, and providing students with feedback. Alpha coefficients ranged between 0.89 and 0.93 and there were no redundant or ambiguous items. Generalisability analysis showed that between 7 and 11 raters would be needed to achieve acceptable reliability. There is validity evidence to support using the simple 8-item, mixed methods Manchester Clinical Placement Index to measure key conditions for undergraduate medical students' experience based learning: the quality of the learning environment and the training provided within it. Its conceptual orientation is towards Communities of Practice, which is a dominant contemporary theory in undergraduate medical education.

Overcoming barriers to addressing education problems with research design: a panel discussion

A plenary panel session at the 2012 Academic Emergency Medicine consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success" discussed barriers educators face in imagining, designing, and implementing studies to address educational challenges. This proceedings article presents a general approach to getting started in education research. Four examples of studies from the medical education literature that illustrate a distinct way to approach specific research questions are discussed. The study designs used are applicable to a variety of education research problems in emergency medicine (EM). Potential applications of studies are discussed, as well as effects and lessons learned.

Workplace learning

This critical review found Dutch research to be strong at the undergraduate and residency levels and more or less absent in continuing medical education. It confirms the importance of coaching medical students, giving constructive feedback, and ensuring practice environments are conducive to learning though it has proved hard to improve them. Residents learn primarily from experiences encountered in the course of clinical work but the fine balance between delivering clinical services and learning can easily be upset by work pressure. More intervention studies are needed. Qualitative research designs need to be more methodologically sophisticated and use a wider range of data sources including direct observation, audio-diaries, and text analysis. Areas for improvement are clear but achieving results will require persistence and patience.

Randomized controlled trials and meta-analysis in medical education: what role do they play?

Education researchers seek to understand what works, for whom, in what circumstances. Unfortunately, educational environments are complex and research itself is highly context dependent. Faced with these challenges, some have argued that qualitative methods should supplant quantitative methods such as randomized controlled trials (RCTs) and meta-analysis. I disagree. Good qualitative and mixed-methods research are complementary to, rather than exclusive of, quantitative methods. The complexity and challenges we face should not beguile us into ignoring methods that provide strong evidence. What, then, is the proper role for RCTs and meta-analysis in medical education? First, the choice of study design depends on the research question. RCTs and meta-analysis are appropriate for many, but not all, study goals. They have compelling strengths but also numerous limitations. Second, strong methods will not compensate for a pointless question. RCTs do not advance the science when they make confounded comparisons, or make comparison with no intervention. Third, clinical medicine now faces many of the same challenges we encounter in education. We can learn much from other fields about how to handle complexity in RCTs. Finally, no single study will definitively answer any research question. We need carefully planned, theory-building, programmatic research, reflecting a variety of paradigms and approaches, as we accumulate evidence to change the art and science of education.