Exploration of perceived effects of innovations in postgraduate medical education

How workplace-based assessments guide learning in postgraduate education: A scoping review

INTRODUCTION

Competency-based medical education (CBME) led to the widespread adoption of workplace-based assessment (WBA) with the promise of achieving assessment for learning. Despite this, studies have illustrated tensions between the summative and formative role of WBA which undermine learning goals. Models of workplace-based learning (WBL) provide insight, however, these models excluded WBA. This scoping review synthesizes the primary literature addressing the role of WBA to guide learning in postgraduate medical education, with the goal of identifying gaps to address in future studies.

METHODS

The search was applied to OVID Medline, Web of Science, ERIC and CINAHL databases, articles up to September 2020 were included. Titles and abstracts were screened by two reviewers, followed by a full text review. Two members independently extracted and analysed quantitative and qualitative data using a descriptive-analytic technique rooted in Billett's four premises of WBL. Themes were synthesized and discussed until consensus.

RESULTS

All 33 papers focused on the perception of learning through WBA. The majority applied qualitative methodology (70%), and 12 studies (36%) made explicit reference to theory. Aligning with Billett's first premise, results reinforce that learning always occurs in the workplace. WBA helped guide learning goals and enhanced feedback frequency and specificity. Billett's remaining premises provided an important lens to understand how tensions that existed in WBL have been exacerbated with frequent WBA. As individuals engage in both work and WBA, they are slowly transforming the workplace. Culture and context frame individual experiences and the perceived authenticity of WBA. Finally, individuals will have different goals, and learn different things, from the same experience.

CONCLUSION

Analysing WBA literature through the lens of WBL theory allows us to reframe previously described tensions. We propose that future studies attend to learning theory, and demonstrate alignment with philosophical position, to advance our understanding of assessment-for-learning in the workplace.

Factors that may improve paediatric workplace-based assessments: an exploratory study

OBJECTIVES

To establish if paediatric trainees are satisfied with the current workplace-based assessment (WBA) process. To identify factors that contribute both positively and negatively to the educational experience during WBAs. To find out if trainees and their supervisors experience any challenges conducting WBAs. To establish potential ways to improve future assessments.

DESIGN

Qualitative semistructured interviews.

SETTING

Participants included fifteen trainees (ST1-8) in general paediatric and subspecialty posts and four consultants or equivalent across five hospital sites in the Thames Valley Deanery. All participants had regular exposure to WBAs.

INTERVENTIONS

Interviews were undertaken between June 2020 and January 2021 via video link. Data collection and analysis were conducted iteratively using constant comparison until theoretical sufficiency was achieved.

MAIN OUTCOME MEASURE

Using Constructivist Grounded Theory, a theoretical framework, grounded in the data, was developed that depicted the core elements that should be present to optimise WBAs.

RESULTS

A number of key components were reported to affect the educational value of WBAs. A positive departmental culture towards education and training is essential. Chosen cases should be challenging, and direct observation or in-depth discussion, depending on the assessment type, is fundamental. Timely constructive feedback and immediate completion of the assessment form are also imperative.

CONCLUSION

Some trainees experienced WBAs where these key components aligned, but many did not, and this negatively affected their learning. Three main challenges or future targets for further improvements include increasing time, improving training and optimising technology.

Bridging the Gap: Using Consensus to Explore Entrustment Decisions and Feedback Receptivity in Competency-Based Emergency Medicine Residency Programs Through the Construction of a Q-Sample Incorporating a Delphi Technique

Background

Recent changes in medical education calls for a shift toward student-centered learning. Therefore, it is imperative that clinical educators transparently assess the work-readiness of their medical residents through entrustment-based supervision decisions toward independent practice. Similarly, it is critical that medical residents are vocal about the quality of supervision and feedback they receive. This study aimed to explore the factors that influence entrustment-based supervision decisions and feedback receptivity by establishing a general consensus among Taiwanese clinical educators and medical residents regarding entrustment decisions and feedback uptake, respectively.

Methods

In Q-methodology studies, a set of opinion statement (i.e., the Q-sample) is generated to represent the phenomenon of interest. To explore the factors that influence entrustment-based supervision decisions and feedback receptivity, a Q-sample was developed using a four-step approach: (1) literature search using electronic databases, such as PubMed and Google Scholar, and interviews with emergency clinical educators and medical residents to generate opinion statements, (2) thematic analysis and grouping using The Model of Trust, the Ready, Wiling, and Able model, and the theory of self-regulated learning, (3) translation, and (4) application of a Delphi technique, including two expert panels comprised of clinical educators and medical residents, to establish a consensus of the statements and validation for a subsequent Q-study.

Results

A total of 585 and 1,039 statements from the literature search and interviews were extracted to populate the sample of statements (i.e., the concourse) regarding entrustment-based supervision decisions for clinical educators and feedback receptivity emergency medicine residents, respectively. Two expert panels were invited to participate in a Delphi Technique, comprised of 11 clinical educators and 13 medical residents. After two-rounds of a Delphi technique, the panel of clinical educators agreed on 54 statements on factors that influence entrustment-based supervision decisions and were categorized into five themes defined by the Model of Trust. Similarly, a total of 60 statements on the factors that influence feedback receptivity were retained by the panel of medical residents and were categorized into five themes defined by the Ready, Willing, and Able model and the theory of self-regulated learning.

Conclusion

Though not exhaustive, the key factors agreed upon by clinical educators and medical residents reflect the characteristics of entrustment-based supervision decisions and feedback receptivity across specialties. This study provides insight on an often overlooked issue of the paths to teaching and learning in competency-based residency training programs. Additionally, incorporation of the Delphi technique further adds to the existing literature and puts emphasis as an important tool that can be used in medical education to rigorously validate Q-statements and develop Q-samples in various specialties.

Emotion as reflexive practice: A new discourse for feedback practice and research

INTRODUCTION

Like medicine and health care, feedback is a practice imbued with emotions: saturated with feelings relevant to one's identity and status within a given context. Often this emotional dimension of feedback is cast as an impediment to be ignored or managed. Such a perspective can be detrimental to feedback practices as emotions are fundamentally entwined with learning. In this critical review, we ask: What are the discourses of emotion in the feedback literature and what 'work' do they do?

METHODS

We conducted a critical literature review of emotion and feedback in the three top journals of the field: Academic Medicine, Medical Education and Advances in Health Sciences Education. Analysis was informed by a Foucauldian critical discourse approach and involved identifying discourses of emotion and interpreting how they shape feedback practices.

FINDINGS

Of 32 papers, four overlapping discourses of emotion were identified. Emotion as physiological casts emotion as internal, biological, ever-present, immutable and often problematic. Emotion as skill positions emotion as internal, mainly cognitive and amenable to regulation. A discourse of emotion as reflexive practice infers a social and interpersonal understanding of emotions, whereas emotion as socio-cultural discourse extends the reflexive practice discourse seeing emotion as circulating within learning environments as a political force.

DISCUSSION

Drawing on scholarship within the sociology of emotions, we suggest the merits of studying emotion as inevitable (not pathological), as potentially paralysing and motivating and as situated within (and often reinforcing) a hierarchical social health care landscape. For future feedback research, we suggest shifting towards recognising the discourse-theory-practice connection with emotion in health professional education drawing from reflexive and socio-cultural discourses of emotion.

Identifying Player Types to Tailor Game-Based Learning Design to Learners: Cross-sectional Survey using Q Methodology

BACKGROUND

Game-based learning appears to be a promising instructional method because of its engaging properties and positive effects on motivation and learning. There are numerous options to design game-based learning; however, there is little data-informed knowledge to guide the choice of the most effective game-based learning design for a given educational context. The effectiveness of game-based learning appears to be dependent on the degree to which players like the game. Hence, individual differences in game preferences should be taken into account when selecting a specific game-based learning design.

OBJECTIVE

We aimed to identify patterns in students' perceptions of play and games-player types and their most important characteristics.

METHODS

We used Q methodology to identify patterns in opinions on game preferences. We recruited undergraduate medical and dental students to participate in our study and asked participants to sort and rank 49 statements on game preferences. These statements were derived from a prior focus group study and literature on game preferences. We used by-person factor analysis and varimax rotation to identify common viewpoints. Both factors and participants' comments were used to interpret and describe patterns in game preferences.

RESULTS

From participants' (n=102) responses, we identified 5 distinct patterns in game preferences: the social achiever, the explorer, the socializer, the competitor, and the troll. These patterns revolved around 2 salient themes: sociability and achievement. The 5 patterns differed regarding cheating, playing alone, story-telling, and the complexity of winning.

CONCLUSIONS

The patterns were clearly interpretable, distinct, and showed that medical and dental students ranged widely in how they perceive play. Such patterns may suggest that it is important to take students' game preferences into account when designing game-based learning and demonstrate that not every game-based learning-strategy fits all students. To the best of our knowledge, this study is the first to use a scientifically sound approach to identify player types. This can help future researchers and educators select effective game-based learning game elements purposefully and in a student-centered way.

Chinese doctors' views on workplace-based assessment: trainee and supervisor perspectives of the mini-CEX

Purpose: This study investigated whether the mini-clinical evaluation exercise (mini-CEX) has been successfully integrated into the Chinese context, following its introduction as part of the national general training programme. Materials and methods: Online questionnaires (N = 91) and interviews (N = 22) were conducted with Year 1 trainee doctors and clinical supervisors at a cancer hospital in China to explore users' experiences, attitudes and opinions of the mini-CEX. Results" Trainees were more likely than supervisors to report understanding the purpose of the mini-CEX and agree that it encouraged reflection and helped improve overall performance. Both trainees and supervisors felt that it provided a framework for learning, that it was useful in identifying underperformance, and that it informed learning progression. Groups were equally positive about the commitment of their counterpart in the process and valued the focus on detailed feedback. It was perceived as cultivating the learner-teacher relationship. Overall, both groups felt they 'bought in' to using the mini-CEX. However, concerns were raised about subjectivity of ratings and lack of benchmarking with expected standards of care. Conclusions: Chinese trainees and supervisors generally perceived the mini-CEX as an acceptable and valuable medical training tool, although both groups suggested enhancements to improve its efficacy.

Can case-based discussions in a group setting be used to assess residents' clinical skills?

OBJECTIVES

The purpose of this study was to explore residents' and assessors' perception of a new group assessment concept.

METHODS

This qualitative study consists of observations of four group assessment sessions, followed by semi-structured interviews with six residents and four assessors (specialists in internal medicine), who all volunteered to be interviewed. All residents at a medical department (eleven to fifteen each time) and four assessors participated in four group assessments, where the residents' clinical skills were assessed through case-based discussions. An external consultant (an anthropologist) performed the observations and the interviews. Notes from the observations and the interviews were analyzed using an inductive approach.

RESULTS

Eight of the ten interviewed participants preferred group assessment to individual assessment. Results from the interviews suggested that the group assessments were more consistent and that the level of discussion was perceived to be higher in the group discussions compared to the one-to-one discussions. All residents indicated that they had acquired new knowledge during their assessment and reported having learned from listening to the assessment of their peers. Assessors similarly reported gaining new knowledge.

CONCLUSIONS

The residents and assessors expressed very favourable attitudes toward the new group assessment concept. The assessment process was perceived to be higher in quality and more consistent, contributing to learning for all participating doctors in the department. Group assessment is feasible and acceptable, and provides a promising tool for assessment of clinical skills in the future.

"There Is a Lot of Change Afoot": A Qualitative Study of Faculty Adaptation to Elimination of Tiered Grades With Increased Emphasis on Feedback in Core Clerkships

PURPOSE

Medical schools responding to challenges with fairness, equity, learning environments, and student wellness have reconsidered clerkship grades. This study used the Concerns-Based Adoption Model (CBAM) to explore how faculty responded to a change in the approach to assessment from focus on grades toward focus on feedback.

METHOD

This qualitative study used an inductive approach to analyze data from semistructured interviews with teaching faculty and education leaders at University of California, San Francisco, School of Medicine 6 months following the elimination of traditional tiered clerkship grades. Participants were recruited in 2019 using purposive sampling. Interview questions addressed participants' perceptions of the new approach to assessment and grading that emphasizes assessment for learning, including advantages and concerns. The authors analyzed data using thematic analysis informed by sensitizing concepts from CBAM.

RESULTS

Nineteen faculty participated (11 medicine, 8 surgery). Faculty expressed optimism about the impact of the change on their clinical educator roles and sought faculty development to enhance feedback and assessment skills. Perceived benefits to students addressed learning and wellness, with concern for students' motivation and professional development. Faculty shared uncertainty about their roles and larger systemic consequences involving the residency match and institutional reputation. Participants acknowledged that traditional grading is imperfect, yet felt a departure from this system represented significant culture change.

CONCLUSIONS

Faculty perceptions of the assessment and grade change in 2 large departments demonstrated tensions around grading. Their descriptions of change as an ongoing process aligned with CBAM and required them to consider new approaches and perspectives. While discourse about assessment and grading frequently focuses on learners, these findings support institutional consideration of the impact of changes in assessment on teaching faculty.

From prescription to guidance: a European framework for generic competencies

In postgraduate medical education, required competencies are described in detail in existing competency frameworks. This study proposes an alternative strategy for competency-based medical education design, which is supported by change management theories. We demonstrate the value of allowing room for re-invention and creative adaptation of innovations. This new strategy was explored for the development of a new generic competency framework for a harmonised European curriculum in Obstetrics and Gynaecology. The generic competency framework was developed through action research. Data were collected by four European stakeholder groups (patients, nurses, midwives and hospital boards), using a variety of methods. Subsequently, the data were analysed further in consensus discussions with European specialists and trainees in Obstetrics and Gynaecology. These discussions ensured that the framework provides guidance, is specialty-specific, and that implementation in all European countries could be feasible. The presented generic competency framework identifies four domains: 'Patient-centred care', 'Teamwork', 'System-based practice' and 'Personal and professional development'. For each of these four domains, guiding competencies were defined. The new generic competency framework is supported by European specialists and trainees in Obstetrics and Gynaecology, as well as by their European stakeholders. According to change management theories, it seems vital to allow room for re-invention and creative adaptation of the competency framework by medical professionals. Therefore, the generic competency framework offers guidance rather than prescription. The presented strategy for competency framework development offers leads for implementation of competency-based medical education as well as for development of innovations in postgraduate medical education in general.

Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students

Background:

Despite an increasing emphasis on workplace-based assessment (WPBA) during medical training, the existing assessment system largely relies on summative assessment while formative assessment is less valued. Various tools have been described for WPBA, mini-clinical evaluation exercise (mini-CEX) being one of them. Mini-CEX is well accepted in Western countries, however, reports of its use in India are scarce. We conducted this study to assess acceptability and feasibility of mini-CEX as a formative assessment tool for WPBA of surgical postgraduate students in an Indian setting.

Methods:

Faculty members and 2^nd year surgical residents were sensitized toward mini-CEX and requisite numbers of exercises were conducted. The difficulties during conduction of these exercises were identified, recorded, and appropriate measures were taken to address them. At the conclusion, the opinion of residents and faculty members regarding their experience with mini-CEX was taken using a questionnaire. The results were analyzed using simple statistical tools.

Results:

Nine faculty members out of 11 approached participated in the study (81.8%). All 16 2^nd year postgraduate surgical residents participated (100%). Sixty mini-CEX were conducted over 7 months. Each resident underwent 3–5 encounters. The mean time taken by the assessor for observation was 12.3 min (8–30 min) while the mean feedback time was 4.2 min (3–10 min). The faculty reported good overall satisfaction with mini-CEX and found it acceptable as a formative assessment tool. Three faculty members (33.3%) reported mini-CEX as more time-consuming while 2 (22.2%) found it difficult to carry the exercises often. All residents accepted mini-CEX and most of them reported good to high satisfaction with the exercises conducted.

Conclusions:

Mini-CEX is well accepted by residents and faculty as a formative assessment tool. It is feasible to utilize mini-CEX for WPBA of postgraduate students of surgery.

A qualitative study on harmonization of postgraduate medical education in Europe: negotiating flexibility is key

INTRODUCTION

International harmonization of postgraduate medical education is gaining importance in the globalization of medical education. Harmonization is regarded as the establishment of common standards in education, while maintaining regional or local freedom to adapt training to contexts. During the development of a harmonized curriculum, tensions between standardization and contextualization may surface. To allow future harmonization projects to recognize these tensions and deal with them in a timely manner, this study aims to gain insight into tensions that may arise when developing a harmonized curriculum for postgraduate medical education in Obstetrics and Gynaecology in Europe.

METHODS

Focus groups were held with international curriculum developers to discuss challenges that resulted from tensions between standardization and contextualization when developing a harmonized European curriculum for postgraduate medical education in Obstetrics and Gynaecology. Data were analyzed through conventional content analysis, using the principles of template analysis.

RESULTS AND DISCUSSION

Tensions between standardization and contextualization in the development of a harmonized curriculum were apparent in two domains: 1) Varying ideas about what the harmonized curriculum means for the current curriculum and 2) Inconsistencies between educational principles and the reality of training. Additionally, we identified ways of dealing with these tensions, which were characterized as 'negotiating flexibility'. Tensions between standardization and contextualization surfaced in the development phase of harmonizing a curriculum, partly because it was anticipated that there would be problems when implementing the curriculum.

Resident and attending perceptions of direct observation in internal medicine: a qualitative study

OBJECTIVES

Direct observation is the foundation of assessment and learning in competency-based medical education (CBME). Despite its importance, there is significant uncertainty about how to effectively implement frequent and high-quality direct observation. This is particularly true in specialties where observation of non-procedural skills is highly valued and presents unique challenges. It is therefore important to understand perceptions of direct observation to ensure successful acceptance and implementation. In this study, we explored perceptions of direct observation in internal medicine.

METHODS

We interviewed internal medicine attending physicians (n = 9) and residents (n = 8) at the University of Toronto, purposively sampled for diversity. Using a constructivist grounded theory approach, constant comparative analysis was performed to develop a framework to understand perceptions of direct observation on the clinical teaching units.

RESULTS

Participants articulated a narrow perception of what constitutes direct observation, in contrast to their own descriptions of skills that were observed. This resulted in the perception that certain valuable skills that participants felt were routinely observed were nonetheless not 'directly observable', such as clinical reasoning, observed through case presentations and patient care discussions. Differentiating direct observation from informal observation led to overestimation of the time and resource requirements needed to enhance direct observation, which contributed to scepticism and lack of engagement related to CBME implementation.

CONCLUSIONS

In an internal medicine training programme, perceptions of what constitutes direct observation can lead to under-recognition and hinder acceptance in workplace-based assessment and learning. Our results suggest a reframing of 'direct observation' for residents and attending physicians, by explicitly identifying desired skills in non-procedurally-based specialties. These findings may help CBME-based training programmes improve the process of direct observation, leading to enhanced assessment and learning.

Design of simulation-based medical education and advantages and disadvantages of in situ simulation versus off-site simulation

BACKGROUND

Simulation-based medical education (SBME) has traditionally been conducted as off-site simulation in simulation centres. Some hospital departments also provide off-site simulation using in-house training room(s) set up for simulation away from the clinical setting, and these activities are called in-house training. In-house training facilities can be part of hospital departments and resemble to some extent simulation centres but often have less technical equipment. In situ simulation, introduced over the past decade, mainly comprises of team-based activities and occurs in patient care units with healthcare professionals in their own working environment. Thus, this intentional blend of simulation and real working environments means that in situ simulation brings simulation to the real working environment and provides training where people work. In situ simulation can be either announced or unannounced, the latter also known as a drill. This article presents and discusses the design of SBME and the advantage and disadvantage of the different simulation settings, such as training in simulation-centres, in-house simulations in hospital departments, announced or unannounced in situ simulations.

DISCUSSION

Non-randomised studies argue that in situ simulation is more effective for educational purposes than other types of simulation settings. Conversely, the few comparison studies that exist, either randomised or retrospective, show that choice of setting does not seem to influence individual or team learning. However, hospital department-based simulations, such as in-house simulation and in situ simulation, lead to a gain in organisational learning. To our knowledge no studies have compared announced and unannounced in situ simulation. The literature suggests some improved organisational learning from unannounced in situ simulation; however, unannounced in situ simulation was also found to be challenging to plan and conduct, and more stressful among participants. The importance of setting, context and fidelity are discussed. Based on the current limited research we suggest that choice of setting for simulations does not seem to influence individual and team learning. Department-based local simulation, such as simulation in-house and especially in situ simulation, leads to gains in organisational learning. The overall objectives of simulation-based education and factors such as feasibility can help determine choice of simulation setting.

Twelve tips for assessing surgical performance and use of technical assessment scales

Using validated assessment scales for technical competence can help structure and standardize assessment and feedback for both the trainee and the supervisor and thereby avoid bias and drive learning. Correct assessment of operative skills can establish learning curves and allow adequate monitoring. However, the assessment of surgical performance is not an easy task, since it includes many proxy parameters, which are hard to measure. Although numerous technical assessment scales exist, both within laparoscopic and open surgery, the validity evidence is often sparse, and this can raise doubts about reliability and educational outcome. Furthermore, the implementation of technical assessment scales varies due to several obstacles and doubts about accurate use. In this 12-tips article, we aim to give the readers a critical and useful appraisal of some of the common questions and misunderstandings regarding the use of surgical assessment scales and provide tips to ease and overcome potential pitfalls.

The Role of Generic Competencies in the Entrustment of Professional Activities: A Nationwide Competency-Based Curriculum Assessed

BACKGROUND

Entrustable professional activities (EPAs) seek to translate essential physician competencies into clinical practice. Until now, it is not known whether EPA-based curricula offer enhanced assessment and feedback to trainees.

OBJECTIVE

This study examined program directors' and senior residents' justifications for entrustment decisions and what role generic, cross-specialty competencies (such as communication skills, collaboration, and understanding health care systems) play in these decisions.

METHODS

Entrustment decisions for all Dutch obstetrics and gynecology residents between January 2010 and April 2014 were retrieved from their electronic portfolios. Justifications for entrustment were divided into 4 categories: the resident's experience, his or her technical performance, the presence of a generic competency, and training. Template analysis was used to analyze in depth the types of justifications, which play a role in entrustment decisions.

RESULTS

A total of 5139 entrustment decisions for 375 unique residents were extracted and analyzed. In 59% of all entrustment decisions, entrusting a professional task to a resident was justified by the experience of the resident. Generic competencies were mentioned in 0.5% of all entrustment decisions. Template analysis revealed that the amount of exposure and technical skills are leading factors, while the quality of the performance was not reported to be of any influence.

CONCLUSIONS

Entrustment decisions only rarely are based on generic competencies, despite the introduction of competency frameworks and EPAs. For program directors, a leading factor in entrustment decisions is a resident's exposure to an activity, and the quality of a resident's performance appears to play only a minor role.

Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings

Workplace based assessments (WBAs) are now commonplace in postgraduate medical training. User acceptability and engagement is essential to the success of any medical education innovation. To this end, possessing an insight into trainee and trainer perceptions towards WBAs will help identify the major problems, permitting strategies to be introduced to improve WBA implementation. A review of literature was performed to identify studies examining trainee and trainer perceptions towards WBAs. Studies were excluded if non-English or sampling a non-medical/dental population. The identified literature was synthesised for the purpose of this critical narrative review. It is clear that there is widespread negativity towards WBAs in the workplace. This has negatively impacted on the effectiveness of WBA tools as learning aids. This negativity exists in trainees but also to an extent in their trainers. Insight gained from the literature reveals three dominant problems with WBA implementation: poor understanding as to the purpose of WBAs; insufficient time available for undertaking these assessments; and inadequate training of trainers. Approaches to addressing these three problems with WBA implementation are discussed. It is likely that a variety of solutions will be required. The prevalence of negativity towards WBAs is substantial in both trainees and trainers, eroding the effectiveness of learning that is consequent upon them. The educational community must now listen to the concerns being raised by the users and consider the range of strategies being proposed to improve the experiences of trainees, and their trainers.

Dealing with the complex dynamics of teaching hospitals

Innovation and change in postgraduate medical education programs affects teaching hospital organizations, since medical education and clinical service are interrelated.Recent trends towards flexible, time-independent and individualized educational programs put pressure on this relationship. This pressure may lead to organizational uncertainty, unbalance and friction making it an important issue to analyze.The last decade was marked by a transition towards outcome-based postgraduate medical education. During this transition competency-based programs made their appearance. Although competency-based medical education has the potential to make medical education more efficient, the effects are still under debate. And while this debate continues, the field of medical education is already introducing next level innovations: flexible and individualized training programs. Major organizational change, like the transition to flexible education programs, can easily lead to friction and conflict in teaching hospital organizations.This article analyses the organizational impact of postgraduate medical education innovations, with a particular focus on flexible training and competency based medical education. The characteristics of teaching hospital organizations are compared with elements of innovation and complexity theory.With this comparison the article argues that teaching hospital organizations have complex characteristics and behave in a non-linear way. This perspective forms the basis for further discussion and analysis of this unexplored aspect of flexible and competency based education.

Professional association group mentoring for allied health professionals

Purpose

– Mentoring is widely used in the health sector, particularly for early career professionals in the public health system. However, many allied health professionals are employed in private practice and rely on their professional association to provide mentoring support and training. This mentoring context is under-researched. The paper aims to discuss these issues.

Design/methodology/approach

– A purposeful sample of 15 allied health professionals were interviewed using semi-structured interviews that were then analyzed using template analysis.

Findings

– The many-to-many group mentoring program delivered valuable knowledge, diagnostic skills and networking opportunities but did not provide inclusion, role modeling or psychosocial support to participants. Also identified were structural and operational issues including; the role of the coordinator in addressing contribution reluctance and participant confidence, confidentiality issues, lack of mentor training and overall organization of the program.

Practical implications

– Group mentoring is a valuable method of delivery for professional associations. The many-to-many group mentoring model is beneficial in a situation where the availability of mentors is limited. Further, the importance of having a dedicated program coordinator and a skilled facilitator is emphasized.

Originality/value

– This research contributes to the limited literature on many-to-many group mentoring by reviewing the effectiveness of an existing many-to-many group mentoring program for allied health professionals delivered by a professional association.

Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study

In 2010, workplace-based assessment (WBA) was formally integrated as a method of formative trainee assessment into 29 basic and higher specialist medical training (BST/HST) programmes in six postgraduate training bodies in Ireland. The aim of this study is to explore how WBA is being implemented and to examine if WBA is being used formatively as originally intended. A retrospective cohort study was conducted and approved by the institution’s Research Ethics Committee. A profile of WBA requirements was obtained from 29 training programme curricula. A data extraction tool was developed to extract anonymous data, including written feedback and timing of assessments, from Year 1 and 2 trainee ePortfolios in 2012–2013. Data were independently quality assessed and compared to the reference standard number of assessments mandated annually where relevant. All 29 training programmes mandated the inclusion of at least one case-based discussion (max = 5; range 1–5). All except two non-clinical programmes (93 %) required at least two mini-Clinical Evaluation Exercise assessments per year and Direct Observation of Procedural Skills assessments were mandated in 27 training programmes over the course of the programme. WBA data were extracted from 50 % of randomly selected BST ePortfolios in four programmes (n = 142) and 70 % of HST ePortfolios (n = 115) in 21 programmes registered for 2012–2013. Four programmes did not have an eligible trainee for that academic year. In total, 1142 WBAs were analysed. A total of 164 trainees (63.8 %) had completed at least one WBA. The average number of WBAs completed by HST trainees was 7.75 (SD 5.8; 95 % CI 6.5–8.9; range 1–34). BST trainees completed an average of 6.1 assessments (SD 9.3; 95 % CI 4.01–8.19; range 1–76). Feedback—of varied length and quality—was provided on 44.9 % of assessments. The majority of WBAs were completed in the second half of the year. There is significant heterogeneity with respect to the frequency and quality of feedback provided during WBAs. The completion of WBAs later in the year may limit available time for feedback, performance improvement and re-evaluation. This study sets the scene for further work to explore the value of formative assessment in postgraduate medical education.

Innovating the practice of medical speciality training

Educational innovations are being introduced into medical speciality training. But how do people who participate in medical speciality training (residents, consultants, programme directors) deal with these innovations? And what effects do educational innovations have according to these people?By addressing these questions, this thesis contributes to the knowledge about the challenging process of innovating medical speciality training.

Feedback-giving behaviour in performance evaluations during clinical clerkships

CONTEXT

Narrative feedback documented in performance evaluations by the teacher, i.e. the clinical supervisor, is generally accepted to be essential for workplace learning. Many studies have examined factors of influence on the usage of mini-clinical evaluation exercise (mini-CEX) instruments and provision of feedback, but little is known about how these factors influence teachers' feedback-giving behaviour. In this study, we investigated teachers' use of mini-CEX in performance evaluations to provide narrative feedback in undergraduate clinical training.

METHODS

We designed an exploratory qualitative study using an interpretive approach. Focusing on the usage of mini-CEX instruments in clinical training, we conducted semi-structured interviews to explore teachers' perceptions. Between February and June 2013, we conducted interviews with 14 clinicians participated as teachers during undergraduate clinical clerkships. Informed by concepts from the literature, we coded interview transcripts and iteratively reduced and displayed data using template analysis.

RESULTS

We identified three main themes of interrelated factors that influenced teachers' practice with regard to mini-CEX instruments: teacher-related factors; teacher-student interaction-related factors, and teacher-context interaction-related factors. Four issues (direct observation, relationship between teacher and student, verbal versus written feedback, formative versus summative purposes) that are pertinent to workplace-based performance evaluations were presented to clarify how different factors interact with each other and influence teachers' feedback-giving behaviour. Embedding performance observation in clinical practice and establishing trustworthy teacher-student relationships in more longitudinal clinical clerkships were considered important in creating a learning environment that supports and facilitates the feedback exchange.

CONCLUSION

Teachers' feedback-giving behaviour within the clinical context results from the interaction between personal, interpersonal and contextual factors. Increasing insight into how teachers use mini-CEX instruments in daily practice may offer strategies for creating a professional learning culture in which feedback giving and seeking would be enhanced.

Evaluation of oral microbiology lab curriculum reform

BACKGROUND

According to the updated concept of oral microbiology, the School of Stomatology, Wuhan University, has carried out oral microbiology teaching reforms during the last 5 years. There was no lab curriculum before 2009 except for a theory course of oral microbiology. The school has implemented an innovative curriculum with oral medicine characteristics to strengthen understanding of knowledge, cultivate students' scientific interest and develop their potential, to cultivate the comprehensive ability of students. This study was designed to evaluate the oral microbiology lab curriculum by analyzing student performance and perceptions regarding the curriculum from 2009 to 2013.

METHODS

The lab curriculum adopted modalities for cooperative learning. Students collected dental plaque from each other and isolated the cariogenic bacteria with selective medium plates. Then they purified the enrichment culture medium and identified the cariogenic strains by Gram stain and biochemical tests. Both quantitative and qualitative data for 5 years were analysed in this study. Part One of the current study assessed student performance in the lab from 2009 to 2013. Part Two used qualitative means to assess students' perceptions by an open questionnaire.

RESULTS

The 271 study students' grades on oral microbiology improved during the lab curriculum: "A" grades rose from 60.5 to 81.2 %, and "C" grades fell from 28.4 to 6.3 %. All students considered the lab curriculum to be interesting and helpful. Quantitative and qualitative data converge to suggest that the lab curriculum has strengthened students' grasp of important microbiology-related theory, cultivated their scientific interest, and developed their potential and comprehensive abilities.

CONCLUSION

Our student performance and perception data support the continued use of the innovative teaching system. As an extension and complement of the theory course, the oral microbiology lab curriculum appears to improve the quality of oral medicine education and help to cultivate high-quality innovative medical talents.

Bureaucrapathologies: Galloping Regulosis, Assessment Degradosis, and Other Unintended Organizational Maladies in Post-Graduate Medical Education

As decadelong observers of evolving administrative regulations governing academic medicine, the authors have identified several organizational disorders they define as "bureaucrapathologies," pathological conditions caused by dysfunctional bureaucratic processes that generate excesses of wasted time, effort, and other resources. Appearing wherever bureaucratic organizations exist, they have become particularly egregious in health care, research, and education. In past decades, graduate medical education has been beset by proliferating assessment requirements accompanied by corresponding documentation requirements imposed by academic educational regulatory agencies (specifically the Accreditation Council on Graduate Medical). Although originating from the best of intentions, these largely untested, unvalidated, and unfunded mandates generate burdensome personnel, time, and resource requirements. As they trickle down organizational levels, the intentions of the originators are inevitably degraded. As motivations and incentives of lower level administrators and faculty differ considerably from those at higher levels, we inevitably encounter debatable assessment practices yielding results of dubious reliability and validity. These processes invariably lead to proliferating reports and paperwork. All of this raises serious questions about the benefits vs. harms of these enterprises. In our view, these pathogenic processes can be recognized as diagnosable subtypes of bureaucrapathology. Here the authors briefly describe two, Galloping Regulosis and Assessment Degradosis, which reflect on their pathogenesis and offer preliminary thoughts for potential remedies. Several other recently identified bureaucrapathological syndromes awaiting future delineation are noted.

Experience, trajectories, and reifications: an emerging framework of practice-based learning in healthcare workplaces

Learning by working is omnipresent in healthcare education. It enables people to learn how to perform, think, and interact in ways that work for their specific context. In this paper, I review my approach to studying this process. It centers on the question why healthcare professionals do what they do and how their actions and learning are intertwined. The aim of this paper is to illustrate what I have learned from the research I have been involved in, in such a way that it enables other researchers, educators, and clinicians to understand and study practice-based learning in healthcare workplaces. Therefore, I build on a programmatic line of research to present a framework of practice-based learning consisting of three inextricably linked levels of analysis. The first level focuses on how situations lead to personal experiences, the second level looks at strings of experiences that lead to multiple trajectories, and the third level deals with reifications arising from recurrent activities. This framework, and its interrelations and inherent tensions, helps to understand why healthcare workplaces can be both a powerful learning environment and a frustratingly hard place to change.

How faculty members experience workplace-based assessment rater training: a qualitative study

CONTEXT

Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training.

OBJECTIVES

This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and a modified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed.

METHODS

In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data.

RESULTS

Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in.

CONCLUSIONS

Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

The impact of programmatic assessment on student learning: theory versus practice

CONTEXT

It is widely acknowledged that assessment can affect student learning. In recent years, attention has been called to 'programmatic assessment', which is intended to optimise both learning functions and decision functions at the programme level of assessment, rather than according to individual methods of assessment. Although the concept is attractive, little research into its intended effects on students and their learning has been conducted.

OBJECTIVES

This study investigated the elements of programmatic assessment that students perceived as supporting or inhibiting learning, and the factors that influenced the active construction of their learning.

METHODS

The study was conducted in a graduate-entry medical school that implemented programmatic assessment. Thus, all assessment information, feedback and reflective activities were combined into a comprehensive, holistic programme of assessment. We used a qualitative approach and interviewed students (n = 17) in the pre-clinical phase of the programme about their perceptions of programmatic assessment and learning approaches. Data were scrutinised using theory-based thematic analysis.

RESULTS

Elements from the comprehensive programme of assessment, such as feedback, portfolios, assessments and assignments, were found to have both supporting and inhibiting effects on learning. These supporting and inhibiting elements influenced students' construction of learning. Findings showed that: (i) students perceived formative assessment as summative; (ii) programmatic assessment was an important trigger for learning, and (iii) the portfolio's reflective activities were appreciated for their generation of knowledge, the lessons drawn from feedback, and the opportunities for follow-up. Some students, however, were less appreciative of reflective activities. For these students, the elements perceived as inhibiting seemed to dominate the learning response.

CONCLUSIONS

The active participation of learners in their own learning is possible when learning is supported by programmatic assessment. Certain features of the comprehensive programme of assessment were found to influence student learning, and this influence can either support or inhibit students' learning responses.

Exploring frontline faculty perspectives after a curriculum change

CONTEXT

Curriculum renewal is an essential and continual process for undergraduate medical education programmes. Although there is substantial literature on the critical role of leadership in successful curricular change, the voices of frontline faculty teachers implementing such change have not been explored. We aimed not only to examine and understand the perceptions of faculty members as they face curriculum change, but also to explore the influences on their engagement with change.

METHODS

We used a constructivist grounded approach in this exploratory study. Sixteen faculty members teaching in the pre-clinical years were interviewed on their perspectives on a recent curricular change in the undergraduate medical programme at a single Canadian medical school. Constant comparative analysis was conducted to identify recurring themes.

RESULTS

Faculty teachers' engagement with curriculum change was influenced by three critical tensions during three phases of the change: (i) tension between individual and institutional values, which was prominent as change was being introduced; (ii) tension between drivers of change and restrainers of change, which was prominent as change was being enacted, and (iii) tension between perceived gains and perceived losses, which was prominent as teachers reflected on change once implemented.

CONCLUSIONS

We propose a model of faculty engagement with curricular change that elucidates the need to consider individual experiences and motivations within the broader context of the institutional culture of medical schools. Importantly, if individual and institutional values are misaligned, barriers to change outweigh facilitators, or perceived losses prevail; subsequently faculty teachers' engagement may be threatened, exposing the medical education programme to risk.

Perceived effects of innovations in postgraduate medical education: a Q study focusing on workplace-based assessment

PURPOSE

Anticipating users' perceptions of the effects an innovation will have in daily practice prior to implementation may lead to a more optimal innovation process. In this study, the authors aimed to identify the kinds of perceptions that exist concerning the effects of workplace-based assessment (WBA), an innovation that is widely used in medical education, among its users.

METHOD

In 2012, the authors used Q methodology to ascertain the principal user perceptions of effects of WBA in practice. Participating obstetrics-gynecology residents and attending physicians (including residency program directors) at six hospitals in the Netherlands performed individual Q sorts to rank 36 statements concerning WBA and WBA tools according to their level of agreement. The authors conducted by-person factor analysis to uncover patterns in the rankings of the statements. They used the statistical results and participant comments about their sorts to interpret and describe distinct perceptions.

RESULTS

The analysis of 65 Q sorts (completed by 22 residents and 43 attendings) identified five distinct user perceptions regarding the effects of WBA in practice, which the authors labeled enthusiasm, compliance, effort, neutrality, and skepticism. These perceptions were characterized by differences in views on three main issues: the intended goals of the innovation, its applicability (ease of applying it to practice), and its actual impact.

CONCLUSIONS

User perceptions of the effects of innovations in medical education can be typified and should be anticipated. This study's insights into five principal user perceptions can support the design and implementation of innovations in medical education.

Evaluation of a teaching strategy based on integration of clinical subjects, virtual autopsy, pathology museum, and digital microscopy for medical students

Background:

Learning pathology is fundamental for a successful medical practice. In recent years, medical education has undergone a profound transformation toward the development of an integrated curriculum incorporating both basic science and clinical material. Simultaneously, there has been a shift from a magisterial teaching approach to one centered around problem-based learning. Now-a-days, informatics tools are expected to help better implement these strategies.

Aim:

We applied and evaluated a new teaching method based on an active combination of clinical problems, gross pathology, histopathology, and autopsy pathology, all given through informatics tools, to teach a group of medical students at the Universidad de Santander, Colombia.

Design:

Ninety-four medical students were followed in two consecutive semesters. Students were randomized to receive teaching either through traditional methodology or through the new integrated approach.

Results:

There was no significant difference between the intervention group and the control group at baseline. At the end of the study, the scores in the intervention group were significantly higher compared to the control group (3.91/5.0 vs. 3.33/5.0, P = 0.0008). Students and tutors endorsed the benefits of the integrated approach. Participants were very satisfied with this training approach and rated the program an 8.7 out of 10, on average.

Conclusion:

This study confirms that an integrated curriculum utilizing informatics systems provides an excellent opportunity to associate pathology with clinical medicine early in training of medical students. This can be possible with the use of virtual microscopy and digital imaging.