Assessment, feedback and the alchemy of learning

Exploring the Impact of Entrustable Professional Activities on Feedback Culture: A Qualitative Study of Anesthesiology Residents and Attendings

PURPOSE

Entrustable professional activities (EPAs) were introduced as a potential way to optimize workplace-based assessments. Yet, recent studies suggest that EPAs have not yet overcome all of the challenges to implementing meaningful feedback. The aim of this study was to explore the extent to which the introduction of EPAs via mobile app impacts feedback culture as experienced by anesthesiology residents and attending physicians.

METHOD

Using a constructivist grounded theory approach, the authors interviewed a purposive and theoretical sample of residents (n = 11) and attendings (n = 11) at the Institute of Anaesthesiology, University Hospital of Zurich, where EPAs had recently been implemented. Interviews took place between February and December 2021. Data collection and analysis were conducted iteratively. The authors used open, axial, and selective coding to gain knowledge and understanding on the interplay of EPAs and feedback culture.

RESULTS

Participants reflected on a number of changes in their day-to-day experience of feedback culture with the implementation of EPAs. Three main mechanisms were instrumental in this process: lowering the feedback threshold, change in feedback focus, and gamification. Participants felt a lower threshold to feedback seeking and giving and that the frequency of feedback conversations increased and tended to be more focused on a specific topic and shorter, while feedback content tended to focus more on technical skills and more attention was given to average performances. Residents indicated that the app-based approach fostered a game-like motivation to "climb levels," while attendings did not perceive a game-like experience.

CONCLUSIONS

EPAs may offer a solution to problems of infrequent occurrence of feedback and invite attention to average performances and technical competencies, but may come at the expense of feedback on nontechnical skills. This study suggests that feedback culture and feedback instruments have a mutually interacting influence on each other.

Assessment-Seeking Strategies: Navigating the Decision to Initiate Workplace-Based Assessment

Phenomenon: Competency-based medical education (CBME) relies on workplace-based assessment (WBA) to generate formative feedback (assessment for learning-AfL) and make inferences about competence (assessment of learning-AoL). When approaches to CBME rely on residents to initiate WBA, learners experience tension between seeking WBA for learning and for establishing competence. How learners resolve this tension may lead to unintended consequences for both AfL and AoL. We sought to explore the factors that impact both decisions to seek and not to seek WBA and use the findings to build a model of assessment-seeking strategy used by residents. In building this model we consider how the link between WBA and promotion or progression within a program impacts an individual's assessment-seeking strategy. Approach: We conducted 20 semi-structured interviews with internal medicine residents at Queen's University about the factors that influence their decision to seek or avoid WBA. Using grounded theory methodology, we applied a constant comparative analysis to collect data iteratively and identify themes. A conceptual model was developed to describe the interaction of factors impacting the decision to seek and initiate WBA. Findings: Participants identified two main motivations when deciding to seek assessments: the need to fulfill program requirements and the desire to receive feedback for learning. Analysis suggested that these motivations are often at odds with each other. Participants also described several moderating factors that impact the decision to initiate assessments, irrespective of the primary underlying motivation. These included resident performance, assessor factors, training program expectations, and clinical context. A conceptual framework was developed to describe the factors that lead to strategic assessment-seeking behaviors. Insights: Faced with the dual purpose of WBA in CBME, resident behavior in initiating assessment is guided by specific assessment-seeking strategies. Strategies reflect individual underlying motivations, influenced by four moderating factors. These findings have broad implications for programmatic assessment in a CBME context including validity considerations for assessment data used in summative decision-making including readiness for unsupervised practice.

What does "Timely" Mean to Residents? Challenging Feedback Assumptions in Postgraduate Education

Introduction

Current orthodoxy states that feedback should be timely and face-to-face, yet the optimal timing and mode of delivery for feedback is unclear. We explored what "optimal timing" means from residents' points of view as feedback providers and receivers, to ultimately inform strategies to optimize feedback in training.

Methods

As near-peers who have dual roles in both providing and receiving feedback, 16 subspecialty (PGY4 and 5) internal medicine residents were interviewed about their perceptions of the optimal timing and format of feedback. Using constructivist grounded theory, interviews were conducted and analyzed iteratively.

Results

Drawing on their experiences as providers and recipients, residents described simultaneously considering and weighing multiple factors when deciding on when and how to provide feedback. These included their own readiness to engage in providing meaningful feedback, the perceived receptiveness of the learner and the apparent urgency of feedback delivery (e.g., if patient safety was at stake). Face-to-face verbal feedback was valued for encouraging dialogue but could be uncomfortable and limited by time constraints. Written feedback could be more honest and concise, and the possibility of asynchronous delivery had potential to overcome issues with timing and discomfort.

Discussion

Participants' perceptions of the optimal timing of feedback challenge current assumptions about the benefits of "immediate" versus "delayed". The concept of "optimal timing" for feedback was found to be complex and context-dependent, defying a formulaic approach. There may be a role for asynchronous and/or written feedback, which has potential to address unique issues identified issues in near-peer relationships.

A qualitative study examining the critical differences in the experience of and response to formative feedback by undergraduate medical students in Japan and the UK

BACKGROUND

Formative feedback plays a critical role in guiding learners to gain competence, serving as an opportunity for reflection and feedback on their learning progress and needs. Medical education in Japan has historically been dominated by a summative paradigm within assessment, as opposed to countries such as the UK where there are greater opportunities for formative feedback. How this difference affects students' interaction with feedback has not been studied. We aim to explore the difference in students' perception of feedback in Japan and the UK.

METHODS

The study is designed and analysed with a constructivist grounded theory lens. Medical students in Japan and the UK were interviewed on the topic of formative assessment and feedback they received during clinical placements. We undertook purposeful sampling and concurrent data collection. Data analysis through open and axial coding with iterative discussion among research group members was conducted to develop a theoretical framework.

RESULTS

Japanese students perceived feedback as a model answer provided by tutors which they should not critically question, which contrasted with the views of UK students. Japanese students viewed formative assessment as an opportunity to gauge whether they are achieving the pass mark, while UK students used the experience for reflective learning.

CONCLUSIONS

The Japanese student experience of formative assessment and feedback supports the view that medical education and examination systems in Japan are focused on summative assessment, which operates alongside culturally derived social pressures including the expectation to correct mistakes. These findings provide new insights in supporting students to learn from formative feedback in both Japanese and UK contexts.

Teacher feedback-based collaborative testing improves students' knowledge gaps of parasitology

Collaborative testing has been demonstrated the ability to improve students' performance, enhance students' learning, and aid in knowledge retention in many different courses. However, this examination mode lacks the process of teacher feedback. Herein, a short teacher feedback from was added immediately after the collaborative testing to improve the students' performance. A parasitology class of 121 undergraduates was randomized into two groups: group A and group B. Collaborative testing was carried out at the end of theoretical teaching. During the test, students would first answer questions as individuals for 20 minutes. Then, students from group A answered the same questions in groups (5 students in each group) for 20 minutes, while the group-testing duration was only 15 minutes in group B. Immediately after the group testing, teachers conducted a 5-minute feedback about the morphology identification according to the analysis of the answers by group B. Four weeks later, a final test was conducted in an individual test. The total scores and scores for each examination content were analyzed. The results showed that there was no significant difference in the final exam scores between both groups (t = -1.278, P = 0.204). However, the morphological and diagnostic test results of the final examination in group B were significantly higher than those of the midterm examination, while there was no significant change in group A (t = 4.333, P = 0.051). The results confirmed that the teacher feedback after the collaborative testing can effectively make up for the students' knowledge gaps.NEW & NOTEWORTHY This study found that collaborative group testing is helpful for teachers to grasp students' knowledge gaps more easily and the teacher feedback after the collaborative group testing can effectively make up for the knowledge gaps of students.

How workplace-based assessments guide learning in postgraduate education: A scoping review

INTRODUCTION

Competency-based medical education (CBME) led to the widespread adoption of workplace-based assessment (WBA) with the promise of achieving assessment for learning. Despite this, studies have illustrated tensions between the summative and formative role of WBA which undermine learning goals. Models of workplace-based learning (WBL) provide insight, however, these models excluded WBA. This scoping review synthesizes the primary literature addressing the role of WBA to guide learning in postgraduate medical education, with the goal of identifying gaps to address in future studies.

METHODS

The search was applied to OVID Medline, Web of Science, ERIC and CINAHL databases, articles up to September 2020 were included. Titles and abstracts were screened by two reviewers, followed by a full text review. Two members independently extracted and analysed quantitative and qualitative data using a descriptive-analytic technique rooted in Billett's four premises of WBL. Themes were synthesized and discussed until consensus.

RESULTS

All 33 papers focused on the perception of learning through WBA. The majority applied qualitative methodology (70%), and 12 studies (36%) made explicit reference to theory. Aligning with Billett's first premise, results reinforce that learning always occurs in the workplace. WBA helped guide learning goals and enhanced feedback frequency and specificity. Billett's remaining premises provided an important lens to understand how tensions that existed in WBL have been exacerbated with frequent WBA. As individuals engage in both work and WBA, they are slowly transforming the workplace. Culture and context frame individual experiences and the perceived authenticity of WBA. Finally, individuals will have different goals, and learn different things, from the same experience.

CONCLUSION

Analysing WBA literature through the lens of WBL theory allows us to reframe previously described tensions. We propose that future studies attend to learning theory, and demonstrate alignment with philosophical position, to advance our understanding of assessment-for-learning in the workplace.

"A whole other competence story": exploring faculty perspectives on the process of workplace-based assessment of entrustable professional activities

The centrality of entrustable professional activities (EPAs) in competency-based medical education (CBME) is predicated on the assumption that low-stakes, high-frequency workplace-based assessments used in a programmatic approach will result in accurate and defensible judgments of competence. While there have been conversations in the literature regarding the potential of this approach, only recently has the conversation begun to explore the actual experiences of clinical faculty in this process. The purpose of this qualitative study was to explore the process of EPA assessment for faculty in everyday practice. We conducted 18 semi-structured interviews with Anesthesia faculty at a Canadian academic center. Participants were asked to describe how they engage in EPA assessment in daily practice and the factors they considered. Interviews were audio-recorded, transcribed, and analysed using the constant comparative method of grounded theory. Participants in this study perceived two sources of tension in the EPA assessment process that influenced their scoring on official forms: the potential constraints of the assessment forms and the potential consequences of their assessment outcome. This was particularly salient in circumstances of uncertainty regarding the learner's level of competence. Ultimately, EPA assessment in CBME may be experienced as higher-stakes by faculty than officially recognized due to these tensions, suggesting a layer of discomfort and burden in the process that may potentially interfere with the goal of assessment for learning. Acknowledging and understanding the nature of this burden and identifying strategies to mitigate it are critical to achieving the assessment goals of CBME.

Digital feedback during clinical education in the emergency medical services: a qualitative study

BACKGROUND

Clinical education is essential for students' progress towards becoming registered nurses (RN) in Sweden. Assessment of caring skills in the Emergency Medical Services (EMS) is complex due to the ever-changing scenarios and the fact that multiple supervisors are involved in the student's education. Currently, assessments of student's skills are summative and occur twice during the six weeks of clinical education. A digitalized assessment tool (DAT) with an adaptation for formative assessment is a new approach to assessment of nursing skills in the EMS. Since new technologies and changes in procedures are likely to affect both students and supervisors, our aim in this study is to describe students' and clinical supervisors' experience of formative assessments using DAT in the EMS.

METHOD

This study is qualitative, using semi-structured group interviews (N = 2) with students and semi-structured individual telephone interviews (N = 13) with supervisors. The data was analysed according to Graneheim and Landman's method for content analysis. This analysis generated 221 codes organized into 10 categories within which three themes were identified. The students in this study were nursing students in their last semester and all supervisors were experienced RNs.

RESULTS

The results showed that students and supervisors had mainly positive views of the DAT and the formative assessment stating that the information they provided while using the DAT offered opportunities for reflection. The DAT supported the students' learning by visualizing strengths and areas of improvement, as well as displaying progress using a Likert scale. The application improved communication, but additional features linking the assessment tool with the university were requested. The application contributed to transparency in the assessments and was seen as preferable to the traditional 'pen and paper' method.

CONCLUSION

A digital system was described in a positive manner, and the assessment using the DAT facilitated reflection and formative assessment. The use of a Likert scale was considered positive in order to demonstrate progression which with advantage could be demonstrated visually.

Implications of Entrustable Professional Activities for Motivation and Learning

INTRODUCTION

This study sought to explore the experience of entrustable professional activity-based assessment and its effect on self-determination, a well-supported measure of motivation. The goal was to understand EPA-based assessment's implications for student motivation, learning, and wellbeing.

METHODS

A qualitative case study design was utilized through a standardized simulation activity with EPA-based assessment. An analysis of interviews and reflection journals was completed by the coding of emergent themes and significant statements with a subsequent thematic analysis framed in self-determination theory (SDT).

RESULTS

Entrustment was found to be an affectively and interpersonally impactful metric that provided the learner with a clear understanding of clinical performance. Quality entrustment feedback supported participants' sense of competence and increased their confidence in their ability to be autonomous as well as a valuable part of the healthcare team.

DISCUSSION

EPAs appeared to support learners' sense of competence and to provide a bridge to support autonomy and the ability to relate to others. To the extent that entrustment supported learners' sense of competence, other SDT propositions could be subsequently bolstered as well. Self-determination and intrinsic motivation were uniquely supported by EPA-based assessment, if implemented effectively. Further research is needed to better define effective implementation, faculty development, and feedback strategies.

A phenomenology of direct observation in residency: Is Miller's 'does' level observable?

INTRODUCTION

Guidelines on direct observation (DO) present DO as an assessment of Miller's 'does' level, that is, the learner's ability to function independently in clinical situations. The literature, however, indicates that residents may behave 'inauthentically' when observed. To minimise this 'observer effect', learners are encouraged to 'do what they would normally do' so that they can receive feedback on their actual work behaviour. Recent phenomenological research on patients' experiences with DO challenges this approach; patients needed-and caused-some participation of the observing supervisor. Although guidelines advise supervisors to minimise their presence, we are poorly informed on how some deliberate supervisor participation affects residents' experience in DO situations. Therefore, we investigated what residents essentially experienced in DO situations.

METHODS

We performed an interpretive phenomenological interview study, including six general practice (GP) residents. We collected and analysed our data, using the four phenomenological lenses of lived body, lived space, lived time and lived relationship. We grouped our open codes by interpreting what they revealed about common structures of residents' pre-reflective experiences.

RESULTS

Residents experienced the observing supervisor not just as an observer or assessor. They also experienced them as both a senior colleague and as the patient's familiar GP, which led to many additional interactions. When residents tried to act as if the supervisor was not there, they could feel insecure and handicapped because the supervisor was there, changing the situation.

DISCUSSION

Our results indicate that the 'observer effect' is much more material than was previously understood. Consequently, observing residents' 'authentic' behaviour at Miller's 'does' level, as if the supervisor was not there, seems impossible and a misleading concept: misleading, because it may frustrate residents and cause supervisors to neglect patients' and residents' needs in DO situations. We suggest that one-way DO is better replaced by bi-directional DO in working-and-learning-together sessions.

How progress evaluations are used in postgraduate education with longitudinal supervisor-trainee relationships: a mixed method study

The combination of measuring performance and giving feedback creates tension between formative and summative purposes of progress evaluations and can be challenging for supervisors. There are conflicting perspectives and evidence on the effects supervisor-trainee relationships have on assessing performance. The aim of this study was to learn how progress evaluations are used in postgraduate education with longitudinal supervisor-trainee relationships. Progress evaluations in a two-year community-pharmacy specialization program were studied with a mixed-method approach. An adapted version of the Canadian Medical Education Directives for Specialists (CanMEDS) framework was used. Validity of the performance evaluation scores of 342 trainees was analyzed using repeated measures ANOVA. Semi-structured interviews were held with fifteen supervisors to investigate their response processes, the utility of the progress evaluations, and the influence of supervisor-trainee relationships. Time and CanMEDS roles affected the three-monthly progress evaluation scores. Interviews revealed that supervisors varied in their response processes. They were more committed to stimulating development than to scoring actual performance. Progress evaluations were utilized to discuss and give feedback on trainee development and to add structure to the learning process. A positive supervisor-trainee relationship was seen as the foundation for feedback and supervisors preferred the roles of educator, mentor, and coach over the role of assessor. We found that progress evaluations are a good method for directing feedback in longitudinal supervisor-trainee relationships. The reliability of scoring performance was low. We recommend progress evaluations to be independent of formal assessments in order to minimize roles-conflicts of supervisors.

Professional identity formation: linking meaning to well-being

Trainee distress and burnout continue to be serious concerns for educational programs in medicine, prompting the implementation of numerous interventions. Although an expansive body of literature suggests that the experience of meaning at work is critical to professional wellbeing, relatively little attention has been paid to how this might be leveraged in the educational milieu. We propose that professional identity formation (PIF), the process by which trainees come to not only attain competence, but additionally to "think, act and feel" like physicians, affords us a unique opportunity to ground trainees in the meaningfulness of their work. Using the widely accepted tri-partite model of meaning, we outline how this process can contribute to wellbeing. We suggest strategies to optimize the influence of PIF on wellbeing, offering curricular suggestions, as well as ideas regarding the respective roles of communities of practice, teachers, and formative educational experiences. Collectively, these encourage trainees to act as intentional agents in the making of their novel professional selves, anchoring them to the meaningfulness of their work, and supporting their short and long-term wellbeing.

'For the most part it works': Exploring how authors navigate peer review feedback

BACKGROUND

Peer review aims to provide meaningful feedback to research authors so that they may improve their work, and yet it constitutes a particularly challenging context for the exchange of feedback. We explore how research authors navigate the process of interpreting and responding to peer review feedback, in order to elaborate how feedback functions when some of the conditions thought to be necessary for it to be effective are not met.

METHODS

Using constructivist grounded theory methodology, we interviewed 17 recently published health professions education researchers about their experiences with the peer review process. Data collection and analysis were concurrent and iterative. We used constant comparison to identify themes and to develop a conceptual model of how feedback functions in this setting.

RESULTS

Although participants expressed faith in peer review, they acknowledged that the process was emotionally trying and raised concerns about its consistency and credibility. These potential threats were mitigated by factors including time, team support, experience and the exercise of autonomy. Additionally, the perceived engagement of reviewers and the cultural norms and expectations surrounding the process strengthened authors' willingness and capacity to respond productively. Our analysis suggests a model of feedback within which its perceived usefulness turns on the balance of threats and countermeasures.

CONCLUSIONS

Feedback is a balancing act. Although threats to the productive uptake of peer review feedback abound, these threats may be neutralised by a range of countermeasures. Among these, opportunities for autonomy and cultural normalisation of both the professional responsibility to engage with feedback and the challenge of doing so may be especially influential and may have implications beyond the peer review setting.

Walking the line between assessment, improvement and learning: a qualitative study on opportunities and risks of incorporating peer discussion of audit and feedback within quality improvement in general practice

OBJECTIVES

There is a broad call for change towards 'new era' quality systems in healthcare, in which the focus lies on learning and improving. A promising way to establish this in general practice care is to combine audit and feedback with peer group discussion. However, it is not known what different stakeholders think of this type of quality improvement. The aim of this research was to explore the opinions of different stakeholders in general practice on peer discussion of audit and feedback and on its opportunities and risks. Second, their thoughts on transparency versus accountability, regarding this system, were studied.

DESIGN

An exploratory qualitative study within a constructivist paradigm. Semistructured interviews and focus group discussions were held and coded using thematic analysis. Included stakeholders were general practitioners (GP), patients, professional organisations and insurance companies.

SETTING

General practice in the Netherlands.

PARTICIPANTS

22 participants were purposively sampled for eight interviews and two focus group discussions.

RESULTS

Three main opportunities of peer discussion of audit and feedback were identified: deeper levels of reflection on data, adding context to numbers and more ownership; and three main risks: handling of unwilling colleagues, lacking a safe group and the necessity of patient involvement. An additional theme concerned disagreement on the amount of transparency to be offered: insurance companies and patients advocated for complete transparency on data and improvement of outcomes, while GPs and professional organisations urged to restrict transparency to giving insight into the process.

CONCLUSIONS

Peer discussion of audit and feedback could be part of a change movement, towards a quality system based on learning and trust, that is initiated by the profession. Creating a safe learning environment and involving patients is key herein. Caution is needed when complete transparency is asked, since it could jeopardise practitioners' reflection and learning in safety.

Structured feedback: A teaching and learning tool for the postgraduate medical students in transfusion medicine

CONTEXT

Structured Feedback is a learning and assessment tool designed to provide feedback to students and educators to adjust learning and teaching during the training. Lack of provision of structured feedback to postgraduate (PG) medical students prompted us to plan a study to introduce a structured feedback module into the existing monthly assessment schedules in the Department of Transfusion Medicine.

AIM

This study aims to introduce a structured feedback module and evaluate its efficacy after incorporation into the existing monthly assessment schedules for the PG students in the Department of Transfusion Medicine.

DESIGN AND SETTING

A quasi-experimental study was commenced after obtaining clearance from the Institutional Ethics Committee in the Department of Transfusion Medicine for the students pursuing postgraduation in Transfusion Medicine.

METHODOLOGY

A peer-validated feedback module was designed and implemented for MD students by the core team faculty. The students underwent the structured feedback sessions after each monthly assessment for of 3 months. One on one, verbal feedback was conducted using Pendleton's method, for monthly online assessment for the learning that happened during study period.

DATA COLLECTION AND STATISTICAL ANALYSIS

The data were collected from the open-ended and closed-ended questions using Google form-based Student/Faculty perception and students' pre-post self-efficacy questionnaires on 5-point Likert Scale and the quantitative data analysis was done using percentage of Likert scores, median values for each item for pre-and post-responses and comparison using nonparametric test - Wilcoxon signed-rank test. The qualitative data analysis was done using thematic analysis from the open ended questions.

RESULTS

All (n = 9; 100%) the PG students strongly agreed and agreed (median score of 5 and 4) that the feedback they received made them aware of their learning gaps, enabled them in bridging those gaps and provided ample opportunity to interact with faculty. Both students and faculty agreed that the feedback session should be an ongoing and continuous process in the department.

CONCLUSION

Both the students as well as faculty were satisfied with the implementation of the feedback module in the department. Students reported awareness about the learning gaps, identification of appropriate study resources, and ample opportunity to interact with faculty, after taking the feedback sessions. The faculty felt satisfied on the acquisition of new skill for delivering structured feedback to students.

On enhancing students' cognitive abilities in online learning using brain activity and eye movements

The COVID-19 pandemic has interrupted education institutions in over 150 nations, affecting billions of students. Many governments have forced a transition in higher education from in-person to remote learning. After this abrupt, worldwide transition away from the classroom, some question whether online education will continue to grow in acceptance in post-pandemic times. However, new technology, such as the brain-computer interface and eye-tracking, have the potential to improve the remote learning environment, which currently faces several obstacles and deficiencies. Cognitive brain computer interfaces can help us develop a better understanding of brain functions, allowing for the development of more effective learning methodologies and the enhancement of brain-based skills. We carried out a systematic literature review of research on the use of brain computer interfaces and eye-tracking to measure students' cognitive skills during online learning. We found that, because many experimental tasks depend on recorded rather than real-time video, students don't have direct and real-time interaction with their teacher. Further, we found no evidence in any of the reviewed papers for brain-to-brain synchronization during remote learning. This points to a potentially fruitful future application of brain computer interfaces in education, investigating whether the brains of student-teacher pairs who interact with the same course content have increasingly similar brain patterns.

Informed self-assessment during the transition to medical school: a longitudinal narrative study

OBJECTIVES

To explore how medical students' narratives of informed self-assessment (ISA) change during their first 18 months of study.

DESIGN

This longitudinal study used student narratives drawn from qualitative interviews and written reflections during the transition to medical school, to examine changes in ISA. Our analysis was informed by Situated Cognition Theory which recognises the impact and interplay of personal and environmental factors in cognition.

SETTING

To study medicine, first year students need to adapt their self-regulated learning in the context of a new peer group, study demands and educational culture. During this adaptation, students need to seek and interpret available cues to inform their self-assessment.

PARTICIPANTS

Longitudinal data were collected at five time points over 18 months from a diverse sample of seven first year medical students in an undergraduate medical programme, including 13.5 hours of interviews and 12 written reflections.

RESULTS

Before and after starting medical school, the participants' self-assessments were informed by environmental influences (exam results and comparison with peers), and personal influences (fear of failure and anxiety about not belonging). Early uncertainty meant self-assessments were overestimated and underestimated.By the end of first year, an enhanced sense of belonging coincided with less fear of failure, less emphasis on exam performance and reduced competition with peers. Self-assessments became increasingly informed by evidence of clinical skills and knowledge gained related to future professional competence.

CONCLUSION

Influences on medical students' ISAs change during the transition to studying medicine. A greater sense of belonging, and evidence of progress towards clinical competence became more important to self-assessment than comparison with peers and exam performance. Our findings reinforce the importance of formative assessments, opportunities to study and socialise with peers and early clinical experiences during first year. These experiences enhance ISA skills during the transition to medical school.

The Development and Implementation of a National Veterinary Graduate Development Programme (VetGDP) to Support Veterinarians Entering the UK Workforce

The UK veterinary profession is facing significant challenges, including high rates of veterinarians leaving the profession amidst workforce shortages, alongside high levels of dissatisfaction, stress, and poor mental health. The highest rates of attrition are associated with veterinarians who have recently graduated and are at an early stage in their careers. Although there may be many contributory factors, a lack of adequate support during the transition from vet school into their first professional role following graduation may be one important cause. Consequently, it has never been more important to develop an effective system for supporting new graduates, which is accessible to all. A new Veterinary Graduate Development Programme (VetGDP) has been developed, using a framework of professional activities that are sufficiently flexible to create a bespoke, individualized program for each graduate depending on the role they enter. Each new graduate is assigned a dedicated coach (Adviser) within their workplace, who has been trained to provide effective support and has committed to doing so throughout the program. VetGDP has been implemented on a national scale in the UK; engagement is assured through the RCVS Code of Professional Conduct and quality assurance is in place. VetGDP has been developed using established educational, sociocultural, and behavioral theories, and the latest research in coaching and feedback within medical education. These methods, which aim to ensure there is the best possible impact on graduates' professional development, and the creation of a positive learning culture within the workplace, are taught to all Advisers via an e-learning package.

Teacher, Gatekeeper, or Team Member: supervisor positioning in programmatic assessment

Competency-based assessment is undergoing an evolution with the popularisation of programmatic assessment. Fundamental to programmatic assessment are the attributes and buy-in of the people participating in the system. Our previous research revealed unspoken, yet influential, cultural and relationship dynamics that interact with programmatic assessment to influence success. Pulling at this thread, we conducted secondary analysis of focus groups and interviews (n = 44 supervisors) using the critical lens of Positioning Theory to explore how workplace supervisors experienced and perceived their positioning within programmatic assessment. We found that supervisors positioned themselves in two of three ways. First, supervisors universally positioned themselves as a Teacher, describing an inherent duty to educate students. Enactment of this position was dichotomous, with some supervisors ascribing a passive and disempowered position onto students while others empowered students by cultivating an egalitarian teaching relationship. Second, two mutually exclusive positions were described-either Gatekeeper or Team Member. Supervisors positioning themselves as Gatekeepers had a duty to protect the community and were vigilant to the detection of inadequate student performance. Programmatic assessment challenged this positioning by reorientating supervisor rights and duties which diminished their perceived authority and led to frustration and resistance. In contrast, Team Members enacted a right to make a valuable contribution to programmatic assessment and felt liberated from the burden of assessment, enabling them to assent power shifts towards students and the university. Identifying supervisor positions revealed how programmatic assessment challenged traditional structures and ideologies, impeding success, and provides insights into supporting supervisors in programmatic assessment.

Predictors of faculty narrative evaluation quality in medical school clerkships

INTRODUCTION

Narrative approaches to assessment provide meaningful and valid representations of trainee performance. Yet, narratives are frequently perceived as vague, nonspecific and low quality. To date, there is little research examining factors associated with narrative evaluation quality, particularly in undergraduate medical education. The purpose of this study was to examine associations of faculty- and student-level characteristics with the quality of faculty member's narrative evaluations of clerkship students.

METHODS

The authors reviewed faculty narrative evaluations of 50 students' clinical performance in their inpatient medicine and neurology clerkships, resulting in 165 and 87 unique evaluations in the respective clerkships. The authors evaluated narrative quality using the Narrative Evaluation Quality Instrument (NEQI). The authors used linear mixed effects modelling to predict total NEQI score. Explanatory covariates included the following: time to evaluation completion, number of weeks spent with student, faculty total weeks on service per year, total faculty years in clinical education, student gender, faculty gender, and an interaction term between student and faculty gender.

RESULTS

Significantly higher narrative evaluation quality was associated with a shorter time to evaluation completion, with NEQI scores decreasing by approximately 0.3 points every 10 days following students' rotations (p = .004). Additionally, women faculty had statistically higher quality narrative evaluations with NEQI scores 1.92 points greater than men faculty (p = .012). All other covariates were not significant.

CONCLUSIONS

The quality of faculty members' narrative evaluations of medical students was associated with time to evaluation completion and faculty gender but not faculty experience in clinical education, faculty weeks on service, or the amount of time spent with students. Findings advance understanding on ways to improve the quality of narrative evaluations which are imperative given assessment models that will increase the volume and reliance on narratives.

Beyond Competency: A Student Perspective on Growth Through Clerkship Feedback

This article proposes a paradigm shift from the competency-based model of clerkship feedback using checklists to a coaching-based, action plan-oriented process that centers on individualized student-oriented goals. Using a student perspective, the authors examine the feedback literature and put forward a proposal to use an impact model whose emphasis is to improve the learning climate for students. Several techniques are reviewed which include goal generation and creation of dynamic action plans. By intentionally focusing on coaching relationships as a platform for feedback, the learners and mentors share goals and the result of feedback becomes action-based behaviors which may help negate personal attribution and bias in the feedback process.

'Next steps are…': An exploration of coaching and feedback language in EPA assessment comments

PURPOSE

Entrustable Professional Activities (EPA) assessments are intended to facilitate meaningful, low-stakes coaching and feedback, partly through the provision of written comments. We sought to explore EPA assessment comments provided to internal medicine (IM) residents for evidence of feedback and coaching language as well as politeness.

METHODS

We collected all written comments from EPA assessments of communication from a first-year IM resident cohort at the University of Toronto. Sensitized by politeness theory, we analyzed data using principles of constructivist grounded theory.

RESULTS

Nearly all EPA assessments (94%) contained written feedback based on focused clinical encounters. The majority of comments demonstrated coaching language, including phrases like 'don't forget to,' and 'next steps are,' followed by specific suggestions for improvement. A variety of words, including 'autonomy' and 'independence' denoted entrustment decisions. Linguistic politeness strategies such as hedging were pervasive, seemingly to minimize harm to the supervisor-trainee relationship.

CONCLUSION

Evidence of written coaching feedback suggests that EPA assessment comments are being used as intended as a means of formative feedback to promote learning. Yet, the frequent use of polite language suggests that EPAs may be higher-stakes than expected, highlighting a need for changes to the assessment culture and improved feedback literacy.

Understanding the near-peer relationship: resident perspectives around a novel on-call workplace-based assessment

BACKGROUND

Workplace-based assessment (WBA) is a critical component of competency-based medical education (CBME), though literature on WBA for overnight call is limited. We evaluated a WBA tool completed by supervising subspecialty trainees on paediatric residents during subspecialty overnight call, for usefulness facilitating feedback/coaching in this setting.

METHODS

Web-based surveys were sent to residents pre- and post-WBA tool implementation monthly for four months (August-December 2018), exploring feedback frequency, Likert-scaled opinions of tool feasibility/usefulness facilitating feedback, and qualitative experiences. Assessor comments were categorized as actionable/non-actionable. Quantitative data was summarized using descriptive statistics. Qualitative data was coded to identify themes.

RESULTS

Total response rates averaged 41% (total 25 responses, average five respondents/12 residents on-call each month). Post-implementation (n = 16 responses), a non-sustained trend of increased Medical Expert feedback was observed. Residents were generally divided or disagreed on tool usefulness facilitating feedback and feasibility. Comments contained actionable feedback in < 10% of completed WBAs. Qualitative analysis revealed barriers to tool-facilitated coaching including: feedback quality and setting/environment, role of senior near-peer as assessor, interpersonal burden in encounters, and tool-specific issues.

CONCLUSIONS

Increasing frequency of WBA tool completion is not sufficient to achieve CBME goals. Factors impacting feedback/coaching within the resident/near-peer dyad must be addressed.

Validity evidence for the Quality of Assessment for Learning score: a quality metric for supervisor comments in Competency Based Medical Education

BACKGROUND

Competency based medical education (CBME) relies on supervisor narrative comments contained within entrustable professional activities (EPA) for programmatic assessment, but the quality of these supervisor comments is unassessed. There is validity evidence supporting the QuAL (Quality of Assessment for Learning) score for rating the usefulness of short narrative comments in direct observation.

OBJECTIVE

We sought to establish validity evidence for the QuAL score to rate the quality of supervisor narrative comments contained within an EPA by surveying the key end-users of EPA narrative comments: residents, academic advisors, and competence committee members.

METHODS

In 2020, the authors randomly selected 52 de-identified narrative comments from two emergency medicine EPA databases using purposeful sampling. Six collaborators (two residents, two academic advisors, and two competence committee members) were recruited from each of four EM Residency Programs (Saskatchewan, McMaster, Ottawa, and Calgary) to rate these comments with a utility score and the QuAL score. Correlation between utility and QuAL score were calculated using Pearson's correlation coefficient. Sources of variance and reliability were calculated using a generalizability study.

RESULTS

All collaborators (n = 24) completed the full study. The QuAL score had a high positive correlation with the utility score amongst the residents (r = 0.80) and academic advisors (r = 0.75) and a moderately high correlation amongst competence committee members (r = 0.68). The generalizability study found that the major source of variance was the comment indicating the tool performs well across raters.

CONCLUSION

The QuAL score may serve as an outcome measure for program evaluation of supervisors, and as a resource for faculty development.

Facilitating Residents' Understanding of Electronic Health Record Report Card Data Using Faculty Feedback and Coaching

PURPOSE

Feedback continues to present a challenge for competency-based medical education. Clear, consistent, and credible feedback is vital to supporting one's ongoing development, yet it can be difficult to gather clinical performance data about residents. This study sought to determine whether providing residents with electronic health record (EHR)-based report cards, as well as an opportunity to discuss these data with faculty trained using the R2C2 model, can help residents understand and interpret their clinical performance metrics.

METHOD

Using action research methodology, the author team collected EHR data from July 2017 to February 2020, for all residents (n = 21) in one 5-year Emergency Medicine program and created personalized report cards for each resident. During October 6-17, 2020, 8 out of 17 eligible residents agreed to have their feedback conversations recorded and participate in a subsequent interview with a nonphysician member of the research team. Data were analyzed using thematic analysis, and the authors used inductive analysis to identify themes in the data.

RESULTS

In analyzing both the feedback conversations as well as the individual interviews with faculty and residents, the authors identified 2 main themes: (1) Reactions and responses to receiving personalized EHR data and (2) The value of EHR data for assessment and feedback purposes. All participants believed that EHR data metrics are useful for prompting self-reflection, and many pointed to their utility in providing suggestions for actionable changes in their clinical practice. For faculty, having a tool through which underperforming residents can be shown "objective" data about their clinical performance helps underscore the need for improvement, particularly when residents are resistant.

CONCLUSIONS

The EHR is a valuable source of educational data, and this study demonstrates one of the many thoughtful ways it can be used for assessment and feedback purposes.

Assessment for Learning with Ungraded and Graded Assessments

Introduction

Assessment for learning has many benefits, but learners will still encounter high-stakes decisions about their performance throughout training. It is unknown if assessment for learning can be promoted with a combination model where scores from some assessments are factored into course grades and scores from other assessments are not used for course grading.

Methods

At the University of Utah School of Medicine, year 1-2 medical students (MS) completed multiple-choice question quiz assessments and final examinations in six systems-based science courses. Quiz and final examination performance counted toward course grades for MS2017-MS2018. Starting with the MS2020 cohort, quizzes no longer counted toward course grades. Quiz, final examination, and Step 1 scores were compared between ungraded quiz and graded quiz cohorts with independent samples t-tests. Student and faculty feedback was collected.

Results

Quiz performance was not different for the ungraded and graded cohorts (p = 0.173). Ungraded cohorts scored 4% higher on final examinations than graded cohorts (p ≤ 0.001, d = 0.88). Ungraded cohorts scored above the national average and 11 points higher on Step 1 compared to graded cohorts, who had scored below the national average (p ≤ 0.001, d = 0.64). During the study period, Step 1 scores increased by 2 points nationally. Student feedback was positive, and faculty felt it improved their relationship with students.

Discussion

The change to ungraded quizzes did not negatively affect final examination or Step 1 performance, suggesting a combination of ungraded and graded assessments can effectively promote assessment for learning.

Gender differences in emergency medicine resident assessment: A scoping review

Background

Growing literature within postgraduate medical education demonstrates that female resident physicians experience gender bias throughout their training and future careers. This scoping review aims to describe the current body of literature on gender differences in emergency medicine (EM) resident assessment.

Methods

We conducted a scoping review which adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. We included research involving resident physicians or fellows in EM (population and context), which focused on the impact of gender on assessments (concept). We searched seven databases from the databases' inception to April 4, 2022. Two reviewers independently screened citations, completed full-text review, and abstracted data. A third reviewer resolved any discrepancies.

Results

A total of 667 unique citations were identified; 10 studies were included, and all were conducted within the United States. Four studies reported differences in EM resident assessments attributable to gender within workplace-based assessments (qualitative comments and quantitative scores) by both attending physicians and nonphysicians. Six studies investigating clinical competency committee scores, procedural scores, and simulation-based assessments did not report any significant differences attributable to gender.

Conclusions

This scoping review found that gender bias exists within EM resident assessment most notably at the level of narrative comments typically received via workplace-based assessments. As female EM residents receive higher rates of negative or critical comments and discordant feedback documented on assessment, these findings raise concern about added barriers female EM residents may face while progressing through residency and the impact on their clinical and professional development.

Using Learning Analytics to Examine Differences in Assessment Forms From Continuous Versus Episodic Supervisors of Family Medicine Residents

BACKGROUND

It is assumed that there is a need for continuity of supervision within competency-based medical education, despite most evidence coming from the undergraduate medical education rather than the graduate medical education (GME) context. This evidence gap must be addressed to justify the time and effort needed to redesign GME programs to support continuity of supervision.

OBJECTIVE

To examine differences in assessment behaviors of continuous supervisors (CS) versus episodic supervisors (ES), using completed formative assessment forms, FieldNotes, as a proxy.

METHODS

The FieldNotes CS- and ES-entered for family medicine residents (N=186) across 3 outpatient teaching sites over 3 academic years (2015-2016, 2016-2017, 2017-2018) were examined using 2-sample proportion z-tests to determine differences on 3 FieldNote elements: competency (Sentinel Habit [SH]), Clinical Domain (CD), and Progress Level (PL).

RESULTS

Sixty-nine percent (6104 of 8909) of total FieldNotes were analyzed. Higher proportions of CS-entered FieldNotes indicated SH3 (Managing patients with best practices), z=-3.631, P<.0001; CD2 (Care of adults), z=-8.659, P<.0001; CD3 (Care of the elderly), z=-4.592, P<.0001; and PL3 (Carry on, got it), z=-4.482, P<.0001. Higher proportions of ES-entered FieldNotes indicated SH7 (Communication skills), z=4.268, P<.0001; SH8 (Helping others learn), z=20.136, P<.0001; CD1 (Doctor-patient relationship/ethics), z=14.888, P<.0001; CD9 (Not applicable), z=7.180, P<.0001; and PL2 (In progress), z=5.117, P<.0001.

CONCLUSIONS

The type of supervisory relationship impacts assessment: there is variability in which competencies are paid attention to, which contexts or populations are included, and which progress levels are chosen.

A qualitative study of medical students' perceptions of resident feedback

CONTEXT

Residents play a pivotal role in medical students' clinical education. From a feedback lens, the near-peer relationship between student and resident holds the potential to foster an educational alliance that could influence learning. We undertook the current qualitative study to explore medical students' perceptions of feedback experiences with residents, addressing when, how and why (and conversely when not and why not) resident feedback plays a role in their clinical education.

METHODS

Our methodology was qualitative interpretive description, informed by phenomenology. We conducted 24 semi-structured interviews with third and fourth year medical students at one institution. The interviews aimed to foster rich discussion about students' feedback experiences with residents during clinical rotations. Data collection and analysis proceeded iteratively. Initial interviews were independently open-coded by three investigators and then collaboratively refined. Codes were applied to subsequent interviews, and new codes were developed. During the final stages of analysis, we organised our themes by drawing on a sociocultural perspective to examine students' perceptions of relationship-building with residents and when and how this influenced feedback and learning.

RESULTS

From the students' perspectives, when residents contributed to building interpersonal relationships with students, this in turn influenced students' receptivity to both encouraging and constructive feedback conversations. In the context of resident-student relationships that were perceived as supportive, resident feedback influenced how students approached learning and working in the clinical environment, as well as students' visions of their future selves. In unsupportive relationships, students were less inclined to engage in feedback with residents and students noted resident behaviours that they wanted to avoid in themselves.

CONCLUSION

Residents are uniquely positioned to create a strong educational alliance with students in which feedback conversations can flourish. Focusing educational efforts on resident feedback conversations has the potential to significantly impact the feedback culture of our clinical environments.

Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships

PURPOSE

To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation.

METHOD

This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system.

RESULTS

Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning.

CONCLUSIONS

Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.

Student perspectives on programmatic assessment in a large medical programme: A critical realist analysis

BACKGROUND

Fundamental challenges exist in researching complex changes of assessment practice from traditional objective-focused 'assessments of learning' towards programmatic 'assessment for learning'. The latter emphasise both the subjective and social in collective judgements of student progress. Our context was a purposively designed programmatic assessment system implemented in the first year of a new graduate entry curriculum. We applied critical realist perspectives to unpack the underlying causes (mechanisms) that explained student experiences of programmatic assessment, to optimise assessment practice for future iterations.

METHODS

Data came from 14 in-depth focus groups (N = 112/261 students). We applied a critical realist lens drawn from Bhasker's three domains of reality (the actual, empirical and real) and Archer's concept of structure and agency to understand the student experience of programmatic assessment. Analysis involved induction (pattern identification), abduction (theoretical interpretation) and retroduction (causal explanation).

RESULTS

As a complex educational and social change, the assessment structures and culture systems within programmatic assessment provided conditions (constraints and enablements) and conditioning (acceptance or rejection of new 'non-traditional' assessment processes) for the actions of agents (students) to exercise their learning choices. The emergent underlying mechanism that most influenced students' experience of programmatic assessment was one of balancing the complex relationships between learner agency, assessment structures and the cultural system.

CONCLUSIONS

Our study adds to debates on programmatic assessment by emphasising how the achievement of balance between learner agency, structure and culture suggests strategies to underpin sustained changes (elaboration) in assessment practice. These include; faculty and student learning development to promote collective reflexivity and agency, optimising assessment structures by enhancing integration of theory with practice, and changing learning culture by both enhancing existing and developing new social structures between faculty and the student body to gain acceptance and trust related to the new norms, beliefs and behaviours in assessing for and of learning.

The Advice Given During Near-Peer Interactions Before and After Curriculum Change

PhenomenonNear-peer interactions (NPIs) provide formal and informal mentorship that can allow medical students to share strategies for successful training. Such capacity to convey valuable advice, however, may depend on the similitude of experiences. Given that many factors can disrupt homogeneity, including curriculum renewal, we must better understand whether junior trainees feel disadvantaged when they do not have more senior peers with similar experiences. This study was, therefore, conducted to examine the focus of, and engagement with, advice available through NPIs during curriculum renewal. Approach: We used a generic exploratory qualitative research approach. Twenty MD undergraduate students, seven from the Class of 2019 (the first cohort post-curriculum change), and thirteen from the Class of 2020 (the first cohort with access to more senior students in the new curriculum), participated in semi-structured interviews. Anonymized transcriptions were analyzed with open, axial, and selective coding to generate themes until saturation was attained. Findings: Participants from the Class of 2019 reported having particularly few reasons to seek advice; because curriculum renewal disrupted their near peers' capacity to provide critical insights, students exerted little effort to learn from them. That said, this vacuum was not generally cause for concern. Deeper probing illustrated why: advice given during NPIs in both classes more commonly focused on nonacademic (e.g., work-life balance issues) than academic advice; academic advice, when sought or offered, tended not to be aimed at improving understanding of curriculum dependent content; and, while students in both classes welcomed advice, both were wary of accepting it at face value, precluding a sense of dependence on senior peers. Insights: Students' skepticism about the overall utility of academic advice raises a number of important issues for medical education and training. Positively, it shielded students from feeling loss when advice from similarly trained students was not available, reducing concerns about disadvantage that could arise during periods of curriculum revision. On the other hand, knowing that what students perceive and what educators claim to be important aspects of training can be at odds and knowing that self-assessment is flawed makes it surprising and unsettling, respectively, that participants so readily treated the lessons learned by those who came before them as irrelevant.

An international study on the implementation of programmatic assessment: Understanding challenges and exploring solutions

INTRODUCTION

Programmatic assessment is an approach to assessment aimed at optimizing the learning and decision function of assessment. It involves a set of key principles and ground rules that are important for its design and implementation. However, despite its intuitive appeal, its implementation remains a challenge. The purpose of this paper is to gain a better understanding of the factors that affect the implementation process of programmatic assessment and how specific implementation challenges are managed across different programs.

METHODS

An explanatory multiple case (collective) approach was used for this study. We identified 6 medical programs that had implemented programmatic assessment with variation regarding health profession disciplines, level of education and geographic location. We conducted interviews with a key faculty member from each of the programs and analyzed the data using inductive thematic analysis.

RESULTS

We identified two major factors in managing the challenges and complexity of the implementation process: knowledge brokers and a strategic opportunistic approach. Knowledge brokers were the people who drove and designed the implementation process acting by translating evidence into practice allowing for real-time management of the complex processes of implementation. These knowledge brokers used a 'strategic opportunistic' or agile approach to recognize new opportunities, secure leadership support, adapt to the context and take advantage of the unexpected. Engaging in an overall curriculum reform process was a critical factor for a successful implementation of programmatic assessment.

DISCUSSION

The study contributes to the understanding of the intricacies of implementation processes of programmatic assessment across different institutions. Managing opportunities, adaptive planning, awareness of context, were all critical aspects of thinking strategically and opportunistically in the implementation of programmatic assessment. Future research is needed to provide a more in-depth understanding of values and beliefs that underpin the assessment culture of an organization, and how such values may affect implementation.

Does direct observation influence the quality of workplace-based assessment documentation?

BACKGROUND

A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect).

METHODS

The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis.

RESULTS

A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84.

CONCLUSIONS

Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

The Importance of Sociocultural Factors in Moderating the Applicability of Test-Enhanced Learning to Continuing Professional Development

Many studies have shown that repeated testing promotes learning, an effect known as test-enhanced learning. Although test-enhanced learning has been well-established within health care trainees, there are mixed findings of whether testing facilitates learning within continuing professional development (CPD) contexts. This piece argues that for testing to support learning in CPD, there is need to understand the various social, cultural, and structural factors that influence clinicians' willingness to participate in regular testing activities. The solution to this problem is conceptual in nature, whereby CPD programs must consider how learning cultures shape practicing clinicians' perspectives toward testing.

It's a 'two-way street': resident perspectives of effective coaching relationships in the clinical learning environment

BACKGROUND

Coaching has gained traction in postgraduate medical education to enhance interactions between residents and clinical teachers, but these relationships present unique challenges and tensions. In order to realize the promises of coaching in medical education, we must understand how coaching relationships can be enacted to optimize resident development. The purpose of this study was to explore residents' perceptions of key characteristics for effective clinical teacher-resident (CT-R) coaching relationships.

METHODS

We conducted four focus groups and eight interviews with residents at a Canadian academic center. Using a social constructionist approach, focus groups and interviews were audio recorded, transcribed verbatim, and thematically analyzed.

RESULTS

Residents described three main characteristics that contributed to effective CT-R coaching relationships: safe, meaningful, and collaborative. Residents emphasized that these characteristics needed to be bidirectional in nature to be most effective, in that both the resident and clinical teacher embodied these characteristics.

CONCLUSIONS

Residents identified that effective coaching relationships were shaped not only by clinical teacher behaviours, but importantly, the quality of the interpersonal relationship that was fostered. Thus, it is imperative to consider the bidirectional nature of the CT-R coaching relationship when striving to enhance resident development.

A Mixed-Methods Program Evaluation of a Self-directed Learning Panel Management Curriculum in an Internal Medicine Residency Clinic

BACKGROUND

Panel management (PM) curricula in internal medicine (IM) residency programs often assign performance measures which may not address the varied interests or needs of resident-learners.

AIM

To evaluate a self-directed learning (SDL)-based PM curriculum.

SETTING

University-based primary care practice in Burlington, Vermont.

PARTICIPANTS

Thirty-five internal medicine residents participated.

PROGRAM DESCRIPTION

Residents completed a PM curriculum that integrated SDL, electronic health record (EHR)-driven performance feedback, mentorship, and autonomy to set learning and patient care goals.

PROGRAM EVALUATION

Pre/post-curricular surveys assessed EHR tool acceptability, weekly curricular surveys and post-curricular focus groups assessed resident perceptions and goals, and an interrupted time series analysis of care gap closure rates was used to compare the pre-intervention and intervention periods. Majority of residents (28-32 or 80-91%) completed the surveys and focus groups. Residents found the EHR tools acceptable and valued protected time, mentorship, and autonomy to set goals. A total of 13,313 patient visits were analyzed. There were no significant differences between rates between the pre-intervention period and the first intervention period (p=0.44).

DISCUSSION

A longitudinal PM curriculum that incorporated SDL and goal setting with EHR-driven performance feedback was well-received by residents, however did not significantly impact the rate of care gap closure.

Consensus marking as a grading method for the development of evaluative judgement: Comparing assessor and students

AIM

This study explored postgraduate nursing students' perceptions, anxiety and satisfaction of an innovative and novel grading method for online vivas, consensus marking, compared with traditional assessor judgement.

BACKGROUND

Reflection, self-evaluation and feedback conversations have the potential to develop nursing students' evaluative judgement. Consensus marking is a novel method of grading students' performance that supports students to reflect, self-evaluate and grade their own work. Active engagement in a feedback dialogue supports students to calibrate their self-evaluation to the required standard in a grade negotiation. Through this approach, students are supported to develop evaluative judgement and lifelong learning skills.

DESIGN

A convergent mixed-methods parallel research design was used.

METHODS

Students enrolled in a postgraduate emergency nursing unit of study completed two online viva assessments. One viva was graded using traditional assessor judgement and the other used consensus marking, involving a two-way feedback dialogue, where students had an opportunity to actively engage in grading their own work with the assessor. Student perceptions of each grading method were explored through semi-structured interviews. Interview data were analysed thematically using a six-stage approach. Student anxiety and satisfaction were measured pre- and post each viva using valid and reliable questionnaires. Non-parametric analyses explored differences in anxiety and satisfaction between the two grading methods. Alpha was set at 0.05.

RESULTS

Forty-six participants had complete data for anxiety and satisfaction across both test occasions (82%) and were included in the analysis. Of these, 13 students participated in follow up interviews. Students perceived that the ability to self-evaluate performance and discuss their grade with the assessor using consensus marking was less hierarchical and similar to a collegial debrief. Student anxiety was significantly lower prior to consensus marking compared with the assessor judged viva (p < 0.001). Students were significantly more satisfied with consensus marking compared with assessor judgement (p < 0.01).

CONCLUSIONS

Consensus marking created an opportunity for students to identify knowledge deficits through reflection and self-evaluation of their own performance prior to external judgement. Students were more satisfied and less anxious with the consensus marking grading method compared with traditional assessor judgement. These findings have implications for the development and application of new grading methods in nursing education to facilitate the development of evaluative judgement.

Patient involvement in assessment of postgraduate medical learners: A scoping review

CONTEXT

Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent.

METHODS

We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment.

RESULTS

We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise.

CONCLUSION

Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

The Senior Medical Resident's New Role in Assessment in Internal Medicine

PURPOSE

With the introduction of competency-based medical education, senior residents have taken on a new, formalized role of completing assessments of their junior colleagues. However, no prior studies have explored the role of near-peer assessment within the context of entrustable professional activities (EPAs) and competency-based medical education. This study explored internal medicine residents' perceptions of near-peer feedback and assessment in the context of EPAs.

METHOD

Semistructured interviews were conducted from September 2019 to March 2020 with 16 internal medicine residents (8 first-year residents and 8 second- and third-year residents) at the University of Toronto, Toronto, Ontario, Canada. Interviews were conducted and coded iteratively within a constructivist grounded theory approach until sufficiency was reached.

RESULTS

Senior residents noted a tension in their dual roles of coach and assessor when completing EPAs. Senior residents managed the relationship with junior residents to not upset the learner and potentially harm the team dynamic, leading to the documentation of often inflated EPA ratings. Junior residents found senior residents to be credible providers of feedback; however, they were reticent to find senior residents credible as assessors.

CONCLUSIONS

Although EPAs have formalized moments of feedback, senior residents struggled to include constructive feedback comments, all while knowing the assessment decisions may inform the overall summative decision of their peers. As a result, EPA ratings were often inflated. The utility of having senior residents serve as assessors needs to be reexamined because there is concern that this new role has taken away the benefits of having a senior resident act solely as a coach.

Affecting Change: Enhancing Feedback Interactions with Radiology Trainees

Feedback is a critical part of the learning process and is a valuable tool to empower adult learners. Modern feedback theory places the learner at the center of the feedback encounter. Individual and institutional barriers to effective giving and receiving of feedback can be overcome through education and attention to the form and content of feedback. We review the elements of effective feedback and address issues of framing, environmental, and social factors which aid in providing psychological safety and trust, as necessary elements to create a culture of feedback in radiology training programs. We provide practical strategies to empower learners with the necessary skills to solicit, receive, and reflect on feedback.

How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study

PURPOSE

In competency-based medical education, workplace-based assessment provides trainees with an opportunity for guidance and supervisors the opportunity to judge the trainees' clinical practice. Learning from assessment is enhanced when trainees reveal their thinking and are open to critique, which requires trust in the assessor. If supervisors knew more about how trainees come to trust them in workplace-based assessment, they could better engender trainee trust and improve trainees' learning experience.

METHOD

From August 2018 to September 2019, semistructured interviews were conducted with 17 postgraduate anesthesia trainees across Australia and New Zealand. The transcripts were analyzed using constructivist grounded theory methods sensitized by a sociocultural view of learning informed by Wenger's communities of practice theory.

RESULTS

Participants described a continuum from a necessary initial trust to an experience-informed dynamic trust. Trainees assumed initial trust in supervisors based on accreditation, reputation, and a perceived obligation of trustworthiness inherent in the supervisor's role. With experience and time, trainees' trust evolved based on supervisor actions. Deeper levels of trainee trust arose in response to perceived supervisor investment and allowed trainees to devote more emotional and cognitive resources to patient care and learning rather than impression management. Across the continuum from initial trust to experience-informed trust, trainees made rapid trust judgments that were not preceded by conscious deliberation; instead, they represented a learned "feel for the game."

CONCLUSIONS

While other factors are involved, our results indicate that the trainee behavior observed in workplace-based assessment is a product of supervisor invitation. Supervisor trustworthiness and investment in trainee development invite trainees to work and present in authentic ways in workplace-based assessment. This authentic engagement, where learners "show themselves" to supervisors and take risks, creates assessment for learning.

Test-enhanced learning improves learner attendance during a laparoscopic box trainer simulation program

Background

Laparoscopy is the gold standard approach for many surgical procedures, but it is a complex skill to learn. Laparoscopic simulation training may help, but it is unclear how to best engage trainees in these programs. Test‐enhanced learning (TEL) uses regular, well‐defined assessments of performance throughout the training phase of learning.

Aim

The aim of this study was to assess the effects of TEL on a laparoscopic simulation program involving a cohort of medical student volunteers.

Materials and methods

A prospective cohort study was performed with a convenience sample of 40 medical students. Students were recruited to participate in a ten‐week laparoscopic simulation program. Twenty students participated in a laparoscopic surgical program with TEL (‘TEL group’), and 20 students participated in a standard laparoscopic simulation program (‘control group’).

Results

Attendance in the TEL group was significantly higher than in the standard group (71 vs 51.5%, P = 0.03). There was no difference between groups in mean time scores. Four themes were identified in qualitative data drawn from student surveys – personal traits and motivators, training context, clear goals and feedback enabling understanding of one's own performance.

Conclusion

Testing laparoscopic skills throughout a learning program, in conjunction with individualised feedback and tracking of learning trajectory, increases trainee attendance. Laparoscopic simulation training programs are encouraged to reflect on the pedagogic framework in which their procedural skills training operates.

Profiles for Success: Examining the Relationship between Student Profiles and Clerkship Performance Using Latent Profile Analysis

PHENOMENON

Learning is a complex phenomenon in which learners can vary in their learning orientation, learning approaches, and perception of the learning environment. Learners may be characterized as mastery oriented or performance oriented, and this learning orientation can influence their learning approaches, such as whether to seek feedback or ask for help. The learning environment includes institutional assessment policies and informal interactions that emphasize outcomes and differentiation among students. When learners perceive the learning environment as performance oriented, they feel competition from other learners and fear negative feedback. This complex interplay of learning orientation, its influence on feedback-seeking behavior, and the environmental influence and reinforcement of both can be captured through learner profiles.

APPROACH

In this multi-institutional cross-sectional survey study of students from six U.S. medical schools, we investigated learner profiles that characterize how these domains manifest together across individual learners. We then determined how these profiles are associated with students' clerkship grades. Measures included the Patterns of Adaptive Learning for learning orientation, self-reports for learning approaches, and perceptions of the learning environment. We used latent profile analysis to cluster students who share common characteristics around orientation, behavior, and environment. The relationship between these profiles and the percentage of honors earned was examined using multiple regression analysis.

FINDINGS

We found four distinct learner profiles within 666 of 974 (68%) analyzable responses: (1) performance oriented with negative perceptions of environment, (2) mastery oriented with desirable learning approaches, (3) "average" group, and (4) mastery orientation only. Profile 1 (M = 39%) and Profile 4 (M = 38%) students received fewer clerkship honors compared with other profiles. Profile 2 students earned a significantly higher percentage of honors grades (M = 54%) compared with other students. Profile 3 students (average group) earned the second highest percentage of honors (M = 46%). Effect sizes comparing the percentage of honors varied from 0.32 to 0.65, indicating medium to large differences.

INSIGHTS

The highest performers, as measured by clerkship honors grades, were mastery oriented, perceived a more positive learning environment, and were comfortable asking questions and seeking feedback for their learning. The lowest performing students were performance oriented and had negative perceptions of their learning environments. Contrary to previous findings, we found that the relationship between mastery orientation and learning approaches is not unidirectional but rather seems to be mediated by perceptions of the learning environment.

New approach to learning medical procedures using a smartphone and the Moodle platform to facilitate assessments and written feedback

Background: To overcome communication obstacles between medical students and trainers, we designed serial learning activities utilizing a smartphone and web-based instruction (WBI) on the Moodle platform to provide clear and retrievable trainer feedback to students on an objective structured clinical examination (OSCE) item. Methods: We evaluated students’ learning achievement and satisfaction with the new learning tool. A total of 80 fourth-year medical students participated. They installed the Moodle app (the WBI platform) on their smartphones and practiced an endotracheal suction procedure on a medical simulation mannequin while being evaluated by a trainer regarding competence in clinical skills on the smartphone app. Students’ competency was evaluated by comparing the scores between the formative assessment and the summative assessment. The degree of satisfaction and usefulness for the smartphone and WBI system were analyzed. Results: The means (standard deviations, SDs) of the formative and summative assessments were 8.80 (2.53) and 14.24 (1.97) out of a total of 17 points, respectively, reflecting a statistically significant difference (P<0.05). The degree of satisfaction and perceived usefulness of the smartphone app and WBI system were excellent, with means (SDs) of 4.60 (0.58), and 4.60 (0.65), respectively. Conclusion: We believe that the learning process using a smartphone and the Moodle platform offers good guidance for OSCE skill development because trainers’ written feedback is recorded online and is retrievable at all times, enabling students to build and maintain competency through frequent feedback review.

Residents' transformational changes through self-regulated, experiential learning for professionalism

PURPOSE

Postgraduate trainees ('residents") are required to convey professional behaviours as they navigate complex clinical environments. However, little is known about experiential learning for professionalism. Thus, we asked residents about professionalism challenges within the clinical learning environment: 1) how challenges were identified, 2) what supported successfully addressing challenges and 3) the impact of addressing challenges to further inform resident education.

METHOD

From 2015-2016, twenty-five residents across specialties and multiple university affiliated teaching hospitals participated in appreciative inquiry informed audio-taped semi-structured interviews. Transcripts were categorized deductively for the 2015 CanMEDS Professional Role element addressed (commitment to patients, society, the profession, and physician health). A pragmatic research paradigm focussed descriptive data analysis on actions and outcomes.

RESULTS

Residents actively identify opportunities for experiential learning of professionalism within the clinical workplace- addressing conflicting priorities with interprofessional clinicians to ensure excellent patient care, providing informal feedback regarding peers' and other healthcare clinicians' professionalism lapses and by gaining self-awareness and maintaining wellness. There were no descriptions of commitment to society. Values, relationships, and reflection supported professional behaviours. Many described transformative personal and professional growth as an outcome of addressing professionalism challenges.

CONCLUSIONS

Residents self-regulated experiential learning for professionalism often results in transformational changes personally and professionally. Elucidation of how residents successfully navigate power dynamics and conflict to provide excellent patient care and feedback for professional regulatory behaviour will support professionalism education. An interprofessional research lens will be valuable to explore how best to incorporate commitment to society within clinical environments.