Workplace-based assessment as an educational tool: AMEE Guide No. 31

Exploring effective video-review strategies of patient encounters for medical students: precepted review versus peer discussion

BACKGROUND

Video-recordings review of patient encounters is reported to improve the clinical performance of medical students. However, evidence on specific remediation strategies or outcomes are lacking. We aimed to implement videorecording-based remediation of standardized patient encounters among medical students, combined with preceptor one-on-one feedback or peer group discussion, and evaluate the effectiveness of the two remediation methods using objective structured clinical examination (OSCE).

METHODS

Following standardized patient encounters, 107 final-year medical students were divided into two groups based on different remediation methods of video review: (1) precepted video review with preceptor feedback (N = 55) and (2) private video review and subsequent peer group discussion under supervision (N = 52). All students underwent twelve-stations of OSCE both before and after the video review. Students' pre- and post-remediation OSCE scores, self-efficacy level in patient encounters, and level of educational satisfaction with each method were assessed and compared between different video-based remediation methods to evaluate their respective effects.

RESULTS

After remediation, the total and subcomponent OSCE scores, such as history taking, physical examination, and patient - physician interaction (PPI), among all students increased significantly. Post-remediation OSCE scores showed no significant difference between two remediation methods (preceptor module, 79.6 ± 4.3 vs. peer module, 79.4 ± 3.8 in the total OSCE score). Students' self-efficacy levels increased after remediation in both modules (both p-value <0.001), with no difference between the two modules. However, students' satisfaction level was higher in the preceptor module than in the peer module (80.1 ± 17.7 vs. 59.2 ± 25.1, p-value <0.001). Among students with poor baseline OSCE performance, a prominent increase in PPI scores was observed in the preceptor-based module.

CONCLUSION

Video-based remediation of patient encounters, either through preceptor review with one-on-one feedback or through private review with peer discussion, was equally effective in improving the OSCE scores and self-efficacy levels of medical students. Underperforming students can benefit from precepted video reviews for building PPI.

A Systematic Review: What Are the Impacts of Receiving Extrinsic Feedback on Health Professions Students in Higher Education?

Effective feedback is crucial for educating health professions students. This systematic review investigates the impact of extrinsic feedback on health professions students, encompassing medical, dental, and nursing fields. Through meticulous gathering and analysis of 37 studies, this review highlights verbal and visual feedback as predominant forms, often delivered immediately by instructors and supplemented by peer or simulated input. Notably, technology is increasingly utilised to enhance the provision of feedback. The impacts of feedback span various domains, including surgical skills and patient communication, revealing notable improvements in procedural skills such as suturing and knot-tying, as well as general patient communication proficiency. Meta-analyses underscore significant enhancements in communication skills and provide nuanced insights into chest compression techniques. Overall, the findings provide initial evidence that extrinsic feedback enhances surgical procedural skills and general patient communication proficiency among health professions students. The evolving role of technology in feedback provision is promising. Future studies should assess extrinsic feedback across different health professions to better understand its impacts and alignment with specific educational needs and accreditation standards, thereby enhancing learning outcomes.

Optimizing a national examination for medical undergraduates via modern automated test assembly approaches

BACKGROUND

Automated test assembly (ATA) represents a modern methodology that employs data science optimization on computer platforms to automatically create test form, thereby significantly improving the efficiency and accuracy of test assembly procedures. In the realm of medical education, large-scale high-stakes assessments often necessitate lengthy tests, leading to elevated costs in various dimensions (such as examinee fatigue and expenses associated with item development). This study aims to augment the design of the medical education assessments by leveraging modern ATA approaches.

METHODS

To achieve the objective, a four-step process employing psychometric methodologies was used to calibrate and analyze the item pool of the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test comprising 300 multiple-choice questions (MCQ) in China. Subsequently, two modern ATA approaches were employed to determine the optimal item combination, accounting for both statistical and content requirements specified in the test blueprint. The qualities of the assembled test form, generated using modern ATA approaches, underwent meticulous evaluation.

RESULTS

Through an exploration of the psychometric properties of the SCTCMU as a foundational step, the evaluation revealed commendable quality in the item properties. Furthermore, the evaluation of the quality of assembled test form using modern ATA approaches indicated the ability to ascertain the optimal test length within the predefined measurement precision. Specifically, this investigation demonstrates that the application of modern ATA approaches can substantially reduce the test length of assembled test form, while simultaneously maintaining the required statistical and content standards specified in the test blueprint.

CONCLUSIONS

This study harnessed modern ATA approaches to facilitate the automatic construction of test form, thereby significantly enhancing the efficiency and precision of test assembly procedures. The utilization of modern ATA approaches offers medical educators a valuable tool to enhance the efficiency and cost-effectiveness of medical education assessment.

Longitudinal variation of correlations between different components of assessment within a medical school

BACKGROUND

An assessment program should be inclusive and ensure that the various components of medical knowledge, clinical skills, and professionalism are assessed. The level and the variation over time in the strength of the correlation between these components of assessment is still a matter of study. Based on the meaningful learning theory and the integrated learning theory, we hypothesize that these components increase their connections during the medical school course.

METHODS

This is a retrospective cohort study that analyzed data collected for a 10-year period in one medical school. We included students from the 3rd to 6th year of medical school from 2011 to 2021. Three assessment components were addressed: Knowledge, Clinical Skills, and Professionalism. For data analysis, Pearson correlation coefficients (R) and R2 were calculated to study the correlation between variables and a z-test on Fisher's r-to-z was used to determine the differences between correlation coefficients.

RESULTS

949 medical students were included in the study. The correlation between Clinical Skills and Professionalism showed a medium to strong association (Pearson's R ranging from 0.485 to 0.734), while the correlation between Knowledge and Professionalism was weaker but exhibited a steady evolution with Pearson's R fluctuating between 0.075 and 0.218. The Knowledge and Clinical Skills correlation became statistically significant from 2013 onwards and peaking at Pearson's R of 0.440 for the cohort spanning 2016-2019. We also revealed a strengthening of correlations between Professionalism and Clinical Skills from the beginning to the end of clinical training, but not with the knowledge component.

CONCLUSIONS

This analysis contributes to our understanding of the dynamics of correlations of different assessment components within an institution and provides a framework for how they interact and influence each other.

TRIAL REGISTRATION

This study was not a clinical trial, but a retrospective observational study, without health care interventions. Nevertheless, we provide herein the number of the study as submitted to the Ethics committee - CEICVS 146/2021.

Structure, governance and delivery of specialist training programs in periodontology and implant dentistry

AIM

To update the competences and learning outcomes and their evaluation, educational methods and education quality assurance for the training of contemporary specialists in periodontology, including the impact of the 2018 Classification of Periodontal and Peri-implant Diseases and Conditions (2018 Classification hereafter) and the European Federation of Periodontology (EFP) Clinical Practice Guidelines (CPGs).

METHODS

Evidence was gathered through scientific databases and by searching for European policies on higher education. In addition, two surveys were designed and sent to program directors and graduates.

RESULTS

Program directors reported that curricula were periodically adapted to incorporate advances in diagnosis, classification, treatment guidelines and clinical techniques, including the 2018 Classification and the EFP CPGs. Graduates evaluated their overall training positively, although satisfaction was limited for training in mucogingival and surgical procedures related to dental implants. Traditional educational methods, such as didactic lectures, are still commonly employed, but they are now often associated with more interactive methods such as case-based seminars and problem-based and simulation-based learning. The evaluation of competences/learning outcomes should employ multiple methods of assessment.

CONCLUSION

An update of competences and learning outcomes of specialist training in periodontology is proposed, including knowledge and practical application of the 2018 Classification and CPGs. Harmonizing specialist training in periodontology is a critical issue at the European level.

Co-Existing Vascular Surgery Integrated Residencies are Associated with Increased General Surgery Resident Proficiency and Autonomy in Vascular Cases

BACKGROUND

Integrated vascular surgery residency positions have doubled more than the last decade. Studies have investigated the impact of co-existing subspecialty surgical training programs on case volume of general surgery residents (GSRs). However, no studies have explored the impact of subspecialty training on GSR operative competency. The aim of this study is to understand the impact of integrated residencies on operative performance and autonomy of GSR performing vascular procedures.

METHODS

Autonomy and performance ratings of GSR participating in vascular surgery cases were collected from all institutions participating in the Society for Improving Medical Professional Learning application database from 2015 to 2023. Faculty and self-assessments of autonomy and performance on vascular cases performed by GSRs at programs with co-existing vascular integrated residency (VIR), vascular surgery fellowship (VSF), or no subspecialty vascular training (VX) were compared using Fisher's exact tests with Bonferroni corrections across training levels and case complexity.

RESULTS

Eleven thousand one hundred seventy five assessments (26% at institutions with VIR, 46% VSF, and 28% VX) were submitted by 920 GSRs and 343 faculty. Senior GSRs at programs with VSF achieved lower autonomy than those with VIR (P = 0.049) or VX (P = 0.042) based on faculty assessment. GSRs achieved a level of "practice ready" at significantly higher rates when training at programs with VIR, and at the lowest rates with VSF (P < 0.001). However, self-perception of autonomy and performance was highest among GSRs at programs with VX compared with VIR and VSF (P < 0.001).

CONCLUSIONS

The presence of VIR was associated with higher achievement of "practice ready" competency and higher levels of operative autonomy among senior GSRs performing vascular procedures. Shared-learning among peers and faculty expertise in teaching resident-level trainees may contribute to this finding.

Managing assessment during curriculum change: Ottawa Consensus Statement

Curriculum change is relatively frequent in health professional education. Formal, planned curriculum review must be conducted periodically to incorporate new knowledge and skills, changing teaching and learning methods or changing roles and expectations of graduates. Unplanned curriculum evolution arguably happens continually, usually taking the form of "minor" changes that in combination over time may produce a substantially different programme. However, reviewing assessment practices is less likely to be a major consideration during curriculum change, overlooking the potential for unintended consequences for learning. This includes potentially undermining or negating the impact of even well-designed and important curriculum changes. Changes to any component of the curriculum "ecosystem "- graduate outcomes, content, delivery or assessment of learning - should trigger an automatic review of the whole ecosystem to maintain constructive alignment. Consideration of potential impact on assessment is essential to support curriculum change. Powerful contextual drivers of a curriculum include national examinations and programme accreditation, so each assessment programme sits within its own external context. Internal drivers are also important, such as adoption of new learning technologies and learning preferences of students and faculty. Achieving optimal and sustainable outcomes from a curriculum review requires strong governance and support, stakeholder engagement, curriculum and assessment expertise and internal quality assurance processes. This consensus paper provides guidance on managing assessment during curriculum change, building on evidence and the contributions of previous consensus papers.

Linking assessment to real life practice - comparing work based assessments and objective structured clinical examinations using mystery shopping

Objective Structured Clinical Examinations (OSCEs) and Work Based Assessments (WBAs) are the mainstays of assessing clinical competency in health professions' education. Underpinned by the extrapolation inference in Kane's Validity Framework, the purpose of this study is to determine whether OSCEs translate to real life performance by comparing students' OSCE performance to their performance in real-life (as a WBA) using the same clinical scenario, and to understand factors that affect students' performance. A sequential explanatory mixed methods approach where a grade comparison between students' performance in their OSCE and WBA was performed. Students were third year pharmacy undergraduates on placement at a community pharmacy in 2022. The WBA was conducted by a simulated patient, unbeknownst to students and indistinguishable from a genuine patient, visiting the pharmacy asking for health advice. The simulated patient was referred to as a 'mystery shopper' and the process to 'mystery shopping' in this manuscript. Community pharmacy is an ideal setting for real-time observation and mystery shopping as staff can be accessed without appointment. The students' provision of care and clinical knowledge was assessed by the mystery shopper using the same clinical checklist the student was assessed from in the OSCE. Students who had the WBA conducted were then invited to participate in semi-structured interviews to discuss their experiences in both settings. Overall, 92 mystery shopper (WBA) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 41.7% [IQR 28.3] and significantly lower compared to the OSCE score 80.9% [IQR 19.0] in all participants (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage real-life patients. Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as opposed to an unfamiliar workplace. As suggested by the activity theory, the performance of the student can be driven by their motivation which differed in the two contexts.

Conceptualizing and Developing Competence in Newborn Medicine Among Military Pediatricians

INTRODUCTION

Competence in neonatal care is especially important for military pediatricians because military pediatricians can be asked to serve in remote duty locations with limited resources. We sought to understand how this competence is defined, developed, and assessed by military pediatric training programs.

MATERIALS AND METHODS

After Institutional Review Board approval was obtained, we interviewed educators and recent graduates from every pediatric military training program to construct a shared definition of competence. We then used Kern's Six Steps for curriculum development to understand how competence is taught and assessed.

RESULTS

Participants felt that competence for military pediatricians in the neonatal setting meant that learners should be able to provide a full spectrum of newborn care in any military setting. Participants confirmed that this competence was particularly important for military pediatricians because of the possibility of remote duty locations. Participants felt that specific knowledge, skills, and attitudes supported competence. Knowledge domains include distinguishing normal newborns from abnormal newborns, managing normal newborn care, managing common newborn abnormalities, and creating a safe escalation plan for complicated or uncommon newborn abnormalities. Specific skills that support competence are newborn resuscitation, delivery of effective ventilation, and neonatal circumcision. Specific attitudes that support competence are, understanding the personal limits of knowledge and understanding the resources for escalation of care. Educators use a variety of modalities to teach toward competence, including the structured curricula, bedside teaching, and simulation. According to participants, the assessment of learners occurs primarily through narrative assessment and feedback but would ideally occur through direct observation.

CONCLUSIONS

Competence in the neonatal setting is particularly important for military pediatricians. Essential skills undergo differential assessment and current assessment methods differ from ideal assessment methods. Future work should focus on how these facets can support a unified curriculum in newborn medicine.

A needs assessment for enhancing workplace-based assessment: a grounded theory study

OBJECTIVES

Workplace-based assessment (WBA) has been vigorously criticized for not fulfilling its educational purpose by medical educators. A comprehensive exploration of stakeholders' needs regarding WBA is essential to optimize its implementation in clinical practice.

METHOD

Three homogeneous focus groups were conducted with three groups of stakeholders: General Practitioner (GP) trainees, GP trainers, and GP tutors. Due to COVID-19 measures, we opted for an online asynchronous form to enable participation. An constructivist grounded theory approach was used to employ this study and allow the identification of stakeholders' needs for using WBA.

RESULTS

Three core needs for WBA were identified in the analysis. Within GP Training, stakeholders found WBA essential, primarily, for establishing learning goals, secondarily, for assessment purposes, and, lastly, for providing or receiving feedback.

CONCLUSION

All stakeholders perceive WBA as valuable when it fosters learning. The identified needs were notably influenced by agency, trust, availability, and mutual understanding. These were facilitating factors influencing needs for WBA. Embracing these insights can significantly illuminate the landscape of workplace learning culture for clinical educators and guide a successful implementation of WBA.

Extravert Surgical Resident Applicants Get Higher 360-Degree Evaluations From Coworkers

INTRODUCTION

360-degree evaluations are used as an assessment in order to identify strengths and weaknesses of, or as a continuous evaluation for, residents. The aim of this study was to investigate the relationship between personality and ratings on 360-degree evaluations among surgical residency applicants. A secondary aim was to describe the personality profile of applicants for a surgical residency position.

METHODS

Doctors interviewed for a residency or locum position in general, urology, or pediatric surgery were included. Participants rated their personality on the Neutralized Big Five Inventory. A 360-degree assessment was conducted. Scores from two laparoscopic simulators were used as a measure of technical ability. Univariate analyses were used to assess the results. Student's t-test was used to compare personality and Pearson correlations between 360-degree assessment and personality.

RESULTS

Fifty doctors participated: data were complete for 38. Personality profiles showed higher emotional stability, agreeableness, conscientiousness and openness ratings than the norm. Correlations revealed a significant relationship between extraversion and higher scores on the 360-degree assessments. Significant univariate correlations were found between extraversion and the 360-degree assessments, and two of the correlations held up after adjustment for multiple tests. No correlations with performance when using laparoscopic simulators were found.

CONCLUSIONS

Applicants for surgical residency rated significantly higher than the norm in four personality domains. Extraversion correlated with overall higher scores in 360-degree assessments by coworkers. Higher scores were not related to objective measures of technical skill, highlighting the importance of using objective measures for assessment.

Use of Mini-CEX as Formative Assessment Tool in the Training of Undergraduate Medical Students in ENT Situation Analysis and the Way Forward

Background

Mini-CEX assesses clinical competency and is mainly used as a formative assessment tool. Its use in postgraduate training is well documented. However, Mini-CEX would play a significant role in training undergraduate medical students, especially with the commencement of competency-based medical education in India. This work reports the situational analysis of the Mini-CEX implementation in the department of ENT.

Methods and Material

The Department of ENT is using Mini-CEX for formative assessment of students' clinical competence since 2017. Each student had to complete a minimum of five Mini-CEX encounters before the summative assessment. We reviewed the Mini-CEX assessment records of 149 undergraduate medical students who appeared for the summative exam in 2018.

Results

We analysed the records of 874 Mini-CEX encounters. Each Mini-CEX encounter took 11 min on average. Each student completed five such assessments, which accounted for 55 min of one-to-one teacher-student interaction focused on clinical skills learning. The feedback time varied from 1 to 30 min. Feedback was focused on the cognitive (46%) and psychomotor (42%) domains. However, the majority of students reflected that they learned psychomotor skills during the Mini-CEX. Students selected only a few skills for the Mini-CEX, ignoring many must-know skills.

Conclusions

Mini-CEX is feasible as a formative assessment tool for medical undergraduates' ENT training. It improves the assessor-student interaction, provides effective feedback, and develops the practice of reflection among students. However, regular review and training of the assessors and students are needed as a quality assurance measure.

Barriers and future improvements of workplace-based learning in Korean medicine clinical clerkship: perspectives of graduates

BACKGROUND

Workplace-based learning (WPBL) has emerged as an essential practice in healthcare education. However, WPBL is rarely implemented in Korean medicine (KM) due to the passive attitude of teachers and possible violation of medical laws that limit the participation of trainees in medical treatment. In this study, we implemented WPBL in the clinical clerkship of Acupuncture and Moxibustion Medicine at a single College of KM and explored the barriers and future improvements of WPBL.

METHODS

The WPBL was implemented from January to July 2019. During the clerkship, each senior student was assigned an inpatient at the university hospital. WPBL was conducted as follows: patient presentation by the supervisor, interaction with the patient at the bedside, preparation of medical records, oral case presentation, and discussion with feedback. The student performed a physical examination and review of systems as a clinical task. In addition, six doctors of KM who are currently practicing after three years of WPBL were interviewed in September 2022 to investigate the real-world effects and unmet needs of WPBL in their workplaces.

RESULTS

Two major themes identified from the interview were: "the experience of novice doctors of KM with KM practice" and "Current state of KM clinical education." The five subcategories were: "Clinical competency priorities vary according to the KM workplace," "Difficulties faced by doctors of KM immediately after graduation," "WPBL experience of the interviewees," "Necessary but difficult to implement real patient learning," and "Unmet needs for clinical clerkship in KM."

CONCLUSION

It is essential to consider the unique characteristics of KM practice and the duties required in various workplaces for successful WPBL. We anticipate our study to be a starting point for improving the WPBL and addressing the unmet needs in KM clinical education.

Co-designing Entrustable Professional Activities in General Practitioner's training: a participatory research study

BACKGROUND

In medical education, Entrustable Professional Activities (EPAs) have been gaining momentum for the last decade. Such novel educational interventions necessitate accommodating competing needs, those of curriculum designers, and those of users in practice, in order to be successfully implemented.

METHODS

We employed a participatory research design, engaging diverse stakeholders in designing an EPA framework. This iterative approach allowed for continuous refinement, shaping a comprehensive blueprint comprising 60 EPAs. Our approach involved two iterative cycles. In the first cycle, we utilized a modified-Delphi methodology with clinical competence committee (CCC) members, asking them whether each EPA should be included. In the second cycle, we used semi-structured interviews with General Practitioner (GP) trainers and trainees to explore their perceptions about the framework and refine it accordingly.

RESULTS

During the first cycle, 14 CCC members agreed that all the 60 EPAs should be included in the framework. Regarding the formulation of each EPAs, 20 comments were given and 16 adaptations were made to enhance clarity. In the second cycle, the semi-structured interviews with trainers and trainees echoed the same findings, emphasizing the need of the EPA framework for improving workplace-based assessment, and its relevance to real-world clinical scenarios. However, trainees and trainers expressed concerns regarding implementation challenges, such as the large number of EPAs to be assessed, and perception of EPAs as potentially high-stakes.

CONCLUSION

Accommodating competing stakeholders' needs during the design process can significantly enhance the EPA implementation. Recognizing users as experts in their own experiences empowers them, enabling a priori identification of implementation barriers and potential pitfalls. By embracing a collaborative approach, wherein diverse stakeholders contribute their unique viewpoints, we can only create effective educational interventions to complex assessment challenges.

The impact of prior performance information on subsequent assessment: is there evidence of retaliation in an anonymous multisource assessment system?

Few studies have engaged in data-driven investigations of the presence, or frequency, of what could be considered retaliatory assessor behaviour in Multi-source Feedback (MSF) systems. In this study, authors explored how assessors scored others if, before assessing others, they received their own assessment score. The authors examined assessments from an established MSF system in which all clinical team members - medical students, interns, residents, fellows, and supervisors - anonymously assessed each other. The authors identified assessments in which an assessor (i.e., any team member providing a score to another) gave an aberrant score to another individual. An aberrant score was defined as one that was more than two standard deviations from the assessment receiver's average score. Assessors who gave aberrant scores were categorized according to whether their behaviour was preceded by: (1) receiving a score or not from another individual in the MSF system (2) whether the score they received was aberrant or not. The authors used a multivariable logistic regression model to investigate the association between the type of score received and the type of score given by that same individual. In total, 367 unique assessors provided 6091 scores on the performance of 484 unique individuals. Aberrant scores were identified in 250 forms (4.1%). The chances of giving an aberrant score were 2.3 times higher for those who had received a score, compared to those who had not (odds ratio 2.30, 95% CI:1.54-3.44, P < 0.001). Individuals who had received an aberrant score were also 2.17 times more likely to give an aberrant score to others compared to those who had received a non-aberrant score (2.17, 95% CI:1.39-3.39, P < 0.005) after adjusting for all other variables. This study documents an association between receiving scores within an anonymous multi-source feedback (MSF) system and providing aberrant scores to team members. These findings suggest care must be given to designing MSF systems to protect against potential downstream consequences of providing and receiving anonymous feedback.

Psychometric Analysis of an Integrated Clinical Education Tool for Physical Therapists

INTRODUCTION

Integrated clinical education (ICE) courses require opportunities for practice, assessment of performance, and specific feedback. The purposes of this study were to 1) analyze the internal consistency of a tool for evaluating students during ICE courses, 2) examine the responsiveness of the tool between midterm and final assessments, and 3) develop a model to predict the final score from midterm assessments and explore relationships among the 6 domains.

REVIEW OF LITERATURE

Several clinical education assessment tools have been developed for terminal clinical experiences, but few have focused on the needs of learners during the ICE.

SUBJECTS

Eighty-five student assessments were collected from 2 consecutive cohorts of physical therapist students in a first full-time ICE course.

METHODS

The tool contained 29 items within 6 domains. Items were rated on a 5-point scale from dependent to indirect supervision. Cronbach's alpha was used to analyze the internal consistency of the tool, whereas responsiveness was examined with paired t-test and Cohen's d. A best subsets regression model was used to determine the best combination of midterm variables that predicted the final total scores. Coefficients of determination (R2) were calculated to explore the relationships among domains.

RESULTS

The tool was found to have high internal consistency at midterm and final assessment (α = 0.97 and 0.98, respectively). Mean scores increased over time for each domain score and for the total score (P < .001; d = 1.5). Scores in 3 midterm domains predicted more than 57% of the variance in the final total score.

DISCUSSION AND CONCLUSION

Results support the use of this tool to measure student performance and growth in a first full-time ICE course. Targeted measurement of students' abilities in ICE courses assists with differentiating formative and summative learning needed to achieve academic success.

Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review

BACKGROUND

Feedback processes are crucial for learning, guiding improvement, and enhancing performance. In workplace-based learning settings, diverse teaching and assessment activities are advocated to be designed and implemented, generating feedback that students use, with proper guidance, to close the gap between current and desired performance levels. Since productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured feedback activities within undergraduate workplace-based learning settings. However, these settings are characterized by their unpredictable nature, which can either promote learning or present challenges in offering structured learning opportunities for students. This scoping review maps literature on how feedback processes are organised in undergraduate clinical workplace-based learning settings, providing insight into the design and use of feedback.

METHODS

A scoping review was conducted. Studies were identified from seven databases and ten relevant journals in medical education. The screening process was performed independently in duplicate with the support of the StArt program. Data were organized in a data chart and analyzed using thematic analysis. The feedback loop with a sociocultural perspective was used as a theoretical framework.

RESULTS

The search yielded 4,877 papers, and 61 were included in the review. Two themes were identified in the qualitative analysis: (1) The organization of the feedback processes in workplace-based learning settings, and (2) Sociocultural factors influencing the organization of feedback processes. The literature describes multiple teaching and assessment activities that generate feedback information. Most papers described experiences and perceptions of diverse teaching and assessment feedback activities. Few studies described how feedback processes improve performance. Sociocultural factors such as establishing a feedback culture, enabling stable and trustworthy relationships, and enhancing student feedback agency are crucial for productive feedback processes.

CONCLUSIONS

This review identified concrete ideas regarding how feedback could be organized within the clinical workplace to promote feedback processes. The feedback encounter should be organized to allow follow-up of the feedback, i.e., working on required learning and performance goals at the next occasion. The educational programs should design feedback processes by appropriately planning subsequent tasks and activities. More insight is needed in designing a full-loop feedback process, in which specific attention is needed in effective feedforward practices.

Basic Training in Palliative Medicine for Internal Medicine Residents: Pilot Testing of a Canadian Model in Switzerland

Background

In Switzerland, palliative care (PC) clinical training is well established at undergraduate and specialist postgraduate levels. However, postgraduate nonspecialist training curricula are less documented.

Local Problem

A structured curriculum for nonspecialist rotation within internal medicine (IM) in specialized PC wards is lacking.

Objective

To pilot two versions of a PC nonspecialist curriculum for IM residents in Swiss PC units.

Methods

In the pilot phase, two curricula-short immersion (3-10 weeks, based on the University of Toronto's Internal-Medicine PC Rotation) and standard nonspecialist (11-18 weeks, based on the Canadian Society of Palliative Care Physician Competencies)-were assessed using a mixed-method online survey. One university and two nonuniversity sites participated. The analysis was descriptive.

Results

Five residents and eight supervisors of five training rotations (July-October 2023) responded. Overall, curriculum quality and feasibility (content and time) received positive ratings across all groups, with high satisfaction concerning organization, educational design, learning support, climate, experience, and facilities. Nonuniversity sites were generally rated more positively than university sites. Qualitative feedback paralleled these findings, highlighting the curriculum's relevance and fit with learners' needs and suggesting potential simplifications and more personalized planning.

Conclusions

Establishing short and standard duration curricula for a PC program is viable and well received by nonspecialist trainees. Future implementation should concentrate on personalized learning objectives and streamlining the content and structure of the competencies. Cooperation within various training settings (university and regional hospitals) as well as on an international level (e.g., Canada-Switzerland) may further improve the quality of the proposed training formats.

Assessing Residents in the Department of Surgery at a Tertiary Care Centre Using Mini-Clinical Evaluation Exercise (Mini-CEX)

OBJECTIVE

This study aimed to introduce, sensitize, and train our postgraduate students and faculty of the department of general surgery with the use of mini-Clinical Evaluation Exercise (mini‑CEX) and to assess the perception of students and faculty towards it.

MATERIAL AND METHODS

A cross‑sectional observational study was conducted over a period of four months. Ten surgery residents in the department were asked to volunteer to participate and five professors conducted the session. Five sessions of mini‑CEX (nine points) were conducted for each resident in different settings of the out‑patient department (OPD) and in‑patient department (IPD). A total of five skills were tested. Feedback from faculty and residents regarding the perception of mini‑CEX was also taken.

RESULTS

A statistically significant difference in mean scores of all domains was observed comparing the first and last assessment (p<0.05). Hundred percent of the residents scored superior category (7-9) in the final assessment in all domains, whereas the maximum was in a satisfactory scoring grade in 1st assessment. The time taken for the assessment significantly reduced from 1st assessment to the last assessment in OPD and IPD settings (p=0.001). The mini-CEX assessment tool got 100% feedback from faculty in terms of skill improvement, method, attitude of residents, and ability to identify gaps in knowledge. However, one assessor thought that "time given for assessment" was inadequate and more effort was required than the usual traditional assessment methods. The most identified problem faced by residents was that the "time given during assessment" was less (50%); however, overall residents also found it valid, effective, and helpful in identifying knowledge gaps and improving clinical and communication skills.

CONCLUSION

Mini‑CEX improves the learning environment in residency and also leads to improvement in medical interviewing skills, physical examination skills, humanistic qualities/professionalism, and counseling skills. So, it can be used for residency training in clinical departments.

Evolving from Didactic to Dialogic: How to Improve Faculty Development and Support Faculty Developers by Using Action Research

Problem: Since competency-based medical education has gained widespread acceptance to guide curricular reforms, faculty development has been regarded as an indispensable element to make these programs successful. Faculty developers have striven to design and deliver myriad of programs or workshops to better prepare faculty members for fulfilling their teaching roles. However, how faculty developers can improve workshop delivery by researching their teaching practices remains underexplored. Intervention: Action research aims to understand real world practices and advocates for formulation of doable plans through cycles of investigations, and ultimately contributes to claims of knowledge and a progression toward the goal of practice improvement. This methodology aligns with the aim of this study to understand how I could improve a faculty development workshop by researching my teaching practices. Context: In 2016, we conducted four cycles of action research in the context of mini-Clinical Evaluation Exercise (mini-CEX) workshops within a faculty development program aiming for developing teaching and assessment competence in faculty members. We collected multiple sources of qualitative data for thematic analysis, including my reflective journal, field notes taken by a researcher-observer, and post-workshop written reflection and feedback in portfolio from fourteen workshop attendees aiming to develop faculty teaching and assessment competence. Impact: By doing action research, I scrutinized each step as an opportunity for change, enacted adaptive practice and reflection on my teaching practices, and formulated action plans to transform a workshop design through each cycle. In so doing, my workshop evolved from didactic to dialogic with continuous improvement on enhanced engagement, focused discussion and participant empowerment through a collaborative inquiry into feedback practice. Moreover, these processes of action research also supported my growth as a faculty developer. Lessons Learned: The systematic approach of action research serves as a vehicle to enable faculty developers to investigate individual teaching practices as a self-reflective inquiry, to examine, rectify, and transform processes of program delivery, and ultimately introduce themselves as agents for change and improvement.

Family Medicine Resident Experience Toward Workplace-Based Assessment Form in Improving Clinical Teaching: An Exploratory Qualitative Study

Background

Workplace-Based Assessment (WPBA) has been widely utilized for assessing performance in training sites for both formative and summative purposes. Currently, with the recently updated duration of the family medicine (FM) training program in Saudi Arabia from four years to three years, the possible impact of such a change on assessment would need to be investigated. This objective was to explore the experiences of FM residents regarding the usage of WPBA as an assessment tool for improving clinical teaching at King Abdulaziz Hospital (KAH), Al Ahsa, Saudi Arabia.

Methods

The study involves an exploratory qualitative phenomenological approach targeting family medicine resident in KAH was used. Purposive sampling techniques were used. In this descriptive study, data was collected through the utilization of 1:1 semi-structured interviews guided by directive prompts. All recorded interviews were transcribed verbatim. An inductive analytical approach was applied for thematic analysis of transcripts.

Results

Fifteen participants were individually interviewed until data saturation was reached. The themes that emerged were organized into the categories of underlying principles of WPBA, the impact of the learning environment, associated opportunities and challenges, and making WPBA more effective. Participants expressed that the orientation provided by the program was insufficient, although the core principles were clear to them. They valued the senior peers' support and encouragement for the creation of a positive learning environment. However, time limit, workload, and a lack of optimum ideal implementation reduced the educational value and effectiveness of WPBA among senior residents.

Conclusion

The study examined residents' experiences with WPBA and concluded that low levels of satisfaction were attributed to implementation-related problems. Improvements should be made primarily in two areas: better use of available resources and more systematic prior planning. Revision and assignment of the selection process were suggested, in addition to the implementation of the new curriculum. The research will assist stakeholders in selecting and carrying out evaluation techniques that will enhance residents' abilities.

How Many Operative Performance Ratings Does a Pediatric Surgery Fellow Need to Be Deemed Practice Ready?

PURPOSE

Identifying the number of cases required for a fellow to achieve competence has been challenging. Workplace-based assessment (WBA) systems make collecting performance data practical and create the opportunity to translate WBA ratings into probabilistic statements about a fellow's likelihood of performing to a given standard on a subsequent assessment opportunity.

METHODS

We compared data from two pediatric surgery training programs that used the performance rating scale from the Society for Improving Medical Professional Learning (SIMPL). We used a Bayesian generalized linear mixed effects model to examine the relationship past and future performance for three procedures: Laparoscopic Inguinal Hernia Repair, Laparoscopic Gastrostomy Tube Placement, and Pyloromyotomy.

RESULTS

For site one, 26 faculty assessed 9 fellows on 16 procedures yielding 1094 ratings, of which 778 (71%) earned practice-ready ratings. For site two, 25 faculty rated 3 fellows on 4 unique procedures yielding 234 ratings of which 151 (65%) were deemed practice-ready. We identified similar model-based future performance expectations, with prior practice-ready ratings having a similar average effect across both sites (Site one, B = 0.25; Site two, B = 0.25). Similar prior practice-ready ratings were needed for Laparoscopic G-Tube Placement (Site one = 13; Site two = 14), while greater differences were observed for Laparoscopic Inguinal Hernia Repair (Site one = 10; Site two = 15) and Pyloromyotomy (Site one = 10; Site two = 15).

CONCLUSION

Our approach to modeling operative performance data is effective at determining future practice readiness of pediatric surgery fellows across multiple faculty and fellow groups. This method could be used to establish minimum case number requirements.

TYPE OF STUDY

Original manuscript, Study of Diagnostic Test.

LEVEL OF EVIDENCE

II.

Variation in Competence of Graduating General Surgery Trainees

OBJECTIVE

To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training.

DESIGN

Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy.

SETTING

This study was conducted using data from members of the SIMPL collaborative.

PARTICIPANTS

A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021.

RESULTS

The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy.

CONCLUSIONS

There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.

Confronting new challenges: Faculty perceptions of gaps in current laparoscopic curricula in a changing training landscape

Background

Opportunities for residents to develop laparoscopic skills have decreased with the rise in robotic operations and the development of complex, subspecialized laparoscopic operations. Given the changing training landscape, this study aimed to identify laparoscopic surgeons' perceptions of gaps in current laparoscopic skills in general surgery, obstetrics-gynecology, and urology residency programs.

Methods

Laparoscopic surgeons who operate with residents participated in semi-structured interviews. Questions addressed expectations for resident proficiency, deficits in laparoscopic surgical skills, and barriers to learning and teaching. Two authors independently coded de-identified transcripts followed by a conventional content analysis.

Results

Fourteen faculty members from thirteen subspecialties participated. Faculty identified three main areas to improve laparoscopic training across specialties: foundational knowledge, technical skills, and cognitive skills. They also recognized an overarching opportunity to address faculty development.

Conclusions

This qualitative study highlighted key deficiencies in laparoscopic training that have emerged in the current, changing era of minimally invasive surgery.

Key message

This qualitative study identified laparoscopic educators' perceptions of deficiencies in laparoscopic training. Findings emphasized the importance of incorporating high quality educational practices to optimize training in the current changing landscape of laparoscopic surgery.

Principles for teaching sonography - current status

Since many young medical residents require sonographic skills early on during training, increased attention has been paid to including sonography classes in undergraduate medical education, among both professional societies and medical educators responsible for medical licensing exams. Medical schools worldwide have developed and implemented a variety of ultrasound teaching formats.This article addresses evidence-based solutions to crucial challenges in planning and implementing undergraduate sonography education. In order to achieve a sustainable increase in practical sonographic competence, we suggest small-group classes with sufficient individual hands-on scanning time for each student. We recommend concentrating on a circumscribed topic and teaching it thoroughly and practically rather than superficially outlining a broad subject area. Provided that peer teachers undergo adequate training, student peer teachers are not inferior to physicians as teachers, as far as student satisfaction, theoretical knowledge and practical skills acquisition are concerned. The assessment of acquired practical skills should consist of practical examinations, such as an objective structured clinical examination (OSCE) or a direct observation of procedural skills (DOPS). In contrast to using healthy volunteers as training models, simulation trainers allow the demonstration of pathological findings in authentic sonographic images, with the disadvantages of unrealistically easy image acquisition, as well as the lack of interaction with the patient.

"We just don't have the resources": Supervisor perspectives on introducing workplace-based assessments into medical specialist training in South Africa

BACKGROUND

South Africa (SA) is on the brink of implementing workplace-based assessments (WBA) in all medical specialist training programmes in the country. Despite the fact that competency-based medical education (CBME) has been in place for about two decades, WBA offers new and interesting challenges. The literature indicates that WBA has resource, regulatory, educational and social complexities. Implementing WBA would therefore require a careful approach to this complex challenge. To date, insufficient exploration of WBA practices, experiences, perceptions, and aspirations in healthcare have been undertaken in South Africa or Africa. The aim of this study was to identify factors that could impact WBA implementation from the perspectives of medical specialist educators. The outcomes being reported are themes derived from reported potential barriers and enablers to WBA implementation in the SA context.

METHODS

This paper reports on the qualitative data generated from a mixed methods study that employed a parallel convergent design, utilising a self-administered online questionnaire to collect data from participants. Data was analysed thematically and inductively.

RESULTS

The themes that emerged were: Structural readiness for WBA; staff capacity to implement WBA; quality assurance; and the social dynamics of WBA.

CONCLUSIONS

Participants demonstrated impressive levels of insight into their respective working environments, producing an extensive list of barriers and enablers. Despite significant structural and social barriers, this cohort perceives the impending implementation of WBA to be a positive development in registrar training in South Africa. We make recommendations for future research, and to the medical specialist educational leaders in SA.

Meaning making about performance: A comparison of two specialty feedback cultures

INTRODUCTION

Specialty trainees often struggle to understand how well they are performing, and feedback is commonly seen as a solution to this problem. However, medical education tends to approach feedback as acontextual rather than located in a specialty-specific cultural world. This study therefore compares how specialty trainees in surgery and intensive care medicine (ICM) make meaning about the quality of their performance and the role of feedback conversations in this process.

METHODS

We conducted a qualitative interview study in the constructivist grounded theory tradition. We interviewed 17 trainees from across Australia in 2020, eight from ICM and nine from surgery, and iterated between data collection and analytic discussions. We employed open, focused, axial and theoretical coding.

FINDINGS

There were significant divergences between specialties. Surgical trainees had more opportunity to work directly with supervisors, and there was a strong link between patient outcome and quality of care, with a focus on performance information about operative skills. ICM was a highly uncertain practice environment, where patient outcome could not be relied upon as a source of performance information; valued performance information was diffuse and included tacit emotional support. These different 'specialty feedback cultures' strongly influenced how trainees orchestrated opportunities for feedback, made meaning of their performance in their day-to-day patient care tasks and 'patched together' experiences and inputs into an evolving sense of overall progress.

DISCUSSION

We identified two types of meaning-making about performance: first, trainees' understanding of an immediate performance in a patient-care task and, second, a 'patched together' sense of overall progress from incomplete performance information. This study suggests approaches to feedback should attend to both, but also take account of the cultural worlds of specialty practice, with their attendant complexities. In particular, feedback conversations could better acknowledge the variable quality of performance information and specialty specific levels of uncertainty.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

Striving to thrive or striving to survive: Professional identity constructions of medical trainees in clinical assessment activities

CONTEXT

Assessment plays a key role in competence development and the shaping of future professionals. Despite its presumed positive impacts on learning, unintended consequences of assessment have drawn increasing attention in the literature. Considering professional identities and how these can be dynamically constructed through social interactions, as in assessment contexts, our study sought to understand how assessment influences the construction of professional identities in medical trainees.

METHODS

Within social constructionism, we adopted a discursive, narrative approach to investigate the different positions trainees narrate for themselves and their assessors in clinical assessment contexts and the impact of these positions on their constructed identities. We purposively recruited 28 medical trainees (23 students and five postgraduate trainees), who took part in entry, follow-up and exit interviews of this study and submitted longitudinal audio/written diaries across nine-months of their training programs. Thematic framework and positioning analyses (focusing on how characters are linguistically positioned in narratives) were applied using an interdisciplinary teamwork approach.

RESULTS

We identified two key narrative plotlines, striving to thrive and striving to survive, across trainees' assessment narratives from 60 interviews and 133 diaries. Elements of growth, development, and improvement were identified as trainees narrated striving to thrive in assessment. Neglect, oppression and perfunctory narratives were elaborated as trainees narrated striving to survive from assessment. Nine main character tropes adopted by trainees with six key assessor character tropes were identified. Bringing these together we present our analysis of two exemplary narratives with elaboration of their wider social implications.

CONCLUSION

Adopting a discursive approach enabled us to better understand not only what identities are constructed by trainees in assessment contexts but also how they are constructed in relation to broader medical education discourses. The findings are informative for educators to reflect on, rectify and reconstruct assessment practices for better facilitating trainee identity construction.

Workplace-based assessments-Articulating the playbook

INTRODUCTION

A workplace-based assessment (WBA) is a learning recording device that is widely used in medical education globally. Although entrenched in medical curricula, and despite a substantial body of literature exploring them, it is not yet fully understood how WBAs play out in practice. Adopting a constructivist standpoint, we examine these assessments, in the workplace, using principles based upon naturalist inquiry, drawing from a theoretical framework based on Goffman's dramaturgical analogy for the presentation of self, and using qualitative research methods to articulate what is happening as learners complete them.

METHODS

Learners were voluntarily recruited to participate in the study from a single teaching hospital. Data were generated, in-situ, through observations with field notes and audiovisual recording of WBAs, along with accompanying interviews with learners.

RESULTS

Data from six learners was analysed to reveal a set of general principles-the WBA playbook. These four principles were tacit, unwritten, unofficial and learners applied them to complete their WBA proformas: (1) maintain the impression of progression, (2) manage the authenticity of the individual proforma, (3) avoid losing face with the assessor and (4) complete the proforma in an effort-efficient way. By adhering to these principles, learners expressed their understanding of their social position in their world at that time the documents were created.

DISCUSSION

This paper recognises the value of the WBA as a lived experience, and of the WBA document as a social space, where learners engage in a social performance before the readers of the proforma. Such an interpretation better represents what happens as learners undergo and record WBAs in the real-world, recognising WBAs as learner-centred, learner-driven, meaning-making phenomena. In this way, as a record of interpretation and meanings, the subjective nature of the WBA process is a strength to be harnessed, rather than a weakness to be glossed over.

Interprofessional population health advocacy: Developing and implementing a panel management curriculum in five Veterans Administration primary care practices

Health care systems expect primary care clinicians to manage panels of patients and improve population health, yet few have been trained to do so. An interprofessional panel management (PM) curriculum is one possible strategy to address this training gap and supply future primary care practices with clinicians and teams prepared to work together to improve the health of individual patients and populations. This paper describes a Veterans Administration (VA) sponsored multi-site interprofessional PM curriculum development effort. Five VA Centers of Excellence in Primary Care Education collaborated to identify a common set of interprofessionally relevant desired learning outcomes (DLOs) for the PM and to develop assessment instruments for monitoring trainees' PM learning. Authors cataloged teaching and learning activities across sites. Results from pilot testing were systematically discussed leading to iterative revisions of curricular elements. Authors completed a retrospective self-assessment of curriculum implementation for the academic year 2015-16 using a 5-point scale: contemplation (score = 0), pilot (1), action (2), maintenance (3), and embedded (4). Implementation scores were analyzed using descriptive statistics. DLOs were organized into five categories (individual patients, populations, guidelines/measures, teamwork, and improvement) along with a developmental continuum and mapped to program competencies. Instruction and implementation varied across sites based on resources and priorities. Between 2015 and 2016, 159 trainees (internal medicine residents, nurse practitioner students and residents, pharmacy residents, and psychology post-doctoral fellows) participated in the PM curriculum. Curriculum implementation scores for guidelines/measures and improvement DLOs were similar for all trainees; scores for individual patients, populations, and teamwork DLOs were more advanced for nurse practitioner and physician trainees. In conclusion, collaboratively identified DLOs for PM guided development of assessment instruments and instructional approaches for panel management activities in interprofessional teams. This PM curriculum and associated tools provide resources for educators in other settings.

Process evaluation of paediatric fellowship training programs at a University Hospital in Pakistan

BACKGROUND

Fellowship programs offer training in a subspecialty focusing on distinct and advanced clinical/academic skills. This advanced postgraduate training allows physicians, who desire a more specialized practice, to further develop clinical, academic, research, and leadership/administrative skills. The Aga Khan University (AKU) is one of the few institutes offering paediatric sub-specialty training in Pakistan. We aimed to evaluate the current Paediatric fellowship programs at AKU.

METHODS

Process evaluation of six paediatric fellowship programs (cardiology, neurology, endocrinology, critical care medicine, neonatology, and infectious disease) was conducted from September 2020 to April 2021 by senior clinicians and medical educationists. Evidence was collected through document review (using existing postgraduate medical education program information form), observation of teaching and learning support, and focused group discussions/interviews with program faculty and fellows were conducted. A review of the evaluation report was done as part of this study. This study received an exemption from the ethical review committee. The quantitative data were analyzed using SPSS (22.0) while the reports of discussion with fellows and friends underwent content analysis.

RESULTS

All fellowship programs met the criteria for having a robust competency-based fellowship curriculum as per the institutional and national guidelines. Formative assessment in the form of continuous evaluation was found to be integrated into all the fellowship programs, however, most of the programs were found to lack a summative assessment plan. Fellows in training and program faculty were satisfied with the opportunities for mentorship, teaching, and learning. Thematic analysis of the discussion reports with faculty and fellows revealed three key themes including, program aspects translating into strengthening the training, gaps in the training program in delivering the expectations, and making ways to reach par excellence.

CONCLUSIONS

The process evaluation of paediatric fellowship programs provided an opportunity to holistically review the current strengths and quality of the training in individual programs along with the unmet needs of the trainees. This will help the program stakeholders to prioritize, align and allocate the resources to further enhance the quality of training and outcome of individual fellowship programs to ensure wider impacts at a regional, national, and international health system level.

An international comparison of student nurse practitioner diagnostic reasoning skills

BACKGROUND

Nurse practitioners (NPs) play a pivotal role in health care provision. Diagnostic reasoning is an important core skill of advanced practice.

PURPOSE

The purpose of this study was to compare diagnostic reasoning skills of NP students. It also identified the variability in clinical teaching components of a sample of international NP curricula.

METHODS

An international group of NP students completed an online survey using a validated diagnostic reasoning scale during the first year of their NP program. Program faculty surveyed provided data on core curricula.

RESULTS

The NP students' ( n = 152) mean diagnostic thinking inventory (DTI) score was 142, flexibility in thinking subscale score mean of 73 with a knowledge structure in memory of 69. The programs surveyed required bioscience courses. Most programs provided opportunities for students to practice diagnostic thinking using individual precepted clinical hours (range 500-950) and objective structured clinical examinations.

CONCLUSION

The lower scores of this group of NP students were similar to other NP students and first-year medical residents. Higher mean scores in the DTI reflect expertise and are developed over time. Courses providing a foundation of biomedical knowledge were identified in each program, with opportunities for the NP students to practice diagnostic thinking using objective structured clinical examinations and clinical practice hours.

IMPLICATIONS

The use of the diagnostic reasoning inventory is a useful tool for evaluating student NP's diagnostic reasoning during their NP program. Nurse practitioner programs should consider the provision of dedicated clinical hours, including supervised clinical practice experiences and objective structured clinical examinations to improve diagnostic reasoning.

The Use, Effectiveness, and Impact of Workplace-Based Assessments on Teaching, Supervision and Feedback Across Surgical Specialties

OBJECTIVE

To investigate the use and effectiveness of Workplace-based assessments (WBAs) and their impact on training, feedback, and perioperative teaching in surgical training programs.

DESIGN

A mixed methods cross-sectional, national electronic survey was conducted with surgical trainees and consultant trainers.

SETTINGS

The trainees and supervising faculty were from all 8 major surgical training universities across 11 surgical disciplines in South Africa.

PARTICIPANTS

A total of 108 surgical trainees and 41 supervising consultant trainers from 11 surgical disciplines across 8 surgical training universities responded to the survey.

RESULTS

The most significant educational gap identified by both the surgical trainees and trainers across all surgical disciplines was inadequate perioperative feedback. A third of the respondents were currently using workplace-based assessments. The WBA users (both trainees and trainers) had a higher rating for the general quality of surgical feedback than WBA nonusers (p = 0.02). WBA users also had a higher rating for the general quality of feedback given to trainees on their skills and competence (p = 0.04) and a higher rating for trainee supervision (p = 0.01) and the specialist training program overall (p = 0.01). The WBA users also had a higher rating for the assessment of competencies such as the trainee as an effective communicator (p < 0.01) and collaborator (p = 0.04).

CONCLUSION

This study found that the use of WBAs enhances the quality and effectiveness of feedback in surgical training programs. We also found that the use of WBAs enhance perioperative teaching and learning and improves the assessment of relational competencies. This was also associated with high ratings for the quality of trainee supervision. Faculty and trainee development, strengthening the trainee-trainer relationship, and integrating iterative stakeholder feedback could help realize the full potential of WBAs to augment surgical training across disciplines.

"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education

Assessing learners is foundational to their training and developmental growth throughout the medical education continuum. However, growing evidence shows the prevalence and impact of harmful bias in assessments in medical education, accelerating the urgency to identify solutions. Assessment bias presents a critical problem for all stages of learning and the broader educational system. Bias poses significant challenges to learners, disrupts the learning environment, and threatens the pathway and transition of learners into health professionals. While the topic of assessment bias has been examined within the context of measurement literature, limited guidance and solutions exist for learners in medical education, particularly in the clinical environment. This article presents an overview of assessment bias, focusing on clinical learners. A definition of bias and its manifestations in assessments are presented. Consequences of assessment bias are discussed within the contexts of validity and fairness and their impact on learners, patients/caregivers, and the broader field of medicine. Messick's unified validity framework is used to contextualize assessment bias; in addition, perspectives from sociocultural contexts are incorporated into the discussion to elaborate the nuanced implications in the clinical training environment. Discussions of these topics are conceptualized within the literature and the interventions used to date. The article concludes with practical recommendations to overcome bias and to develop an ideal assessment system. Recommendations address articulating values to guide assessment, designing assessment to foster learning and outcomes, attending to assessment procedures, promoting continuous quality improvement of assessment, and fostering equitable learning and assessment environments.

Demise of the USMLE Step-2 CS exam: Rationalizing a way forward

The COVID-19 pandemic has compelled rethinking and changes in medical education, the most controversial perhaps being the cancelation of USMLE Step-2 Clinical Skills exam (Step-2 CS). What started in March of 2020 as suspension of this professional licensure exam, because of concerns about infection risk for examinees, standardized patients (SPs), and administrators, soon became permanent cancelation in January 2021. Expectedly, it triggered debate in medical education circles. Positively, however, the USMLE regulatory agencies (NBME and FSMB) saw an opportunity to innovate an exam tainted with perceptions of validity deficits, cost, examinee inconvenience, and worries about future pandemics; they therefore called for a public debate to fashion a way forward. We have approached the issue by defining Clinical Skills (CS), exploring its epistemology and historic evolution, including assessment modalities from Hippocratic times to the modern era. We defined CS as the art of medicine manifest in the physician-patient encounter as history taking (driven by communication skills and cultural competence) and physical examination. We classified CS components into knowledge and psychomotor skill domains, established their relative importance in the physician process (clinical reasoning) of diagnosis, thus establishing a theoretical framework for developing valid, reliable, feasible, fair, and verifiable CS assessment. Given the concerns for COVID-19 and future pandemics, we established that CS can largely be assessed remotely, and what could not, can be assessed locally (school/regional consortia level) as part of a USMLE-regulated/supervised assessment regimen with established national standards, thus maintaining USMLE's fiduciary responsibilities. We have suggested a national/regional program for faculty development in CS curriculum development, and assessment, including standard setting skills. This pool of expert faculty will form the nucleus of our proposed USMLE-regulated External Peer Review Initiative (EPRI). Finally, we suggest that CS evolves into an academic discipline/department of its own, rooted in scholarship.

Exploring the Impact of Entrustable Professional Activities on Feedback Culture: A Qualitative Study of Anesthesiology Residents and Attendings

PURPOSE

Entrustable professional activities (EPAs) were introduced as a potential way to optimize workplace-based assessments. Yet, recent studies suggest that EPAs have not yet overcome all of the challenges to implementing meaningful feedback. The aim of this study was to explore the extent to which the introduction of EPAs via mobile app impacts feedback culture as experienced by anesthesiology residents and attending physicians.

METHOD

Using a constructivist grounded theory approach, the authors interviewed a purposive and theoretical sample of residents (n = 11) and attendings (n = 11) at the Institute of Anaesthesiology, University Hospital of Zurich, where EPAs had recently been implemented. Interviews took place between February and December 2021. Data collection and analysis were conducted iteratively. The authors used open, axial, and selective coding to gain knowledge and understanding on the interplay of EPAs and feedback culture.

RESULTS

Participants reflected on a number of changes in their day-to-day experience of feedback culture with the implementation of EPAs. Three main mechanisms were instrumental in this process: lowering the feedback threshold, change in feedback focus, and gamification. Participants felt a lower threshold to feedback seeking and giving and that the frequency of feedback conversations increased and tended to be more focused on a specific topic and shorter, while feedback content tended to focus more on technical skills and more attention was given to average performances. Residents indicated that the app-based approach fostered a game-like motivation to "climb levels," while attendings did not perceive a game-like experience.

CONCLUSIONS

EPAs may offer a solution to problems of infrequent occurrence of feedback and invite attention to average performances and technical competencies, but may come at the expense of feedback on nontechnical skills. This study suggests that feedback culture and feedback instruments have a mutually interacting influence on each other.

A systematic review of effective quality feedback measurement tools used in clinical skills assessment

BACKGROUND

Objective Structured Clinical Examination (OSCE) is a valid tool to assess the clinical skills of medical students. Feedback after OSCE is essential for student improvement and safe clinical practice. Many examiners do not provide helpful or insightful feedback in the text space provided after OSCE stations, which may adversely affect learning outcomes. The aim of this systematic review was to identify the best determinants for quality written feedback in the field of medicine.   Methods: PubMed, Medline, Embase, CINHAL, Scopus, and Web of Science were searched for relevant literature up to February 2021. We included studies that described the quality of good/effective feedback in clinical skills assessment in the field of medicine. Four independent reviewers extracted determinants used to assess the quality of written feedback. The percentage agreement and kappa coefficients were calculated for each determinant. The ROBINS-I (Risk Of Bias In Non-randomized Studies of Interventions) tool was used to assess the risk of bias.

RESULTS

14 studies were included in this systematic review. 10 determinants were identified for assessing feedback. The determinants with the highest agreement among reviewers were specific, described gap, balanced, constructive and behavioural; with kappa values of 0.79, 0.45, 0.33, 0.33 and 0.26 respectively. All other determinants had low agreement (kappa values below 0.22) indicating that even though they have been used in the literature, they might not be applicable for good quality feedback. The risk of bias was low or moderate overall.

CONCLUSIONS

This work suggests that good quality written feedback should be specific, balanced, and constructive in nature, and should describe the gap in student learning as well as observed behavioural actions in the exams.  Integrating these determinants in OSCE assessment will help guide and support educators for providing effective feedback for the learner.

Challenges facing standardised patients representing equity-deserving groups: Insights from health care educators

INTRODUCTION

Health professions training programmes increasingly rely on standardised patient (SP) programmes to integrate equity-deserving groups into learning and assessment opportunities. However, little is known about the optimal approach, and many SP programmes struggle to meet these growing needs. This study explored insights from health care educators working with SP programmes to deliver curricular content around equity-deserving groups.

METHODS

We interviewed 14 key informants in 2021 who were involved in creating or managing SP-based education. Verbatim transcripts were analysed in an iterative coding process, anchored by qualitative content analysis methodology and informed by two theoretical frameworks: sociologic translation and simulation design. Repeated cycles of data collection and analyses continued until themes could be constructed, aligned with existing theories and grounded in empirical data, with sufficient relevance and robustness to inform educators and curricular leads.

RESULTS

Three themes were constructed: (i) creating safety for SPs paid to be vulnerable, (ii) fidelity as an issue broader than who plays the role and (iii) engaging equity-deserving groups. SP work involving traditionally marginalised groups risk re-traumatization, highlighting the importance of (i) informed consent in recruiting SPs, (ii) separating role portrayal from lived experiences, (iii) adequately preparing learners and facilitators, (iv) creating time-outs and escapes for SPs and (v) building opportunity for de-roling with community support.

CONCLUSIONS

SP programmes are well positioned to be allies and advocates to equity-deserving groups and to collaborate and share governance of the educational development process from its outset. SP programmes can support the delivery of curricular content around equity-deserving groups by advocating with curricular leadership, building relationships with community partners, facilitating co-creation and co-delivery of educational content and building safety into simulation.

Preceptors' considerations when entrusting professional activities to postgraduate nursing students: A qualitative study

BACKGROUND

Nurse-preceptors regularly struggle to evaluate students' readiness to take care of patients unsupervised, even with sophisticated workplace-based assessment tools. Preceptors' gut feelings are not always captured well, but are critical for judgement of readiness for learner entrustment with care tasks. Studies in medical education report features that clinicians consider important when trusting students with clinical responsibilities that might also apply in nursing.

OBJECTIVES

To unravel preceptors' considerations when entrusting professional activities to postgraduate nursing students. The findings may contribute to the improvement of workplace-based assessments and the training of preceptors.

METHODS

Thematic analysis of semi-structured interviews with sixteen nurse-preceptors from three postgraduate nursing specialisations in Dutch hospitals.

RESULTS

Three themes emerged: CONCLUSIONS: For preceptors of postgraduate nursing students, entrustment requires more than merely insight into objectively measurable competencies. Entrusting is accompanied by subjectivity related to what preceptors expected of students. These expectations are in line with suggested factors in the literature-capability, integrity, reliability, agency, and humility-considered before entrusting students with clinical responsibilities identified in medical training. Entrusting is also accompanied by what preceptors realise about their own role in entrustment decisions. Combining different information sources made assessment more transparent and the implicit more explicit.

Assessing trainee's performance using reported observations: Perceptions of nurse meta-assessors

BACKGROUND

Educational and health care organizations who prepare meta-assessors to fulfill their role in the assessment of trainees' performance based on reported observations have little literature to rely on. While the assessment of trainees' performance based on reported observations has been operationalized, we have yet to understand the elements that can affect its quality fully. Closing this gap in the literature will provide valuable insight that could inform the implementation and quality monitoring of the assessment of trainees' performance based on reported observations.

OBJECTIVES

The purpose of this study was to explore the elements to consider in the assessment of trainees' performance based on reported observations from the perspectives of meta-assessors.

METHODS

Design, Settings, Participants, data collection and analysis. The authors adopted Sandelowski's qualitative descriptive approach to interview nurse meta-assessors from two nursing programs. A semi-structured interview guide was used to document the elements to consider in the assessment of nursing trainees' performance based on reported observations, and a survey was used to collect sociodemographic data. The authors conducted a thematic analysis of the interview transcripts.

RESULTS

Thirteen meta-assessors participated in the study. Three core themes were identified: (1) meta-assessors' appropriation of their perceived assessment roles and activities, (2) team climate of information sharing, and (3) challenges associated with the assessment of trainees' performance based on reported observations. Each theme is comprised of several sub themes.

CONCLUSIONS

To optimize the quality of the assessment of the trainee's performance based on reported observations and ratings, HPE programs might consider how to clarify better the meta-assessor's roles and activities, as well as how interventions could be created to promote a climate of information sharing and to address the challenges identified. This work will guide educational and health care organizations for better preparation and support for meta-assessors and preceptors.

How workplace-based assessments guide learning in postgraduate education: A scoping review

INTRODUCTION

Competency-based medical education (CBME) led to the widespread adoption of workplace-based assessment (WBA) with the promise of achieving assessment for learning. Despite this, studies have illustrated tensions between the summative and formative role of WBA which undermine learning goals. Models of workplace-based learning (WBL) provide insight, however, these models excluded WBA. This scoping review synthesizes the primary literature addressing the role of WBA to guide learning in postgraduate medical education, with the goal of identifying gaps to address in future studies.

METHODS

The search was applied to OVID Medline, Web of Science, ERIC and CINAHL databases, articles up to September 2020 were included. Titles and abstracts were screened by two reviewers, followed by a full text review. Two members independently extracted and analysed quantitative and qualitative data using a descriptive-analytic technique rooted in Billett's four premises of WBL. Themes were synthesized and discussed until consensus.

RESULTS

All 33 papers focused on the perception of learning through WBA. The majority applied qualitative methodology (70%), and 12 studies (36%) made explicit reference to theory. Aligning with Billett's first premise, results reinforce that learning always occurs in the workplace. WBA helped guide learning goals and enhanced feedback frequency and specificity. Billett's remaining premises provided an important lens to understand how tensions that existed in WBL have been exacerbated with frequent WBA. As individuals engage in both work and WBA, they are slowly transforming the workplace. Culture and context frame individual experiences and the perceived authenticity of WBA. Finally, individuals will have different goals, and learn different things, from the same experience.

CONCLUSION

Analysing WBA literature through the lens of WBL theory allows us to reframe previously described tensions. We propose that future studies attend to learning theory, and demonstrate alignment with philosophical position, to advance our understanding of assessment-for-learning in the workplace.

'Ticked off'? Can a new outcomes-based postgraduate curriculum utilising programmatic assessment reduce assessment burden in Intensive Care Medicine?

Context

Increasing dissatisfaction with existing methods of assessment in the workplace alongside a national drive towards outcomes-based postgraduate curricula led to a recent overhaul of the way Intensive Care Medicine (ICM) trainees are assessed in the United Kingdom. Programmatic assessment methodology was utilised; the existing 'tick-box' approach using workplace-based assessment to demonstrate competencies was de-emphasised and the expertise of trainers used to assess capability relating to fewer, high-level outcomes related to distinct areas of specialist practice.

Methods

A thematic analysis was undertaken investigating attitudes from 125 key stakeholders, including trainees and trainers, towards the new assessment strategy in relation to impact on assessment burden and acceptability.

Results

This qualitative study suggests increased satisfaction with the transition to an outcomes-based model with capability judged by educational supervisors. However, reflecting frustration relating to current assessment in the workplace, participants felt assessment burden has been significantly reduced. The approach taken was felt to be an improved method for assessing professional practice; there was enthusiasm for this change. However, this research highlights trainee and trainer anxiety regarding how to 'pass' these expert judgement decisions of capability in the real world. Additionally, concerns relating to the impact on subgroups of trainees due to the potential influence of implicit biases on the resultant fewer but 'higher stakes' interrogative judgements became apparent.

Conclusion

The move further towards a constructivist paradigm in workplace assessment in ICM reduces assessment burden yet can provoke anxiety amongst trainees and trainers requiring considered implementation. Furthermore, the perception of potential for bias in global judgements of performance requires further exploration.

Introduction of Mini Clinical Evaluation Exercise as an Assessment Tool for M.B.B.S. Interns in the Department of Orthopaedics

Introduction

Mini-CEX helps in judicious use of competencies in authentic settings by simultaneously assessing clinical skills of trainees and providing feedback on their performance. As assessment of M.B.B.S. Interns for their competency in clinical examination skills in Department of Orthopaedics is lacking, this study is taken up to introduce Mini-CEX for M.B.B.S. Interns, by sensitising faculty and interns.

Materials and Methods

A Quasi-experimental study was conducted during June to December 2020, among 60 interns posted in the Department of Orthopaedics. After obtaining IEC Clearance and written informed consent from the study participants, they were sensitised and exposed to five Mini-CEX clinical encounters involving examination of a patient with knee/other joint disorder in the Outpatient/Inpatient clinical setting, with eight faculty. The study tool used was Mini-CEX questionnaire developed by American Board of Internal Medicine (ABIM). Case specific feedback was provided to interns using sandwich technique. The reflections and perceptions of interns and faculty were obtained after completion of all Mini-CEX encounters.

Results

96.7% encounters were conducted in OPD during first encounter. On an average, one Mini-CEX encounter lasted for 17 min. Interns had an overall score for the various domains ranging from 5.38 to 5.58. Comparison of mean scores showed a statistically significant improvement (p value < 0.0001). All the assessors were satisfied with Mini-CEX as an assessment tool.

Conclusion

Interns and faculty opined that Mini-CEX improves clinical examination skills and professional development as focus is on the outcome and learning process, with multiple sampling in a longitudinal manner.

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery

Importance

Understanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education.

Objective

To evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees.

Design, Setting, and Participants

This case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US. The study included ratings for 2605 trainees from 1884 attending surgeon raters. Analyses were conducted between September 2021 and December 2021 using bayesian generalized linear mixed-effects models and marginal predicted probabilities.

Exposures

Longitudinal SIMPL ratings.

Main Outcomes and Measures

Performance expectations for 193 unique general surgery procedures based on an individual trainee's prior successful ratings for a procedure, clinical year of training, and month of the academic year.

Results

Using 63 248 SIMPL ratings, the association between prior and future performance was positive (β, 0.13; 95% credible interval [CrI], 0.12-0.15). The largest source of variation was postgraduate year (α, 3.15; 95% CrI, 1.66-6.03), with rater (α, 1.69; 95% CrI, 1.60-1.78), procedure (α, 1.35; 95% CrI, 1.22-1.51), case complexity (α, 1.30; 95% CrI, 0.42-3.66), and trainee (α, 0.99; 95% CrI, 0.94-1.04) accounting for significant variation in practice ready ratings. After marginalizing overcomplexity and trainee and holding rater constant, mean predicted probabilities had strong overall discrimination (area under the receiver operating characteristic curve, 0.81) and were well calibrated.

Conclusions and Relevance

In this study, prior performance was associated with future performance. This association, combined with an overall modeling strategy that accounted for various facets of an assessment task, may offer a strategy for quantifying competence as performance expectations.

Do clinical interview transcripts generated by speech recognition software improve clinical reasoning performance in mock patient encounters? A prospective observational study

BACKGROUND

To investigate whether speech recognition software for generating interview transcripts can provide more specific and precise feedback for evaluating medical interviews.

METHODS

The effects of the two feedback methods on student performance in medical interviews were compared using a prospective observational trial. Seventy-nine medical students in a clinical clerkship were assigned to receive either speech-recognition feedback (n = 39; SRS feedback group) or voice-recording feedback (n = 40; IC recorder feedback group). All students' medical interviewing skills during mock patient encounters were assessed twice, first using a mini-clinical evaluation exercise (mini-CEX) and then a checklist. Medical students then made the most appropriate diagnoses based on medical interviews. The diagnostic accuracy, mini-CEX, and checklist scores of the two groups were compared.

RESULTS

According to the study results, the mean diagnostic accuracy rate (SRS feedback group:1st mock 51.3%, 2nd mock 89.7%; IC recorder feedback group, 57.5%-67.5%; F(1, 77) = 4.0; p = 0.049), mini-CEX scores for overall clinical competence (SRS feedback group: 1st mock 5.2 ± 1.1, 2nd mock 7.4 ± 0.9; IC recorder feedback group: 1st mock 5.6 ± 1.4, 2nd mock 6.1 ± 1.2; F(1, 77) = 35.7; p < 0.001), and checklist scores for clinical performance (SRS feedback group: 1st mock 12.2 ± 2.4, 2nd mock 16.1 ± 1.7; IC recorder feedback group: 1st mock 13.1 ± 2.5, 2nd mock 13.8 ± 2.6; F(1, 77) = 26.1; p < 0.001) were higher with speech recognition-based feedback.

CONCLUSIONS

Speech-recognition-based feedback leads to higher diagnostic accuracy rates and higher mini-CEX and checklist scores.

TRIAL REGISTRATION

This study was registered in the Japan Registry of Clinical Trials on June 14, 2022. Due to our misunderstanding of the trial registration requirements, we registered the trial retrospectively. This study was registered in the Japan Registry of Clinical Trials on 7/7/2022 (Clinical trial registration number: jRCT1030220188).

Entrustable Professional Activities and Learning: The Postgraduate Trainee Perspective

OBJECTIVE

Entrustable professional activities (EPAs) are used as clinical activities in postgraduate psychiatry training in Australasia. This study aimed to explore psychiatry trainees' perceptions of the impact of EPAs on their motivation and learning.

METHODS

A constructivist grounded theory approach was used to conceptualize the impact of EPAs on trainees' motivation and learning. A purposive sample of trainees was recruited from across New Zealand. Semi-structured individual interviews were used for data collection and continued until theoretical saturation was reached.

RESULTS

The impact of EPAs on learning was mediated by the trainee's appraisals of subjective control, value, and the costs of engaging with EPAs. When appraisals were positive, EPAs encouraged a focus on particular learning needs and structured learning with the supervisor. However, when appraisals were negative, EPAs encouraged a superficial approach to learning. Trainee appraisals and their subsequent impact on motivation and learning were most affected by EPA granularity, alignment of EPAs with clinical practice, and the supervisor's conscientiousness in their approach to EPAs.

CONCLUSIONS

To stimulate learning, EPAs must be valued by both trainees and supervisors as constituting a coherent work-based curriculum that encompasses the key fellowship competencies. If EPAs are to be effective as clinical tasks for learning, ongoing faculty development must be the leading priority.

TEAMwork! An Innovative Hospital Medicine Teaching Program to Enhance Learners' Educational Experience

PROBLEM

Workplace-based learning and assessment are dependent on frequent observations, feedback, teaching, and evaluations. Yet high physician workload and multiple learners limit these efforts. A local needs assessment demonstrated insufficient direct observations, small-group teaching, and high-quality evaluations for learners. In response, the authors developed and implemented Teaching Excellence Among Medical providers (TEAM), a teaching program to enhance the educational experience of learners in pediatric hospital medicine (PHM).

APPROACH

The TEAM program, grounded in a self-regulated learning framework, launched in 2019. The University of Colorado Department of Pediatrics provided salary support. TEAM hospitalists completed professional development for teaching and feedback skills. During their TEAM shifts, they observed, provided feedback, completed evaluations, and led teaching sessions for learners on PHM rotations. The program's impact was evaluated by using an electronic work tracking form and surveying learners, attending hospitalists, and TEAM hospitalists, leading to iterative program improvements.

OUTCOMES

Over an 18-month period, TEAM hospitalists completed 1,573 direct observations, 265 teaching sessions attended by 1,921 learners, and more than 497 learner evaluations. Survey results showed that 95% of learners indicated TEAM was "very effective" or "somewhat effective" in helping achieve individualized learning goals, and a similar percentage "strongly agreed" or "agreed" that TEAM hospitalists were effective educators. Eighty-four percent of attending hospitalist survey respondents reported TEAM contributed "very much" or "somewhat" to learner education. Attending hospitalists particularly appreciated TEAM's ability to focus on learners identified as struggling, while TEAM hospitalists rated observation of rounds and small-group teaching sessions as the most valuable TEAM activities.

NEXT STEPS

The TEAM program employed concepts of self-regulated learning to enhance education in PHM through direct observation, feedback, written evaluations, and teaching efforts. This program can mitigate many challenges facing attending physicians and can serve as a model for other institutions. Next steps include evaluation of higher-level learning outcomes.