The Clinical Learning Environment and Workplace-Based Assessment: Frameworks, Strategies, and Implementation

Linking assessment to real life practice - comparing work based assessments and objective structured clinical examinations using mystery shopping

Objective Structured Clinical Examinations (OSCEs) and Work Based Assessments (WBAs) are the mainstays of assessing clinical competency in health professions' education. Underpinned by the extrapolation inference in Kane's Validity Framework, the purpose of this study is to determine whether OSCEs translate to real life performance by comparing students' OSCE performance to their performance in real-life (as a WBA) using the same clinical scenario, and to understand factors that affect students' performance. A sequential explanatory mixed methods approach where a grade comparison between students' performance in their OSCE and WBA was performed. Students were third year pharmacy undergraduates on placement at a community pharmacy in 2022. The WBA was conducted by a simulated patient, unbeknownst to students and indistinguishable from a genuine patient, visiting the pharmacy asking for health advice. The simulated patient was referred to as a 'mystery shopper' and the process to 'mystery shopping' in this manuscript. Community pharmacy is an ideal setting for real-time observation and mystery shopping as staff can be accessed without appointment. The students' provision of care and clinical knowledge was assessed by the mystery shopper using the same clinical checklist the student was assessed from in the OSCE. Students who had the WBA conducted were then invited to participate in semi-structured interviews to discuss their experiences in both settings. Overall, 92 mystery shopper (WBA) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 41.7% [IQR 28.3] and significantly lower compared to the OSCE score 80.9% [IQR 19.0] in all participants (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage real-life patients. Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as opposed to an unfamiliar workplace. As suggested by the activity theory, the performance of the student can be driven by their motivation which differed in the two contexts.

Initial Experience of Using First-Person Wearable Video Recording Technology During Central Venous Catheter Placement in the Cardiac Operating Room

OBJECTIVE

The aim of this study was to use wearable video-recording technology to measure precisely the timing of discrete events during perioperative central venous catheter (CVC) placements.

DESIGN

A single-center, observational, exploratory study on the use of wearable video-recording technology during intraoperative CVC placement.

SETTING

The study was conducted at a University Hospital.

PARTICIPANTS

Clinical anesthesia residents, cardiothoracic anesthesia fellows, and attending anesthesiologists participated in this study.

INTERVENTIONS

Participants were asked to use eye-tracking glasses prior to the placement of a CVC in the cardiac operating rooms. No other instruction was given to the participants.

MEASUREMENTS AND MAIN RESULTS

The authors measured the total time to complete the CVC placement, phase-specific time, and specific times of interest. They compared these times across 3 training levels and tested differences with analysis of variance. The authors' findings indicated significant differences in total CVC placement time when the procedure included a pulmonary artery catheter insertion (1,170 ± 364, 923 ± 272, and 596 ± 226 seconds; F2,63 = 12.71, p < 0.0001). Additionally, they found differences in interval times and times of interest. The authors observed a reduction of variability with increasing experience during the CVC placement phase.

CONCLUSIONS

In this observational study, the study authors describe their experience using first-person wearable video-recording technology to precisely measure the timing of discrete events during CVC placement by anesthesia residents and anesthesiologists. Future work will leverage the eye-tracking capabilities of the existing hardware to identify areas of inefficiency to develop actionable targets for interventions that could improve trainee performance and patient safety.

Students as Teachers: Development and Implementation of a Workplace-Based Assessment

Students as Teachers programs are prevalent, though assessments within these programs are lacking. A workplace-based assessment for clinical teaching was developed to foster formative feedback and support learner growth. Feedback narratives were analyzed to identify student teaching behaviors and demonstrated the themes medical knowledge, professionalism, communication, and teaching skills, which were subcategorized as clinical relevance, learner stage appropriateness, use of evidence-based teaching strategies, learning environment, feedback-related, and time-appropriate. This analysis supports the use of the assessment form for student teachers in the clinical environment as students received construct-relevant feedback from various raters while teaching in multiple settings.

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

How workplace-based assessments guide learning in postgraduate education: A scoping review

INTRODUCTION

Competency-based medical education (CBME) led to the widespread adoption of workplace-based assessment (WBA) with the promise of achieving assessment for learning. Despite this, studies have illustrated tensions between the summative and formative role of WBA which undermine learning goals. Models of workplace-based learning (WBL) provide insight, however, these models excluded WBA. This scoping review synthesizes the primary literature addressing the role of WBA to guide learning in postgraduate medical education, with the goal of identifying gaps to address in future studies.

METHODS

The search was applied to OVID Medline, Web of Science, ERIC and CINAHL databases, articles up to September 2020 were included. Titles and abstracts were screened by two reviewers, followed by a full text review. Two members independently extracted and analysed quantitative and qualitative data using a descriptive-analytic technique rooted in Billett's four premises of WBL. Themes were synthesized and discussed until consensus.

RESULTS

All 33 papers focused on the perception of learning through WBA. The majority applied qualitative methodology (70%), and 12 studies (36%) made explicit reference to theory. Aligning with Billett's first premise, results reinforce that learning always occurs in the workplace. WBA helped guide learning goals and enhanced feedback frequency and specificity. Billett's remaining premises provided an important lens to understand how tensions that existed in WBL have been exacerbated with frequent WBA. As individuals engage in both work and WBA, they are slowly transforming the workplace. Culture and context frame individual experiences and the perceived authenticity of WBA. Finally, individuals will have different goals, and learn different things, from the same experience.

CONCLUSION

Analysing WBA literature through the lens of WBL theory allows us to reframe previously described tensions. We propose that future studies attend to learning theory, and demonstrate alignment with philosophical position, to advance our understanding of assessment-for-learning in the workplace.

In-hospital education of parents of newborns may benefit from competency-based education: A qualitative focus group and interview study among health professionals

AIMS/OBJECTIVES

The aim of this study was to appraise health professionals' self-reported practices in educating parents of hospitalised newborns from the perspective of competency-based education and to identify areas for improvement of parental learning.

BACKGROUND

Patient education is essential to achieve autonomy in parents of hospitalised newborns. The literature provides descriptions of the use of various components of competency-based education in patient education. This suggests that competency-based education is a valuable concept for patient education.

DESIGN

A case-based qualitative study.

METHODS

Three focus group discussions were conducted and 28 semi-structured interviews with 45 health professionals who practice in a hospital setting that is designed to empower parents. The data were analysed with a framework analysis approach, using a framework of competency-based education themes for a combined inductive and deductive content data analysis. The recommendations of the Standards for Reporting Qualitative Research checklist were followed.

FINDINGS

Two themes of competency-based education emerged as evidently operationalised: (1) 'Learning climate' and (2) 'Role modeling'. Five themes emerged as incompletely operationalised: (1) 'Parent curriculum based on inter-professional consensus'; (2) 'Transparency about the competencies needed'; (3) 'Access to teaching'; (4) 'Assessing and reporting results'; and (5) 'Proficiency statements based on autonomy expectations'. Two themes did not emerge: (1) 'Empowering parents to be active learners' and (2) 'Evaluation and improvement of the education program'.

CONCLUSIONS

Parent education is at risk of being merely on a master-apprentice model and may be more effective if it is designed on competency-based education principles. Identified areas for improvement are empowering parents to be 'active learners' and by involving them in the evaluation and improvement of the educational program. Parent education in neonatal health care may benefit from an appraisal based on competency-based education themes.

RELEVANCE TO CLINICAL PRACTICE

Appraising parent education based on competency-based education principles is feasible for improving the learning process towards parent autonomy.

Effect of Workplace-Based Assessment Utilization as a Formative Assessment for Learning Among Family Medicine Postgraduates at the Faculty of Medicine, Menoufia University: A Prospective Study

Background Workplace-based assessment (WBA) is a group of assessment approaches that assesses the trainees' performance through their observation and monitoring in real clinical settings and then provides them with constructive and relevant feedback. Many WBA tools are available, including the mini-clinical evaluation exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussions, and multisource feedback (peers, seniors, and patients). A WBA can help medical students improve their clinical competencies and ensure that qualified physicians graduate. Methods This prospective study was done in the family medicine department at the Menoufia Faculty of Medicine in Egypt and passed through two phases. Phase I was introducing an orientation lecture for family medicine staff and a convenient sample of 21 family medicine postgraduates about WBA. Phase II was conducting a monthly mini-CEX and DOPS for the postgraduates. Finally, students' satisfaction with the WBA was assessed, and all collected data were analyzed via Statistical Package for Social Science (SPSS) version 23 (IBM Corp., Armonk, NY). Results A total of 105 feedback sheets were obtained. These feedback sheets were subdivided into 63 mini-CEX feedback sheets (21 sheets from each mini-CEX session for three sessions) and 42 DOPS feedback sheets (21 sheets from each DOPS session for two sessions), all of which were collected and analyzed. A significant improvement was detected in the mini-CEX and DOPS feedback scores of the postgraduates throughout the consecutive sessions (9.5 ± 2.7, 24.9 ± 2.5, 27.29 ± 1.5) (P < 0.001) for Mini-CEX and (6.1 ± 1.8 versus 9.0 ± 1.2) (P < 0.001) for DOPS. About 93% of the postgraduates recommended the application of WBA for their peers, and 86% of them requested to perform it again for other different clinical cases and procedures. Conclusion Workplace-based assessment in the form of Mini-CEX and DOPS revealed its ability to improve clinical knowledge and skills among family medicine postgraduates who became motivated to undergo it again in search of improving their clinical performance and reducing their stresses related to final summative and objective structured clinical examinations (OSCEs).

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

Mediating role of career self-efficacy between clinical learning environment and professional identity in nursing students

AIMS

To explore whether the clinical learning environment (CLE) has an indirect effect on professional identity through the mediation of career self-efficacy (CSE) in nursing students.

BACKGROUND

The shortage of nurses has become a universal problem worldwide. Improving nurses' professional identity is considered an effective way to reduce the turnover rate of nurses. However, little is known about the relationship between the CLE, CSE and professional identity.

DESIGN

An observational, questionnaire-based, cross-sectional study.

METHODS

A web-based survey was completed by 212 undergraduate nursing students from June to August 2018. Measures included Chinese translations of the CLE, the Career Self-Efficacy Scale, and the Professional Identity Scale.

RESULTS

Both the CLE (r = 0.552, p < 0.01) and CSE (r = 0.868, p < 0.01) correlated positively with professional identity. The indirect effect of the CLE on professional identity through CSE was positive (β = 0.342, p < 0.05) and the effect was 77.2%.

CONCLUSIONS

A better CLE and higher scores in CSE were associated with professional identity in nursing students, and a better CLE had an indirect effect on the professional identity of students through CSE.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

Pediatric Gastroenterology, Hepatology, and Nutrition Entrustable Professional Activities: Development of an Assessment Tool and Curricular Resources

BACKGROUND

Entrustable professional activities (EPAs) are critical activities performed by medical professionals, which can be observed and assessed. Adding on to common EPAs for all pediatric subspecialty trainees, specialty-specific EPAs for pediatric gastroenterology, hepatology, and nutritional fellowship were developed by the North American Society for Pediatric Gastroenterology, Hepatology, and Nutrition (NASPGHAN) EPA Task Force.

METHODS

Having developed specialty-specific EPAs, building EPA assessments is the next logical step, as EPAs are included under a larger umbrella of competency-based assessment. Thus, the NASPGHAN EPA Task Force and Training Committee collaborated on an assessment tool and associated curricular resources to aid in tracking trainees' progression to entrustment within individual EPAs and readiness for independent practice.

RESULTS

This manuscript reports the development of an EPA assessment tool, including guiding principles and the theory behind the assessment tool, with a focus on simple, meaningful assessments that can provide crucial performance feedback to trainees. In addition, curricular resources were developed, based on the assessment tool, to support training. Ultimately, it is the hope of the NASPGHAN EPA Task Force and Training Committee that this tool can aid training programs in providing formative feedback for trainees, and can be used by training programs and clinical competency committees for summative evaluation.