Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people

Ensuring robust OSCE assessments: A reflective account from a Scottish school of nursing

AIM

This paper reflects on the experience of one Scottish University in conducting a face-to-face Objective Structured Examination (OSCE) for large cohorts of student nurses. It outlines the challenges experienced and learning gained. Borton's model of reflection frames this work due to its simplicity, ease of application and cyclical nature.

BACKGROUND

The theoretical framework for the OSCE is critical thinking, enabling students to apply those skills authentically. OSCE's are designed to transfer classroom knowledge to clinical practice and offer an authentic work-based assessment.

DESIGN

Validity and robustness are key considerations in any assessment and in OSCE, the number of stations that students encounter is important and debated. We used a case-study based OSCE approach initially over four stations and following reflection, changed to one long station with four phases.

RESULTS

In OSCE examinations, interrater reliability is a necessity, and students expect equity of approach. We identified that despite clear marking criteria, marks were polarised, with students achieving high or low marks with little middle ground. Review of examination papers highlighted that although students' overall performance was good, some had failed in at least one station, suggesting a four-station approach may skew results. On reflection we hypothesised that using a one station case study-based, phased approach enabled the examiner to build up a more holistic picture of student knowledge and skills. It also provided the student opportunity to develop a rapport with the examiner and standardised patient, thereby putting them more at ease. We argue that this approach is holistic, authentic and student centred.

CONCLUSIONS

Our experience highlights that a single station, four phase OSCE is preferrable, enabling students to integrate all aspects of the assessment and provides a holistic view of clinical skills and knowledge.

Conceptualizing and Developing Competence in Newborn Medicine Among Military Pediatricians

INTRODUCTION

Competence in neonatal care is especially important for military pediatricians because military pediatricians can be asked to serve in remote duty locations with limited resources. We sought to understand how this competence is defined, developed, and assessed by military pediatric training programs.

MATERIALS AND METHODS

After Institutional Review Board approval was obtained, we interviewed educators and recent graduates from every pediatric military training program to construct a shared definition of competence. We then used Kern's Six Steps for curriculum development to understand how competence is taught and assessed.

RESULTS

Participants felt that competence for military pediatricians in the neonatal setting meant that learners should be able to provide a full spectrum of newborn care in any military setting. Participants confirmed that this competence was particularly important for military pediatricians because of the possibility of remote duty locations. Participants felt that specific knowledge, skills, and attitudes supported competence. Knowledge domains include distinguishing normal newborns from abnormal newborns, managing normal newborn care, managing common newborn abnormalities, and creating a safe escalation plan for complicated or uncommon newborn abnormalities. Specific skills that support competence are newborn resuscitation, delivery of effective ventilation, and neonatal circumcision. Specific attitudes that support competence are, understanding the personal limits of knowledge and understanding the resources for escalation of care. Educators use a variety of modalities to teach toward competence, including the structured curricula, bedside teaching, and simulation. According to participants, the assessment of learners occurs primarily through narrative assessment and feedback but would ideally occur through direct observation.

CONCLUSIONS

Competence in the neonatal setting is particularly important for military pediatricians. Essential skills undergo differential assessment and current assessment methods differ from ideal assessment methods. Future work should focus on how these facets can support a unified curriculum in newborn medicine.

Trainee anaesthetist self-assessment using an entrustment scale in workplace-based assessment

The role of self-assessment in workplace-based assessment remains contested. However, anaesthesia trainees need to learn to judge the quality of their own work. Entrustment scales have facilitated a shared understanding of performance standards among supervisors by aligning assessment ratings with everyday clinical supervisory decisions. We hypothesised that if the entrustment scale similarly helped trainees in their self-assessment, there would be substantial agreement between supervisor and trainee ratings. We collected separate mini-clinical evaluation exercises forms from 113 anaesthesia trainee-supervisor pairs from three hospitals in Australia and New Zealand. We calculated the agreement between trainee and supervisor ratings using Pearson and intraclass correlation coefficients. We also tested for associations with demographic variables and examined narrative comments for factors influencing rating. We found ratings agreed in 32% of cases, with 66% of trainee ratings within one point of the supervisor rating on a nine-point scale. The correlation between trainee and supervisor ratings was 0.71, and the degree of agreement measured by the intraclass correlation coefficient was 0.67. With higher supervisor ratings, trainee ratings better correlated with supervisor ratings. We found no strong association with demographic variables. Possible explanations of divergent ratings included one party being unaware of a vital aspect of the performance and different interpretations of the prospective nature of the scale. The substantial concordance between trainee and supervisor ratings supports the contention that the entrustment scale helped produce a shared understanding of the desired performance standard. Discussion between trainees and supervisors on the reasoning underlying their respective judgements would provide further opportunities to enhance this shared understanding.

Implicit versus explicit first impressions in performance-based assessment: will raters overcome their first impressions when learner performance changes?

First impressions can influence rater-based judgments but their contribution to rater bias is unclear. Research suggests raters can overcome first impressions in experimental exam contexts with explicit first impressions, but these findings may not generalize to a workplace context with implicit first impressions. The study had two aims. First, to assess if first impressions affect raters' judgments when workplace performance changes. Second, whether explicitly stating these impressions affects subsequent ratings compared to implicitly-formed first impressions. Physician raters viewed six videos where learner performance either changed (Strong to Weak or Weak to Strong) or remained consistent. Raters were assigned two groups. Group one (n = 23, Explicit) made a first impression global rating (FIGR), then scored learners using the Mini-CEX. Group two (n = 22, Implicit) scored learners at the end of the video solely with the Mini-CEX. For the Explicit group, in the Strong to Weak condition, the FIGR (M = 5.94) was higher than the Mini-CEX Global rating (GR) (M = 3.02, p < .001). In the Weak to Strong condition, the FIGR (M = 2.44) was lower than the Mini-CEX GR (M = 3.96 p < .001). There was no difference between the FIGR and the Mini-CEX GR in the consistent condition (M = 6.61, M = 6.65 respectively, p = .84). There were no statistically significant differences in any of the conditions when comparing both groups' Mini-CEX GR. Therefore, raters adjusted their judgments based on the learners' performances. Furthermore, raters who made their first impressions explicit showed similar rater bias to raters who followed a more naturalistic process.

A tool for assessing interprofessional collaborative practice: evolution of the Jefferson Teamwork Observation Guide (JTOG)®

The Jefferson Teamwork Observation Guide® (JTOG®), a multi-source tool to formatively assess interprofessional collaborative practice competencies, evolved from a need to teach learners the characteristics of high functioning teams. Over time, researchers developed and refined four versions of the tool - Team, Individual, Patient and Support Person - eliciting feedback from learners, providers, patients and family members to create 360-degree evaluations of team performance. Development of all four versions of the JTOG into a native mobile application facilitated workplace based assessment, enhancing the ability to collect real-time data, gather responses from a variety of stakeholders and provide timely feedback to practice teams and individuals. Several studies have found the JTOG to be feasible to administer in both educational and practice settings across all user groups. Data also suggest the tool is a valid measure of team performance and individual performance on teams. However, scores that trended high indicate the need for continued psychometric work and high demand for the tool highlights the importance of a more robust database and dynamic analytic support. Three main conclusions have emerged from our experience with the JTOG: there is a clear need for a competency-based assessment tool to assist educators and clinicians in improving team functioning; incorporating technology into assessment may help bridge the persistent gap between educational innovation in IPE and organizational change in practice; and an assessment strategy incorporating the voices of the patient and family is highly valued by key stakeholders and has the potential to influence patient experience.

Assessment-Seeking Strategies: Navigating the Decision to Initiate Workplace-Based Assessment

Phenomenon: Competency-based medical education (CBME) relies on workplace-based assessment (WBA) to generate formative feedback (assessment for learning-AfL) and make inferences about competence (assessment of learning-AoL). When approaches to CBME rely on residents to initiate WBA, learners experience tension between seeking WBA for learning and for establishing competence. How learners resolve this tension may lead to unintended consequences for both AfL and AoL. We sought to explore the factors that impact both decisions to seek and not to seek WBA and use the findings to build a model of assessment-seeking strategy used by residents. In building this model we consider how the link between WBA and promotion or progression within a program impacts an individual's assessment-seeking strategy. Approach: We conducted 20 semi-structured interviews with internal medicine residents at Queen's University about the factors that influence their decision to seek or avoid WBA. Using grounded theory methodology, we applied a constant comparative analysis to collect data iteratively and identify themes. A conceptual model was developed to describe the interaction of factors impacting the decision to seek and initiate WBA. Findings: Participants identified two main motivations when deciding to seek assessments: the need to fulfill program requirements and the desire to receive feedback for learning. Analysis suggested that these motivations are often at odds with each other. Participants also described several moderating factors that impact the decision to initiate assessments, irrespective of the primary underlying motivation. These included resident performance, assessor factors, training program expectations, and clinical context. A conceptual framework was developed to describe the factors that lead to strategic assessment-seeking behaviors. Insights: Faced with the dual purpose of WBA in CBME, resident behavior in initiating assessment is guided by specific assessment-seeking strategies. Strategies reflect individual underlying motivations, influenced by four moderating factors. These findings have broad implications for programmatic assessment in a CBME context including validity considerations for assessment data used in summative decision-making including readiness for unsupervised practice.

Emergency residents' self-perceived readiness for practice: the association of milestones, entrustable professional activities, and professional identities-a multi-institutional survey

Background

As a successful innovation, competency-based medical education and its assessment tools continue to be a key strategy in training future doctors and tracking their performance trajectories. Linked to professional identity, evidence suggests that clinical competence is related to thinking, acting and feeling like a physician. Thus, incorporating the values and attitudes of healthcare professions as part of their professional identity in the clinical workplace improves professional performance.

Methods

Through a cross-sectional study, we examined the association of milestone, entrustable professional activities (EPA) and professional identity using self-reported tools among emergency medicine residents from 12 teaching hospitals across Taiwan. Milestone, EPA and professional identity were assessed using the Emergency Medicine Milestone Scale, Entrustable Professional Activity Scale and Emergency Physician Professional Identity and Value Scale, respectively.

Results

The results of a Pearson correlation indicated a significant positive correlation between milestone-based core competencies and EPAs (r = 0.40 ~ 0.74, p < 0.01). The professional identity domain of skills acquisition, capabilities and practical wisdom was positively correlated with milestone-based core competencies of patient care, medical knowledge, practice-based learning and improvement, and system-based practice (r = 0.18 ~ 0.21, p ≤ 0.05), and six items of EPA (r = 0.16 ~ 0.22, p < 0.05). Additionally, the professional identity domain of professional recognition and self-esteem was positively correlated with practice-based learning and improvement, and system-based practice milestone competencies (r = 0.16 ~ 0.19, p < 0.05).

Conclusion

This study demonstrates milestone and EPA assessment tools are highly linked and therefore, can be synergistically used by supervisors and clinical educators to evaluate clinical performance during residency training. Emergency physicians' professional identity is partly influenced by the advancement of skills and a resident's ability to learn, effectively perform tasks and make appropriate medical decisions at the system level in their clinical practice. Further research is warranted to understand the importance of residents' competency in relation to their professional identity development trajectory during clinical training.

'Ticked off'? Can a new outcomes-based postgraduate curriculum utilising programmatic assessment reduce assessment burden in Intensive Care Medicine?

Context

Increasing dissatisfaction with existing methods of assessment in the workplace alongside a national drive towards outcomes-based postgraduate curricula led to a recent overhaul of the way Intensive Care Medicine (ICM) trainees are assessed in the United Kingdom. Programmatic assessment methodology was utilised; the existing 'tick-box' approach using workplace-based assessment to demonstrate competencies was de-emphasised and the expertise of trainers used to assess capability relating to fewer, high-level outcomes related to distinct areas of specialist practice.

Methods

A thematic analysis was undertaken investigating attitudes from 125 key stakeholders, including trainees and trainers, towards the new assessment strategy in relation to impact on assessment burden and acceptability.

Results

This qualitative study suggests increased satisfaction with the transition to an outcomes-based model with capability judged by educational supervisors. However, reflecting frustration relating to current assessment in the workplace, participants felt assessment burden has been significantly reduced. The approach taken was felt to be an improved method for assessing professional practice; there was enthusiasm for this change. However, this research highlights trainee and trainer anxiety regarding how to 'pass' these expert judgement decisions of capability in the real world. Additionally, concerns relating to the impact on subgroups of trainees due to the potential influence of implicit biases on the resultant fewer but 'higher stakes' interrogative judgements became apparent.

Conclusion

The move further towards a constructivist paradigm in workplace assessment in ICM reduces assessment burden yet can provoke anxiety amongst trainees and trainers requiring considered implementation. Furthermore, the perception of potential for bias in global judgements of performance requires further exploration.

Four ways to get a grip on making robust decisions from workplace-based assessments

Synthesising the results of workplace-based assessments to inform robust decisions is seen as both important and difficult. Concerns about failing to fail the trainee not ready to proceed has drawn disproportionate attention to assessors. This paper proposes a model for a more systems-based view so that the value of the assessor's judgement is incorporated while preserving the value and robustness of collective decision-making. Our experience has shown it can facilitate robust decisions on some of the more difficult areas, such as professionalism.

Pre-clerkship EPA assessments: a thematic analysis of rater cognition

BACKGROUND

Entrustable Professional Activities (EPAs) assessments measure learners' competence with an entrustment or supervisory scale. Designed for workplace-based assessment EPA assessments have also been proposed for undergraduate medical education (UME), where assessments frequently occur outside the workplace and may be less intuitive, raising validity concerns. This study explored how assessors make entrustment determinations in UME, with additional specific comparison based on familiarity with prior performance in the context of longitudinal student-assessor relationships.

METHODS

A qualitative approach using think-alouds was employed. Assessors assessed two students (familiar and unfamiliar) completing a history and physical examination using a supervisory scale and then thought-aloud after each assessment. We conducted a thematic analysis of assessors' response processes and compared them based on their familiarity with a student.

RESULTS

Four themes and fifteen subthemes were identified. The most prevalent theme related to "student performance." The other three themes included "frame of reference," "assessor uncertainty," and "the patient." "Previous student performance" and "affective reactions" were subthemes more likely to inform scoring when faculty were familiar with a student, while unfamiliar faculty were more likely to reference "self" and "lack confidence in their ability to assess."

CONCLUSIONS

Student performance appears to be assessors' main consideration for all students, providing some validity evidence for the response process in EPA assessments. Several problematic themes could be addressed with faculty development while others appear to be inherent to entrustment and may be more challenging to mitigate. Differences based on assessor familiarity with student merits further research on how trust develops over time.

Multi-source feedback following simulated resuscitation scenarios: a qualitative study

BACKGROUND

The direct observation and assessment of learners' resuscitation skills by an attending physician is challenging due to the unpredictable and time-sensitive nature of these events. Multisource feedback (MSF) may address this challenge and improve the quality of assessments provided to learners. We aimed to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a simulation-based resuscitation curriculum.

METHODS

We conducted a qualitative content analysis of narrative MSF of medical residents in their first postgraduate year of training who were participating in a simulation-based resuscitation course at two Canadian institutions. Assessments included an entrustment score and narrative comments from attending physicians, registered nurses, and resident peers in addition to self-assessment. Narrative comments were transcribed and analyzed thematically using a constant comparative method.

RESULTS

All 87 residents (100%) participating in the 2017-2018 course provided consent. A total of 223 assessments were included in our analysis. Four themes emerged from the narrative data: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. Relative to other assessor groups, feedback from nurses focused on patient-centred care and communication while attending physicians focused on the medical expert theme. Peer feedback was the most positive. Self-assessments included comments within each of the four themes.

CONCLUSIONS

In the context of a simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale and may offer a more holistic assessment of resuscitation skills within a competency-based medical education (CBME) program of assessment.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

PURPOSE

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data.

METHOD

For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies.

RESULTS

Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale.

CONCLUSIONS

Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

Effect of resident and assessor gender on entrustment-based observational assessment in an internal medicine residency program

Background: Implicit gender bias leads to differences in assessment. Studies examining gender differences in resident milestone assessment data demonstrate variable results. The purpose of this study was to determine if observational entrustment scores differ by resident and assessor gender in a program of assessment based on discrete, observable skills.  Methods: We analyzed overall entrustment scores and entrustment scores by Accreditation Council for Graduate Medical Education (ACGME) core competency for 238 residents (49% female) from 396 assessors (38% female) in one internal medicine residency program from July 2012 to June 2019. We conducted analyses at 1-12 months, 1-36 months, 1-6 months, 7-12 months, and 31-36 months. We used linear mixed-effect models to assess the role of resident and assessor gender, with resident-specific and assessor-specific random effect to account for repeated measures.  Results: Statistically significant interactions existed between resident and assessor gender for overall entrustment at 1-12 months (p < 0.001), 1-36 months (p< 0.001), 1-6 months (p<0.001), 7-12 months (p=0.04), and 31-36 months (p<0.001). However, group differences were not statistically significant. In several instances an interaction was significant between resident and assessor gender by ACGME core competency, but there were no statistically significant group differences for all competencies at any time point. When applicable, subsequent analysis of main effect of resident or assessor gender independently of one another revealed no statistically significant differences.   Conclusions: No significant differences in entrustment scores were found based on resident or assessor gender in our large, robust entrustment-based program of assessment. Determining the reasons for our findings may help identify ways to mitigate gender bias in assessment.

Procedure-based assessment for laparoscopic cholecystectomy can replace global rating scales

INTRODUCTION

Global rating scales (GRSs) such as the Objective Structured Assessment of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Surgery (GOALS) are assessment methods for surgical procedures. The aim of this study was to establish construct validity of Procedure-Based Assessment (PBA) and to compare PBA with GRSs for laparoscopic cholecystectomy.

MATERIAL AND METHODS

OSATS and GOALS GRSs were compared with PBA in their ability to discriminate between levels of performance between trainees who can perform the procedure independently and those who cannot. Three groups were formed based on the number of procedures performed by the trainee: novice (1-10), intermediate (11-20) and experienced (>20). Differences between groups were assessed using the Kruskal-Wallis and Mann-Whitney U tests.

RESULTS

Increasing experience correlated significantly with higher GRSs and PBA scores (all p < .001). Scores of novice and intermediate groups overlapped substantially on the OSATS (p = .1) and GOALS (p = .1), while the PBA discriminated between these groups (p = .03). The median score in the experienced group was higher with less dispersion for PBA (97.2[85.3-100]) compared to OSATS (82.1[60.7-100]) and GOALS (80[60-100]).

CONCLUSION

For assessing skill level or the capability of performing a laparoscopic cholecystectomy independently, PBA has a higher discriminative ability compared to the GRSs.

The agreement of the endoscopic Modified Lund-Kennedy scoring in a clinical research group: An observational study

OBJECTIVES

The main objective was to prove the robustness of the modified Lund-Kennedy staging system and its use in the clinical research group. Secondary objectives were to evaluate the physicians' homogeneity, identify outliers with an unacceptable agreement and define factors for questionable agreement within the group of raters.

MATERIAL AND METHODS

Anonymized endoscopic photos of patients with chronic rhinosinusitis were assessed by independent raters from a clinical research group. The level of agreement between raters was calculated using intra-class correlation and weighted kappa coefficient. Clusters of similarity were identified using Inter-Item Correlation Matrix. The weighted kappa coefficient was calculated for the most homogeneous group and outliers. Age, sex, consultancy years, combined clinical and research work assessed by 5 senior peers were also statistically compared between raters.

RESULTS

Intraclass-correlation coefficients were 0.75 and 0.95 for respectively single and average measures. Single measures value for most homogenous raters was 0.97 (weighted kappa 0.88, (P<0.001). One outlier with less research work score had an unacceptable agreement for single measures coefficient values with the 2 most homogenous raters (respectively 0.59, weighted kappa 0.15, P=0.32 and 0.57, weighted kappa 0.197, P=0.32). Pooled groups were similar in age (P=0.3), sex (P=0.1) and consultancy years (P=0.2) but significantly differentiated in peer-assessed clinical and research work score (P<0.001).

CONCLUSION

Even with a perfect overall agreement, careful examination of correlation matrix revealed an obvious outlier with less than ideal performance. The method may be helpful when studies using endoscopic staging system are designed to involve researchers from different backgrounds. When exploring the most common factors, education and clinical experience play a paramount role.

Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace: Findings From One Institution

PURPOSE

Assessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships.

METHOD

During the 2018-2019 academic year, the Virginia Commonwealth University School of Medicine implemented a mobile-friendly, student-initiated workplace-based assessment (WBA) system to provide formative feedback for the Core EPAs across all clinical clerkships. Students were required to request a specified number of Core EPA assessments in each clerkship. A modified O-SCORE scale (1 = "I had to do" to 4 = "I needed to be in room just in case") was used to rate learner performance. Generalizability theory was applied to assess the generalizability (or reliability) of the assessments. Decision studies were then conducted to determine the number of assessments needed to achieve a reasonable reliability.

RESULTS

A total of 10,680 WBAs were completed on 220 medical students. The majority of ratings were completed on EPA 1 (history and physical) (n = 3,129; 29%) and EPA 6 (oral presentation) (n = 2,830; 26%). Mean scores were similar (3.5-3.6 out of 4) across EPAs. Variance due to the student ranged from 3.5% to 8%, with the majority of the variation due to the rater (29.6%-50.3%) and other unexplained factors. A range of 25 to 63 assessments were required to achieve reasonable reliability (Phi > 0.70).

CONCLUSIONS

The O-SCORE demonstrated modest reliability when used across clerkships. These findings highlight specific challenges for implementing WBAs for the Core EPAs including the process for requesting WBAs, rater training, and application of the O-SCORE scale in medical student assessment.

Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane

PROBLEM

Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system.

APPROACH

To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time.

OUTCOMES

The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps.

NEXT STEPS

Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

Teaching and assessing intra-operative consultations in competency-based medical education: development of a workplace-based assessment instrument

Competency-based medical education (CBME) is being implemented worldwide. In CMBE, residency training is designed around competencies required for unsupervised practice and use entrustable professional activities (EPAs) as workplace “units of assessment”. Well-designed workplace-based assessment (WBA) tools are required to document competence of trainees in authentic clinical environments. In this study, we developed a WBA instrument to assess residents’ performance of intra-operative pathology consultations and conducted a validity investigation. The entrustment-aligned pathology assessment instrument for intra-operative consultations (EPA-IC) was developed through a national iterative consultation and used clinical supervisors to assess residents’ performance at an anatomical pathology program. Psychometric analyses and focus groups were conducted to explore the sources of evidence using modern validity theory: content, response process, internal structure, relations to other variables, and consequences of assessment. The content was considered appropriate, the assessment was feasible and acceptable by residents and supervisors, and it had a positive educational impact by improving performance of intra-operative consultations and feedback to learners. The results had low reliability, which seemed to be related to assessment biases, and supervisors were reluctant to fully entrust trainees due to cultural issues. With CBME implementation, new workplace-based assessment tools are needed in pathology. In this study, we showcased the development of the first instrument for assessing resident’s performance of a prototypical entrustable professional activity in pathology using modern education principles and validity theory.

Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework

Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system

BACKGROUND

Physician professionalism, including anaesthesiologists and intensive care doctors, should be continuously assessed during training and subsequent clinical practice. Multi-source feedback (MSF) is an assessment system in which healthcare professionals are assessed on several constructs (e.g., communication, professionalism, etc.) by multiple people (medical colleagues, coworkers, patients, self) in their sphere of influence. MSF has gained widespread acceptance for both formative and summative assessment of professionalism for reflecting on how to improve clinical practice.

METHODS

Instrument development and psychometric analysis (feasibility, reliability, construct validity via exploratory factor analysis) for MSF questionnaires in a postgraduate specialty training in Anaesthesiology and intensive care in Italy. Sixty-four residents at the Università del Piemonte Orientale (Italy) Anesthesiology Residency Program. Main outcomes assessed were: development and psychometric testing of 4 questionnaires: self, medical colleague, coworker and patient assessment.

RESULTS

Overall 605 medical colleague questionnaires (mean of 9.3 ±1.9) and 543 coworker surveys (mean 8.4 ±1.4) were collected providing high mean ratings for all items (> 4.0 /5.0). The self-assessment item mean score ranged from 3.1 to 4.3. Patient questionnaires (n = 308) were returned from 31 residents (40%; mean 9.9 ± 6.2). Three items had high percentages of "unable to assess" (> 15%) in coworker questionnaires. Factor analyses resulted in a two-factor solution: clinical management with leadership and accountability accounting for at least 75% of the total variance for the medical colleague and coworker's survey with high internal consistency reliability (Cronbach's α > 0.9). Patient's questionnaires had a low return rate, a limited exploratory analysis was performed.

CONCLUSIONS

We provide a feasible and reliable Italian language MSF instrument with evidence of construct validity for the self, coworkers and medical colleague. Patient feedback was difficult to collect in our setting.

Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study

BACKGROUND

Entrustable professional activities (EPAs) in competency-based, undergraduate medical education (UME) have led to new formative workplace-based assessments (WBA) using entrustment-supervision scales in clerkships. We conducted an observational, prospective cohort study to explore the usefulness of a WBA designed to assess core EPAs in a psychiatry clerkship.

METHODS

We analyzed changes in self-entrustment ratings of students and the supervisors' ratings per EPA. Timing and frequencies of learner-initiated WBAs based on a prospective entrustment-supervision scale and resultant narrative feedback were analyzed quantitatively and qualitatively. Predictors for indirect supervision levels were explored via regression analysis, and narrative feedback was coded using thematic content analysis. Students evaluated the WBA after each clerkship rotation.

RESULTS

EPA 1 ("Take a patient's history"), EPA 2 ("Assess physical & mental status") and EPA 8 ("Document & present a clinical encounter") were most frequently used for learner-initiated WBAs throughout the clerkship rotations in a sample of 83 students. Clinical residents signed off on the majority of the WBAs (71%). EPAs 1, 2, and 8 showed the largest increases in self-entrustment and received most of the indirect supervision level ratings. We found a moderate, positive correlation between self-entrusted supervision levels at the end of the clerkship and the number of documented entrustment-supervision ratings per EPA (p < 0.0001). The number of entrustment ratings explained 6.5% of the variance in the supervisors' ratings for EPA 1. Narrative feedback was documented for 79% (n = 214) of the WBAs. Most narratives addressed the Medical Expert role (77%, n = 208) and used reinforcement (59%, n = 161) as a feedback strategy. Students perceived the feedback as beneficial.

CONCLUSIONS

Using formative WBAs with an entrustment-supervision scale and prompts for written feedback facilitated targeted, high-quality feedback and effectively supported students' development toward self-entrusted, indirect supervision levels.

Can case-based discussions in a group setting be used to assess residents' clinical skills?

OBJECTIVES

The purpose of this study was to explore residents' and assessors' perception of a new group assessment concept.

METHODS

This qualitative study consists of observations of four group assessment sessions, followed by semi-structured interviews with six residents and four assessors (specialists in internal medicine), who all volunteered to be interviewed. All residents at a medical department (eleven to fifteen each time) and four assessors participated in four group assessments, where the residents' clinical skills were assessed through case-based discussions. An external consultant (an anthropologist) performed the observations and the interviews. Notes from the observations and the interviews were analyzed using an inductive approach.

RESULTS

Eight of the ten interviewed participants preferred group assessment to individual assessment. Results from the interviews suggested that the group assessments were more consistent and that the level of discussion was perceived to be higher in the group discussions compared to the one-to-one discussions. All residents indicated that they had acquired new knowledge during their assessment and reported having learned from listening to the assessment of their peers. Assessors similarly reported gaining new knowledge.

CONCLUSIONS

The residents and assessors expressed very favourable attitudes toward the new group assessment concept. The assessment process was perceived to be higher in quality and more consistent, contributing to learning for all participating doctors in the department. Group assessment is feasible and acceptable, and provides a promising tool for assessment of clinical skills in the future.

When Feedback Backfires: Influences of Negative Discrepancies Between Physicians' Self and Assessors' Scores on Their Subsequent Multisource Feedback Ratings

INTRODUCTION

With multisource feedback (MSF) physicians might overrate their own performance compared with scores received from assessors. However, there is limited insight into how perceived divergent feedback affects physicians' subsequent performance scores.

METHODS

During 2012 to 2018, 103 physicians were evaluated twice by 684 peers, 242 residents, 999 coworkers, and themselves in three MSF performance domains. Mixed-effect models quantified associations between the outcome variable "score changes" between first and second MSF evaluations, and the explanatory variable "negative discrepancy score" (number of items that physicians rated themselves higher compared with their assessors' scores) at the first MSF evaluation. Whether associations differed across assessor groups and across a physician's years of experience as a doctor was analyzed too.

RESULTS

Forty-nine percent of physicians improved their total MSF score at the second evaluation, as assessed by others. Number of negative discrepancies was negatively associated with score changes in domains "organization and (self)management" (b = -0.02; 95% confidence interval [CI], -0.03 to -0.02; SE = 0.004) and "patient-centeredness" (b = -0.03; 95% CI, -0.03 to -0.02; SE = 0.004). For "professional attitude," only negative associations between score changes and negative discrepancies existed for physicians with more than 6-year experience (b6-10yearsofexperience = -0.03; 95% CI, -0.05 to -0.003; SE = 0.01; b16-20yearsofexperience = -0.03; 95% CI, -0.06 to -0.004; SE = 0.01).

DISCUSSION

The extent of performance improvement was less for physicians confronted with negative discrepancies. Performance scores actually declined when physicians overrated themselves on more than half of the feedback items. PA score changes of more experienced physicians confronted with negative discrepancies and were affected more adversely. These physicians might have discounted feedback due to having more confidence in own performance. Future work should investigate how MSF could improve physicians' performance taking into account physicians' confidence.

Examiners' decision-making processes in observation-based clinical examinations

BACKGROUND

Objective structured clinical examinations (OSCEs) are commonly used to assess the clinical skills of health professional students. Examiner judgement is one acknowledged source of variation in candidate marks. This paper reports an exploration of examiner decision making to better characterise the cognitive processes and workload associated with making judgements of clinical performance in exit-level OSCEs.

METHODS

Fifty-five examiners for exit-level OSCEs at five Australian medical schools completed a NASA Task Load Index (TLX) measure of cognitive load and participated in focus group interviews immediately after the OSCE session. Discussions focused on how decisions were made for borderline and clear pass candidates. Interviews were transcribed, coded and thematically analysed. NASA TLX results were quantitatively analysed.

RESULTS

Examiners self-reported higher cognitive workload levels when assessing a borderline candidate in comparison with a clear pass candidate. Further analysis revealed five major themes considered by examiners when marking candidate performance in an OSCE: (a) use of marking criteria as a source of reassurance; (b) difficulty adhering to the marking sheet under certain conditions; (c) demeanour of candidates; (d) patient safety, and (e) calibration using a mental construct of the 'mythical [prototypical] intern'. Examiners demonstrated particularly higher mental demand when assessing borderline compared to clear pass candidates.

CONCLUSIONS

Examiners demonstrate that judging candidate performance is a complex, cognitively difficult task, particularly when performance is of borderline or lower standard. At programme exit level, examiners intuitively want to rate candidates against a construct of a prototypical graduate when marking criteria appear not to describe both what and how a passing candidate should demonstrate when completing clinical tasks. This construct should be shared, agreed upon and aligned with marking criteria to best guide examiner training and calibration. Achieving this integration may improve the accuracy and consistency of examiner judgements and reduce cognitive workload.

Performance assessment: Consensus statement and recommendations from the 2020 Ottawa Conference

INTRODUCTION

In 2011 the Consensus Statement on Performance Assessment was published in Medical Teacher. That paper was commissioned by AMEE (Association for Medical Education in Europe) as part of the series of Consensus Statements following the 2010 Ottawa Conference. In 2019, it was recommended that a working group be reconvened to review and consider developments in performance assessment since the 2011 publication.

METHODS

Following review of the original recommendations in the 2011 paper and shifts in the field across the past 10 years, the group identified areas of consensus and yet to be resolved issues for performance assessment.

RESULTS AND DISCUSSION

This paper addresses developments in performance assessment since 2011, reiterates relevant aspects of the 2011 paper, and summarises contemporary best practice recommendations for OSCEs and WBAs, fit-for-purpose methods for performance assessment in the health professions.

Assessing leadership in junior resident physicians: using a new multisource feedback tool to measure Learning by Evaluation from All-inclusive 360 Degree Engagement of Residents (LEADER)

Background

The multifaceted nature of leadership as a construct has implications for measuring leadership as a competency in junior residents in healthcare settings. In Canada, the Royal College of Physicians and Surgeons of Canada’s CanMEDS physician competency framework includes the Leader role calling for resident physicians to demonstrate collaborative leadership and management within the healthcare system. The purpose of this study was to explore the construct of leadership in junior resident physicians using a new multisource feedback tool.

Methods

To develop and test the Learning by Evaluation from All-Inclusive 360 Degree Engagement of Residents (LEADER) Questionnaire, we used both qualitative and quantitative research methods in a multiphase study. Multiple assessors including peer residents, attending physicians, nurses, patients/family members and allied healthcare providers as well as residents’ own self-assessments were gathered in healthcare settings across three residency programmes: internal medicine, general surgery and paediatrics. Data from the LEADER were analysed then triangulated using a convergent-parallel mixed-methods study design.

Results

There were 230 assessments completed for 27 residents. Based on key concepts of the Leader role, two subscales emerged: (1) Personal leadership skills subscale (Cronbach’s alpha=0.81) and (2) Physicians as active participant-architects within the healthcare system (abbreviated to active participant-architects subscale, Cronbach’s alpha=0.78). There were seven main themes elicited from the qualitative data which were analogous to the five remaining intrinsic CanMEDS roles. The remaining two themes were related to (1) personal attributes unique to the junior resident and (2) skills related to management and administration.

Conclusions

For healthcare organisations that aspire to be proactive rather than reactive, we make three recommendations to develop leadership competence in junior physicians: (1) teach and assess leadership early in training, (2) empower patients to lead and transform training and care by evaluating doctors, (3) activate frontline care providers to be leaders by embracing patient and team feedback.

Introduction of mini-clinical evaluation exercise in teaching dental radiology-A pilot study

INTRODUCTION

Workplace-based assessments are methods that can be applied for assessing competence and performance. One of these methods is the mini-clinical evaluation exercise (mini-CEX). This study was conducted to determine the role of mini-CEX in students' performance assessment on panoramic X-ray reporting at dental radiology course.

MATERIALS AND METHODS

A workshop as training for the assessors and the participants was conducted before the primary test. All participants (n = 36) were randomly allocated into six groups. Each group had three seminars in which every student reported a panoramic X-ray. Students were directly observed and rated by an assessor on a modified mini-CEX rating form. Then, a self-assessment of the students and a systematic feedback session were performed. Finally, the students and the assessors were evaluated for the acceptability and satisfaction with this tool.

RESULTS

The mean duration of the assessment and the feedback decreased significantly from the first seminar to the third seminar (P < .0001). Comparison of the results of the mini-CEX of all three assessments showed that students displayed a significantly better performance in evaluating the upper jaw and the soft tissue (P < .05). There was no significant improvement for the other aspects of the rating form. Overall, both students and assessors reported a high level of satisfaction in using the mini-CEX rating form.

CONCLUSION

Due to the objectivity and transparency of the assessment, the mini-CEX helped to improve the performance on reporting panoramic X-rays. Besides that, the structured feedback had major impact on the improvement. Overall, the assessors and the participants reported a high level of satisfaction using the rating form. Therefore, the mini-CEX may be an effective method for performing workplace-based assessments to evaluate students' performance on reporting panoramic X-rays.

Entrustment within an objective structured clinical examination (OSCE) progress test: Bridging the gap towards competency-based medical education

PURPOSE

Progress testing aligns well with competency-based medical education (CBME) frameworks, which stress the importance of continuous improvement. Entrustment is a useful assessment concept in CBME models. The purpose of this study was to explore the use of an entrustability rating scale within the context of an objective structured clinical examination (OSCE) Progress Test.

METHODS

A 9-case OSCE Progress Test was administered to Internal Medicine residents (PGYs 1-4). Residents were assessed using a checklist (CL), global rating scale (GRS), training level rating scale (TLRS), and entrustability scale (ENT). Reliability was calculated using Cronbach's alpha. Differences in performance by training year were explored using ANOVA and effect sizes were calculated using partial eta-squared. Examiners completed a post-examination survey.

RESULTS

Ninety one residents and forty two examiners participated in the OSCE. Inter-station reliability was high for all instruments. There was an overall effect of training level for all instruments (p < 0.001). Effect sizes were large. 88% of examiners completed the survey. Most (62%) indicated feeling comfortable in making entrustment decisions during the OSCE.

CONCLUSIONS

An entrustability scale can be used in an OSCE Progress Test to generate highly reliable ratings that discriminate between learners at different levels of training.

Workplace-based assessments in postgraduate medical education: A hermeneutic review

OBJECTIVES

Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education.

METHOD

This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the United Kingdom, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature in order to answer three scoping questions concerning acceptability, effectiveness and assessment training. As each round was coded, themes were refined and questions added until saturation was reached.

RESULTS

Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy, disagreement with definitions and frameworks, and inadequate summative use of WBAs. Effectiveness is influenced by user (eg, engagement and assessment literacy) and tool attributes (eg, definitions and scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessors' assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline.

CONCLUSIONS

The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learned regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.

The Ottawa Emergency Department Shift Observation Tool (O-EDShOT): A New Tool for Assessing Resident Competence in the Emergency Department

OBJECTIVES

The outcome of emergency medicine (EM) training is to produce physicians who can competently run an emergency department (ED) shift. However, there are few tools with supporting validity evidence specifically designed to assess multiple key competencies across an entire shift. The investigators developed and gathered validity evidence for a novel entrustment-based tool to assess a resident's ability to safely run an ED shift.

METHODS

Through a nominal group technique, local and national stakeholders identified dimensions of performance that are reflective of a competent ED physician and are required to safely manage an ED shift. These were included as items in the Ottawa Emergency Department Shift Observation Tool (O-EDShOT), and each item was scored using an entrustment-based rating scale. The tool was implemented in 2018 at the University of Ottawa Department of Emergency Medicine, and quantitative data and qualitative feedback were collected over 6 months.

RESULTS

A total of 1,141 forms were completed by 78 physicians for 45 residents. An analysis of variance demonstrated an effect of training level with statistically significant increases in mean O-EDShOT scores with each subsequent postgraduate year (p < 0.001). Scores did not vary by ED treatment area. Residents rated as able to safely run the shift had significantly higher mean ± SD scores (4.8 ± 0.3) than those rated as not able (3.8 ± 0.6; p < 0.001). Faculty and residents reported that the tool was feasible to use and facilitated actionable feedback aimed at progression toward independent practice.

CONCLUSIONS

The O-EDShOT successfully discriminated between trainees of different levels regardless of ED treatment area. Multiple sources of validity evidence support the O-EDShOT as a tool to assess a resident's ability to safely run an ED shift. It can serve as a stimulus for daily observation and feedback making it practical to use within an EM residency program.

Educational Impact Drives Feasibility of Implementing Daily Assessment in the Workplace

Construct: Authors investigated the perspectives of stakeholders on feasibility elements of workplace-based assessments (WBA) with varying designs. Background: In the transition to competency-based medical education, WBA are taking a more prominent role in assessment programs. However, the increased demand for WBA leads to new challenges for implementing suitable WBA tools with published validity evidence, while also being feasible and useful in practice. Despite the availability of published WBA tools, implementation does not necessarily occur; a more fulsome understanding of the perspectives of stakeholders who are ultimately the end-users of these tools, as well as the system factors that both deter or support their use, could help to explain why evidence-based assessment tools may not be incorporated into residency programs. Approach: We examined the perspectives of two groups of stakeholders, surgical teachers and resident learners, during an assessment intervention that varied the assessment tools while keeping the assessment process constant. We chose diverse exemplars from published assessment tools that each represented a different response format: global rating scales, step-by-step surgical rubrics, and an entrustability scale. The primary purpose was to investigate how stakeholders are impacted by WBA tools with varying response formats to better understand their feasibility for assessment of cataract surgery. Secondarily, we were able to explore the culture of assessment in cataract surgery education including stakeholders' perceptions of WBA unrelated to assessment form design. Semi-structured interviews with teachers and a focus group with the residents enabled discussion of their perspectives on dimensions of the tools such as acceptability, demand, implementation, practicality, adaptation, and integration. Findings: Three themes summarize teachers' and residents' experiences with the assessment tools: (1) Feedback is the priority; (2) Forms informing coaching; and (3) Forcing the conversation. The tools helped to facilitate the feedback conversation by serving as a reminder to initiate the conversation, a framework to structure the conversation, and a memory aid for providing detailed feedback. Surgical teachers preferred the assessment tool with a design that best aligned with their approach to teaching and how they wanted to provide feedback. Orientation to the tools, combined with established remediation pathways, may help preceptors to better use assessment tools and improve their ability to give critical feedback. Conclusions: Feedback, more so than assessment, dominated the comments provided by both teachers and residents after using the various WBA tools. Our typical assessment design efforts focus on the creation or selection of a robust assessment tool according to good design and measurement principles, but the current findings would encourage us to also prioritize the coaching relationship and include efforts to design WBA tools to function as a mediator to augment teaching, learning, and feedback exchange within that relationship in the workplace.

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail

Introduction

Interest is growing in specialty-specific assessments of student candidates based on clinical clerkship performance to assist in the selection process for postgraduate training. The most established and extensively used is the emergency medicine (EM) Standardized Letter of Evaluation (SLOE), serving as a substitute for the letter of recommendation. Typically developed by a program’s leadership, the group SLOE strives to provide a unified institutional perspective on performance. The group SLOE lacks guidelines to direct its development raising questions regarding the assessments, processes, and standardization programs employ. This study surveys EM programs to gather validity evidence regarding the inputs and processes involved in developing group SLOEs.

Methods

A structured telephone interview was administered to assess the input data and processes employed by United States EM programs when generating group SLOEs.

Results

With 156/178 (87.6%) of Accreditation Council of Graduate Medical Education-approved programs responding, 146 (93.6%) reported developing group SLOEs. Issues identified in development include the following: (1) 84.9% (124/146) of programs limit the consensus process by not employing rigorous methodology; (2) several stakeholder groups (nurses, patients) do not participate in candidate assessment placing final decisions at risk for construct under-representation; and (3) clinical shift assessments don’t reflect the task-specific expertise of each stakeholder group nor has the validity of each been assessed.

Conclusion

Success of the group SLOE in its role as a summative workplace-based assessment is dependent upon valid input data and appropriate processes. This study of current program practices provides specific recommendations that would strengthen the validity arguments for the group SLOE.

Linking Workplace-Based Assessment to ACGME Milestones: A Comparison of Mapping Strategies in Two Specialties

Construct: The construct that is assessed is competency in Pediatrics and Internal Medicine residency training. Background: The Accreditation Council for Graduate Medical Education (ACGME) created milestones to measure learner progression toward competence over time but not as direct assessment tools. Ideal measurement of resident performance includes direct observation and assessment of patient care skills in the workplace. Residency programs have linked these concepts by mapping workplace-based assessments to the milestones of ACGME subcompetencies. Mapping is a subjective process, and little is known about specific techniques or the resulting consequences of mapping program-specific assessment data to larger frameworks of competency. Approach: In this article, the authors compare and contrast the techniques used to link workplace-based assessments called Observable Practice Activities (OPAs) to ACGME subcompetencies in two large academic residency programs from different specialties (Internal Medicine and Pediatrics). Descriptive analysis explored the similarities and differences in the assessment data generated by mapping assessment items to larger frameworks of competency. Results: Each program assessed the core competencies with similar frequencies. The largest discrepancy between the two subspecialties was the assessment of Medical Knowledge, which Internal Medicine assessed twice as often. Pediatrics also assessed the core competency Systems-based Practice almost twice as often as Internal Medicine. Both programs had several subcompetencies that were assessed more or less often than what appeared to be emphasized by the blueprint of mapping. Despite using independent mapping processes, both programs mapped each OPA to approximately three subcompetencies. Conclusions: Mapping workplace-based assessments to the ACGME subcompetencies allowed each program to see the whole of their curricula in ways that were not possible before and to identify existing curricular and assessment gaps. Although each program used similar assessment tools, the assessment data generated were different. The lessons learned in this work could inform other programs attempting to link their own workplace-based assessment elements to ACGME subcompetencies.

Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017-18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.

Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can 'work'

Introduction: Borderline regression (BRM) is considered problematic in small cohort OSCEs (e.g. n < 50), with institutions often relying on item-centred standard setting approaches which can be resource intensive and lack defensibility in performance tests.Methods: Through an analysis of post-hoc station- and test-level metrics, we investigate the application of BRM in three different small-cohort OSCE contexts: the exam for international medical graduates wanting to practice in the UK, senior sequential undergraduate exams, and Physician associates exams in a large UK medical school.Results: We find that BRM provides robust metrics and concomitantly defensible cut scores in the majority of stations (percentage of problematic stations 5, 14, and 12%, respectively across our three contexts). Where problems occur, this is generally due to an insufficiently strong relationship between global grades and checklist scores to be confident in the standard set by BRM in these stations.Conclusion: This work challenges previous assumptions about the application of BRM in small test cohorts. Where there is sufficient spread of ability, BRM will generally provide defensible standards, assuming careful design of station-level scoring instruments. However, extant station cut-scores are preferred as a substitute where BRM standard setting problems do occur.

Development and inclusion of an entrustable professional activity (EPA) scale in a simulation-based medicine dispensing assessment

BACKGROUND AND PURPOSE

Effective, safe, and patient-centred dispensing is a core task of community pharmacists. Entrustable professional activities (EPAs) offer a way of defining and assessing these daily practice activities. Although EPAs have become popular within competency-based medical education programs, their use is new to pharmacy education and assessment.

EDUCATIONAL ACTIVITY AND SETTING

A simulation-based assessment framework containing a scale of entrustment was developed to evaluate the readiness of Year 4 undergraduate pharmacy students to safely manage the supply of prescribed medicine(s) in a community pharmacy. The assessment framework was piloted in a fourth year "Transition to Practice" course with 28 simulation-based assessments conducted.

FINDINGS

An entrustment framework was developed and implemented successfully with Year 4 undergraduate pharmacy students. The EPA for medicine dispensing integrates competency domains that include information gathering, providing patient-centred care, clinical reasoning, medicine dispensing, and professional communications. On a scale ranging from level 1 to level 5, the majority (73%) of entrustment ratings were level 2 or level 3; and of the students who achieved different ratings between clinical scenarios, 75% of students improved on their second simulation attempt. There was a strong correlation between the global EPA ratings with the total score achieved across the domains.

SUMMARY

Using simulation-based assessment, entrustment decision making can be incorporated in "entry to profession" undergraduate and postgraduate pharmacy courses to assess students' readiness to transition between learning and professional practice.

A general practice workplace-based assessment instrument: Content and construct validity

Introduction: Relatively few general practice (GP) workplace-based assessment instruments have been psychometrically evaluated. This study aims to establish the content validity and internal consistency of the General Practice Registrar Competency Assessment Grid (GPR-CAG).Methods: The GPR-CAG was constructed as a formative assessment instrument for Australian GP registrars (trainees). GPR-CAG items were determined by an iterative literature review, expert opinion and pilot-testing process. Validation data were collected, between 2014 and 2016, during routine clinical teaching visits within registrars' first two general practice training terms (GPT1 and GPT2) for registrars across New South Wales and the Australian Capital Territory. Factor analysis and expert consensus were used to refine items and establish GPR-CAG's internal structure. GPT1 and GPT2 competencies were analysed separately.Results: Data of 555 registrars undertaking GPT1 and 537 registrars undertaking GPT2 were included in analyses. A four-factor, 16-item solution was identified for GPT1 competencies (Cronbach's alpha range: 0.71-0.83) and a seven-factor 27-item solution for GPT2 competencies (Cronbach's alpha: 0.63-0.84). The emergent factor structures were clinically characterisable and resonant with existing medical education competency frameworks.Discussion: This study establishes initial evidence for the content validity and internal consistency of GPR-CAG. GPR-CAG appears to have utility as a formative GP training WBA instrument.

Entrustable professional activities (EPAs) for postgraduate competency based intensive care medicine training in the Netherlands: The next step towards excellence in intensive care medicine training

INTRODUCTION

The Competency Based Training in Intensive Care Education (CoBaTrICE) programme developed common standards of ICM training by describing competencies of an intensivist. Entrustable Professional Activities (EPAs) of Intensive Care Medicine (ICM) (EPAsICM) are presented as a new workplace-based assessment tool in competency-based training of intensivists. EPAs are activities to be entrusted to a trainee once he (or she) has attained competence. EPAs emphasise the role of trust between trainees and supervisors. EPAs bridge the gap between competencies and competence.

METHODS

An expert panel of ICM (vice)programme directors and intensivists in The Netherlands integrated the CoBaTrICE and CanMEDS competencies into EPAsICM. Comment and feedback was sought from other ICM programme directors and educational experts and processed in the final version of EPAsICM before implementation in the Dutch ICM training programme.

RESULTS

A list of 15 EPAsICM are considered to reflect the spectrum of clinical practice while incorporating the competencies of CoBaTrICE and CanMEDS. The grading system is designed as a 5-point entrustment scale based on the amount of supervision a trainee needs, aligning with daily judgement of trainees by intensivists.

CONCLUSION

EPAsICM is an assessment tool that formalises entrustment decisions and can be a valuable addition in international ICM training.

Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination

BACKGROUND

The concept of EPAs is increasingly applied to assess trainees' workplace performance by means of entrustment ratings. OSCEs assess performance in a simulated setting, and it is unclear whether entrustment ratings can be integrated into these exams. This study explores the introduction of an entrustment rating scale into an existing OSCE.

METHODS

A 6-point entrustment scale was added to the standard ratings in an OSCE administered prior to students' final clerkship year in an undergraduate medical programme. Standard OSCE ratings assess clinical and communication skills. Assessors (n = 54) rated students' performance (n = 227) on a diverse set of clinical tasks and evaluated the addition of entrustment scales to OSCEs. Descriptive and inferential statistics were calculated for analyses.

RESULTS

Student performance varied across the stations, as reflected in both the standard OSCE ratings and the added entrustment ratings. Students received generally high standard OSCE ratings, whereas entrustment ratings were more widely distributed. All students passed the OSCE, and only a small proportion of students did not reach the expected pass threshold of 60% on the standard ratings in the single stations. The proportion of students who did not reach the expected entrustment level in the respective stations was noticeably higher. Both the clinical and communication skill ratings were related to the entrustment rating in most OSCE stations. A majority of the assessors positively evaluated the addition of entrustment ratings into the OSCE.

DISCUSSION

The findings provide an empirical basis to broaden our understanding of the potential use of entrustment ratings in existing OSCEs. They provide directions for future, more specific studies. The ratings might be used for formative feedback on students' readiness for workplace practice.

Workplace-Based Assessments Using Pediatric Critical Care Entrustable Professional Activities

BACKGROUND

Workplace-based assessment (WBA) is critical to graduating competent physicians. Developing assessment tools that combine the needs of faculty, trainees, and governing bodies is challenging but imperative. Entrustable professional activities (EPAs) are emerging as a clinically oriented framework for trainee assessment.

OBJECTIVE

We sought to develop an EPA-based WBA tool for pediatric critical care medicine (PCCM) fellows. The goals of the tool were to promote learning through benchmarking and tracking entrustment.

METHODS

A single PCCM EPA was iteratively subdivided into observable practice activities (OPAs) based on national and local data. Using a mixed-methods approach following van der Vleuten's conceptual model for assessment tool utility and Messick's unified validity framework, we sought validity evidence for acceptability, content, internal structure, relation to other variables, response process, and consequences.

RESULTS

Evidence was gathered after 1 year of use. Items for assessment were based on correlation between the number of times each item was assessed and the frequency professional activity occurred. Phi-coefficient reliability was 0.65. Narrative comments demonstrated all factors influencing trust, identified by current literature, were cited when determining level of entrustment granted. Mean entrustment levels increased significantly between fellow training years (P = .001). Compliance for once- and twice-weekly tool completion was 50% and 100%, respectively. Average time spent completing the assessment was less than 5 minutes.

CONCLUSIONS

Using an EPA-OPA framework, we demonstrated utility and validity evidence supporting the tool's outcomes. In addition, narrative comments about entrustment decisions provide important insights for the training program to improve individual fellow advancement toward autonomy.

Comparison of the Standardized Video Interview and Interview Assessments of Professionalism and Interpersonal Communication Skills in Emergency Medicine

OBJECTIVES

The Association of American Medical Colleges Standardized Video Interview (SVI) was recently added as a component of emergency medicine (EM) residency applications to provide additional information about interpersonal communication skills (ICS) and knowledge of professionalism (PROF) behaviors. Our objective was to ascertain the correlation between the SVI and residency interviewer assessments of PROF and ICS. Secondary objectives included examination of 1) inter- and intrainstitutional assessments of ICS and PROF, 2) correlation of SVI scores with rank order list (ROL) positions, and 3) the potential influence of gender on interview day assessments.

METHODS

We conducted an observational study using prospectively collected data from seven EM residency programs during 2017 and 2018 using a standardized instrument. Correlations between interview day PROF/ICS scores and the SVI were tested. A one-way analysis of variance was used to analyze the association of SVI and ROL position. Gender differences were assessed with independent-groups t-tests.

RESULTS

A total of 1,264 interview-day encounters from 773 unique applicants resulted in 4,854 interviews conducted by 151 interviewers. Both PROF and ICS demonstrated a small positive correlation with the SVI score (r = 0.16 and r = 0.17, respectively). ROL position was associated with SVI score (p < 0.001), with mean SVI scores for top-, middle-, and bottom-third applicants being 20.9, 20.5, and 19.8, respectively. No group differences with gender were identified on assessments of PROF or ICS.

CONCLUSIONS

Interview assessments of PROF and ICS have a small, positive correlation with SVI scores. These residency selection tools may be measuring related, but not redundant, applicant characteristics. We did not identify gender differences in interview assessments.

Twelve tips for constructing a multiple mini-interview

Health professions the world over value various competencies in their practitioners that are not easily captured by academic measures of performance. As a result, many programs have begun using multiple mini-interviews (MMIs) to facilitate the selection of candidates who are most likely to demonstrate and further develop such qualities. In this twelve-tips article, the authors offer evidence- and experience-based advice regarding how to construct an MMI that is fit for purpose. The tips are provided chronologically, offering guidance regarding how one might conceptualize their goals for creating an MMI, how to establish a database of stations that are context appropriate, and how to prepare both candidates and examiners for their task. While MMIs have been shown to have utility in many instances, the authors urge caution against over-generalization by stressing the importance of post-MMI considerations including data monitoring and integration between one's admissions philosophy and one's curricular efforts.

Assessing veterinary students using in-training evaluation scores: what is being assessed?

In-training evaluations are commonly used for assessing veterinary students during clinical training, but are criticised for being unable to discriminate dimensions of performance. This study investigated scores on an in-training evaluation in use at one veterinary school to determine the dimensions being assessed and the influence of the dimensions on the overall score awarded. Common factor analysis and ordinal logistic regression were conducted on a retrospective sample of 3466 evaluations of 197 final year veterinary students. The findings suggested a higher-order dimensional structure, with one overarching factor and two to four subfactors, consistent with the complex construct of competency that thSAS Institute e assessment was intended to assess. In the four -factor model, all dimensions were significantly related to overall grade, with the effects of the professional attitude factor and the knowledge factor dependent on the placement. The professional attitude factor had the strongest effect on overall grade (β=2.71, P=0.0004). There was a significant effect of placement on overall grade (P=0.021). Neither academic status of the supervisor nor grade point average had significant effects on the overall grade (P>0.49), and a student's overall grade did not significantly differ over time (P=1). The results suggest that the complexity of supervisor judgement is effectively represented in evaluation scores.

Exploratory Analysis of Entrustable Professional Activities as a Performance Measure During Early Pharmacy Practice Experiences

Objective. To examine entrustable professional activities (EPAs) as an assessment tool for student pharmacists completing early practice experiences. Methods. Students completed a 2-month practice experience upon conclusion of their first year. Student performance on EPAs was assessed by preceptors and students at the midpoint and conclusion of the experience using a scale that ranged from dependent (1.0) to independent (5.0). Wilcoxon Signed-Rank Test assessed for differences between the midpoint and final evaluations on student self-evaluations and between the midpoint and final evaluation on preceptor-student evaluations. Cronbach's α assessed reliability of the EPAs. Results. From May to August 2016, 147 students completed a practice experience. Student-self and preceptor-student evaluations at the midpoint and final approximated a median score of 3.0 (IQR 2) and 4.0 (IQR 3), respectively, on EPAs 1-14. Analyses revealed statistically significant increases from midpoint to final evaluation for all constructs on both evaluations. Cronbach's α yielded scores of 0.98 for the preceptor evaluations and 0.95 for the student self-evaluation. Conclusion. There was an increase in student performance over time. The EPA statements may be a reliable assessment tool for student performance in pharmacy education.

Contextualizing Work-Based Assessments of Faculty and Residents: Is There a Relationship Between the Clinical Practice Environment and Assessments of Learners and Teachers?

PURPOSE

Competence is bound to context, yet seldom is environment explicitly considered in work-based assessments. This study explored faculty and residents' perspectives of the environment during internal medicine clinical teaching unit (CTU) rotations, the extent that each group accounts for environmental factors in assessments, and relationships between environmental factors and assessments.

METHOD

From July 2014 to June 2015, 212 residents and 54 faculty across 5 teaching hospitals at University of Toronto rated their CTU environment using a novel Practice Environment Rating Scale (PERS) matched by block and hospital. Faculty-PERS data were paired to In-Training Evaluation Reports (ITERs) of residents supervised during each block, and Resident-PERS data to Resident Assessment of Teaching Effectiveness (RATE) scores of the same faculty. Differences between perceptions and assessments were tested using repeated-measures MANOVAs, ANOVAs, and correlations.

RESULTS

One-hundred sixty-four residents completed the PERS; residents rated the CTU environment more positively than faculty (3.91/5 vs. 3.29, P < .001). Residents were less likely to report considering environmental factors when assessing faculty (2.70/5) compared with faculty assessing residents (3.40, P < .0001), d = 1.2. Whereas Faculty-PERS ratings did not correlate with ITER scores, Resident-PERS ratings had weak to moderate correlations with RATE scores (overall r = 0.27, P = .001).

CONCLUSIONS

Residents' perceptions of the environment had small but significant correlations with assessments of faculty. Faculty's perceptions did not affect assessments of residents, potentially because they reported accounting for environmental factors. Understanding the interplay between environment and assessment is essential to developing valid competency judgments.

A New Multisource Feedback Tool for Evaluating the Performance of Specialty-Specific Physician Groups: Validity of the Group Monitor Instrument

INTRODUCTION

Since clinical practice is a group-oriented process, it is crucial to evaluate performance on the group level. The Group Monitor (GM) is a multisource feedback tool that evaluates the performance of specialty-specific physician groups in hospital settings, as perceived by four different rater classes. In this study, we explored the validity of this tool.

METHODS

We explored three sources of validity evidence: (1) content, (2) response process, and (3) internal structure. Participants were 254 physicians, 407 staff, 621 peers, and 282 managers of 57 physician groups (in total 479 physicians) from 11 hospitals.

RESULTS

Content was supported by the fact that the items were based on a review of an existing instrument. Pilot rounds resulted in reformulation and reduction of items. Four subscales were identified for all rater classes: Medical practice, Organizational involvement, Professionalism, and Coordination. Physicians and staff had an extra subscale, Communication. However, the results of the generalizability analyses showed that variance in GM scores could mainly be explained by the specific hospital context and the physician group specialty. Optimization studies showed that for reliable GM scores, 3 to 15 evaluations were needed, depending on rater class, hospital context, and specialty.

DISCUSSION

The GM provides valid and reliable feedback on the performance of specialty-specific physician groups. When interpreting feedback, physician groups should be aware that rater classes' perceptions of their group performance are colored by the hospitals' professional culture and/or the specialty.

Pharmacy Preceptor Judgments of Student Performance and Behavior During Experiential Training

Objective. To report the findings of how Canadian preceptors perceive and subsequently evaluate diverse levels of trainees during pharmacy clerkships. Methods. Using modified Delphi technique, 17 Doctor of Pharmacy (PharmD) preceptors from across Canada categorized 16 student narrative descriptions pertaining to their perception of described student performance: exceeds, meets, or falls below their expectations. Results. Twelve (75%) student narratives profiles were categorized unanimously in the final round, six of which were below expectations. Out of 117 ratings of below expectations by responding preceptors, the majority (115, 98%) of post-baccalaureate PharmD students described would fail. Conversely, if the same narrative instead profiled a resident or an entry-to-practice PharmD student, rotation failure decreased to 95 (81%) and 89 (76%), respectively. Conclusion. Pharmacy preceptors do not uniformly judge the same described student performance and inconsistently apply failing rotation grades when they do agree that performance falls below expectations.