Evolving concepts of assessment in a competency-based world

Elucidating system-level interdependence in electronic health record data: What are the ramifications for trainee assessment?

CONTEXT

The electronic health record (EHR) has been identified as a potential site for gathering data about trainees' clinical performance, but these data are not collected or organised for this purpose. Therefore, a careful and rigorous approach is required to explore how EHR data could be meaningfully used for assessment purposes. The purpose of this study was to identify EHR performance metrics that represent both the independent and interdependent clinical performance of emergency medicine (EM) trainees and explore how they might be meaningfully used for assessment and feedback.

METHODS

Using constructivist grounded theory, we conducted 21 semi-structured interviews with EM faculty members and residents. Participants were asked to identify the clinical actions of trainees that would be valuable for assessment and feedback and describe how those activities are represented in the EHR. Data collection and analysis, which consisted of three stages of coding, occurred iteratively.

RESULTS

When faculty members and trainees in EM were asked to reflect on the usefulness of using EHR performance metrics for resident assessment and feedback they expressed both widespread support for the idea in principle and hesitation that aspects of clinical performance captured in the data would not be representative of residents' individual performance, but would rather reflect their interdependence with other team members and the systems in which they work. We highlight three categorisations of system-level interdependence - medical directives, technological systems and organisational systems - identified by our participants, and discuss strategies participants employed to navigate these forms of interdependence within the health care system.

CONCLUSIONS

System-level interdependence shapes physicians' performances, and yet, this impact is rarely corrected for or noted within clinical performance data. Educators have a responsibility to recognise system-level interdependence when teaching and consider system-level interdependence when assessing the performance of trainees in order to most effectively and fairly utilise the EHR as a source of assessment data.

A case for feedback and monitoring assessment in competency-based medical education

PURPOSE

Within competency-based medical education, self-regulated learning (SRL) requires residents to leverage self-assessment and faculty feedback. We sought to investigate the potential for competency-based assessments to foster SRL by quantifying the relationship between faculty feedback and entrustment ratings as well as the congruence between faculty assessment and resident self-assessment.

MATERIALS AND METHODS

We collected comments in (a) an emergency medicine objective structured clinical examination group (objective structured clinical examinations [OSCE] and emergency medicine OSCE group [EMOG]) and (b) a first-year resident multidisciplinary resuscitation "Nightmares" course assessment group (NCAG) and OSCE group (NOG). We assessed comments across five domains including Initial Assessment (IA), Diagnostic Action (DA), Therapeutic Action (TA), Communication (COM), and entrustment. Analyses included structured qualitative coding and (non)parametric and descriptive analyses.

RESULTS

In the EMOG, faculty's positive comments in the entrustment domain corresponded to lower entrustment score Mean Ranks (MRs) for IA (<11.1), DA (<11.2), and entrustment (<11.6). In NOG, faculty's negative comments resulted in lower entrustment score MRs for TA (<11.8 and <10) and DA (<12.4), and positive comments resulted in higher entrustment score MRs for IA (>15.4) and COM (>17.6). In the NCAG, faculty's positive IA comments were negatively correlated with entrustment scores (ρ = -.27, P = .04). Across programs, faculty and residents made similar domain-specific comments 13% of the time.

CONCLUSIONS

Minimal and inconsistent associations were found between narrative and numerical feedback. Performance monitoring accuracy and feedback should be included in assessment validation.

Necessary but not sufficient: identifying conditions for effective feedback during internal medicine residents' clinical education

Competency-based medical education and programmatic assessment intend to increase the opportunities for meaningful feedback, yet these conversations remain elusive. By comparing resident and faculty perceptions of feedback opportunities within one internal medicine residency training program, we sought to understand whether and how principles underlying meaningful feedback could be supported or constrained across a variety of feedback opportunities. Using case-study qualitative methodology, interviews and focus groups were conducted to explore 19 internal medicine residents' and 7 faculty members' perceptions of feedback across a variety of feedback opportunities: coaching, mini-CEXs, in-training evaluation reports and routine clinical supervision. Our data analysis moved iteratively between developing conceptual understandings and fine-grained analyses, while attending to both deductive and inductive analysis. Our results suggest that all feedback opportunities, including those created through formalized assessments, can foster meaningful feedback if faculty establish a trusting relationship with the resident, base their feedback on direct observation and support resident learning. However, formalized assessments were often perceived as inhibiting the conditions for meaningful feedback. A coaching program provided a context in which meaningful feedback could arise, in part because faculty were supported in shifting their focus from patient to resident. Meaningful feedback in clinical education may be fostered across a variety of feedback opportunities, however, it is often constrained by assessment. We must consider whether increasing the frequency of formative assessments may inhibit efforts to improve our feedback cultures while, in contrast, freeing up faculty to focus on supporting resident learning could improve these cultures.

Training disrupted: Practical tips for supporting competency-based medical education during the COVID-19 pandemic

The COVID-19 pandemic has disrupted healthcare systems around the world, impacting how we deliver medical education. The normal day-to-day routines have been altered for a number of reasons, including changes to scheduled training rotations, physical distancing requirements, trainee redeployment, and heightened level of concern. Medical educators will likely need to adapt their programs to maximize learning, maintain effective care delivery, and ensure competent graduates. Along with a continued focus on learner/faculty wellness, medical educators will have to optimize existing training experiences, adapt those that are no longer viable, employ new technologies, and be flexible when assessing competencies. These practical tips offer guidance on how to adapt medical education programs within the constraints of the pandemic landscape, stressing the need for communication, innovation, collaboration, flexibility, and planning within the era of competency-based medical education.

A systematic review of digital badges in health care education

OBJECTIVES

A challenge to competency-based medical education is the verification of skill acquisition. Digital badges represent an innovative instruction strategy involving the credentialing of competencies to provide evidence for achievement. Despite increasing interest in digital badges, there has been no synthesis of the health care education literature regarding this credentialing strategy. The present authors proposed to address this gap by conducting the first systematic review of digital badges in health care education, to reveal pedagogical and research limitations, and to provide an evidence-based foundation for the design and implementation of digital badges.

METHODS

A systematic search of the medical education literature from January 2008 to March 2019 was conducted using MEDLINE, Web of Science, CAB Abstracts and ScienceDirect. Included studies described digital badges in academic or professional medical education programmes in any health care profession. Included studies were appraised and quality assessment, methodological scoring, quantitative analysis and thematic extraction were conducted.

RESULTS

A total of 1050 relevant records were screened for inclusion; 201 full text articles were then assessed for eligibility, which resulted in the identification of 30 independent papers for analysis. All records had been published since 2013; 77% were journal articles, and 83% involved academic health care education programmes. Scores for quality were relatively moderate. Thematic analyses revealed implications for the design and implementation of digital badges: learner characteristics may moderate student outcomes; the novelty effect can negate the value of digital badges, and educators may overcome instruction-related challenges with digital badges using design and implementation strategies such as the creation of badging ecosystems.

CONCLUSIONS

The results indicate a growing momentum for the use of digital badges as an innovative instruction and credentialing strategy within higher education and provide evidence for outcomes within a learner-centred, competency-based model of medical education. There is a paucity of research to support the design and implementation of this credentialing system in health care education. The potential benefits necessitate future high-quality analyses reporting institutional, patient and workplace-based outcomes to evaluate the effectiveness and moderating conditions of digital badges.

Internal Medicine Resident Professionalism Assessments: Exploring the Association With Patients' Overall Satisfaction With Their Hospital Stay

PURPOSE

Successful training of internal medicine (IM) residents requires accurate assessments. Patients could assess IM residents in a hospital setting, but medical educators must understand how contextual factors may affect assessments. The objective was to investigate relationships between patient, resident, and hospital-encounter characteristics and the results of patient assessments of IM resident professionalism.

METHOD

The authors performed a prospective cohort study of postgraduate year 1 (PGY-1) IM residents and their patients at 4 general medicine inpatient teaching services at Mayo Clinic Hospital-Rochester, Saint Marys Campus in Rochester, Minnesota, from July 1, 2015, through June 30, 2016. Patient assessments of resident professionalism were adapted from validated instruments. Multivariable modeling with generalized estimating equations was used to determine associations between patient assessment scores and characteristics of residents, residents' clinical performance and evaluations (including professionalism assessments in other settings), patients, and hospital encounters and to account for repeated assessments of residents.

RESULTS

A total of 409 patients assessed 72 PGY-1 residents (mean [SD], 5.7 [3.0] patient assessments per resident). In the multivariable model, only the highest rating out of 5 levels for overall satisfaction with hospital stay was significantly associated with patient assessment scores of resident professionalism (β [SE], 0.80 [0.08]; P < .001). Hospitalized patients' assessment scores of resident professionalism were not significantly correlated with assessment scores of resident professionalism in other clinical settings.

CONCLUSIONS

Hospitalized patients' assessment scores of in-hospital resident professionalism were strongly correlated with overall patient satisfaction with hospital stay but were not correlated with resident professionalism in other settings. The limitations of patient evaluations should be considered before incorporating these evaluations into programs of assessment.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation

PURPOSE

Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change.

METHOD

Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences.

RESULTS

In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow.

CONCLUSIONS

Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills

BACKGROUND

Point-of-care ultrasound (POCUS) is increasingly used in a number of medical specialties. To support competency-based POCUS education, workplace-based assessments are essential.

OBJECTIVE

We developed a consensus-based assessment tool for POCUS skills and determined which items are critical for competence. We then performed standards setting to set cut scores for the tool.

METHODS

Using a modified Delphi technique, 25 experts voted on 32 items over 3 rounds between August and December 2016. Consensus was defined as agreement by at least 80% of the experts. Twelve experts then performed 3 rounds of a standards setting procedure in March 2017 to establish cut scores.

RESULTS

Experts reached consensus for 31 items to include in the tool. Experts reached consensus that 16 of those items were critically important. A final cut score for the tool was established at 65.2% (SD 17.0%). Cut scores for critical items are significantly higher than those for noncritical items (76.5% ± SD 12.4% versus 53.1% ± SD 12.2%, P < .0001).

CONCLUSIONS

We reached consensus on a 31-item workplace-based assessment tool for identifying competence in POCUS. Of those items, 16 were considered critically important. Their importance is further supported by higher cut scores compared with noncritical items.

Shadow systems in assessment: how supervisors make progress decisions in practice

Medical educators are tasked with decisions on trainee progression and credentialing for independent clinical practice, which requires robust evidence from workplace-based assessment. It is unclear how the current promotion of workplace-based assessment as a pedagogical approach to promote learning has impacted this use of assessments for decision-making; meeting both these purposes may present unforeseen challenges. In this study we explored how supervisors make decisions on trainee progress in practice. We conducted semi-structured interviews with 19 supervisors of postgraduate anesthesia training across Australia and New Zealand and undertook thematic analysis of the transcripts. Supervisors looked beyond the formal assessment portfolio when making performance decisions. They instead used assessment 'shadow systems' based on their own observation and confidential judgements from trusted colleagues. Supervisors' decision making involved expert judgement of the perceived salient aspects of performance and the standard to be attained while making allowances for the opportunities and constraints of the local learning environment. Supervisors found making progress decisions an emotional burden. When faced with difficult decisions, they found ways to share the responsibility and balance the potential consequences for the trainee with the need to protect their patients. Viewed through the lens of community of practice theory, the development of assessment 'shadow systems' indicates a lack of alignment between local workplace assessment practices and the prescribed programmatic assessment approach to high-stakes progress decisions. Avenues for improvement include cooperative development of formal assessment processes to better meet local needs or incorporating the information in 'shadow systems' into formal assessment processes.

Are we preparing for collaboration, advocacy and leadership? Targeted multi-site analysis of collaborative intrinsic roles implementation in medical undergraduate curricula

BACKGROUND

The Collaborator, Health Advocate and Leader/Manager roles are highly relevant for safe patient management and optimization of healthcare system in rehabilitation and prevention. They are defined in competency-based frameworks and incorporate competencies empowering physicians to master typical daily tasks in interdisciplinary, interprofessional and institutional collaboration. However, appropriate implementation of roles remains difficult in undergraduate medical education (UME) and needs to be closely monitored. The aim of this cross-institutional mapping study was to examine for the roles of Collaborator, Health Advocate and Leader/Manager: (1) To what extent do German UME programs explicitly meet the given standards after 5 years of study? (2) Which information may be obtained from multi-site mapping data for evidence-based reflection on curricula and framework?

METHODS

In a joint project of eight German UME programs, 80 to 100% of courses were mapped from teachers' perspective against given national standards: (sub-)competency coverage, competency level attainment and assessment. All faculties used a common tool and consented procedures for data collection and processing. The roles' representation was characterized by the curricular weighting of each role content expressed by the percentage of courses referring to it (citations). Data were visualized in a benchmarking approach related to a general mean of the intrinsic roles as reference line.

RESULTS

(Sub-)competencies of the Health Advocate are consistently well-integrated in curricula with a wide range of generally high curricular weightings. The Collaborator reveals average curricular representation, but also signs of ongoing curricular development in relevant parts and clear weaknesses regarding assessment and achieved outcomes. The Leader/Manager displays consistently lowest curricular weightings with several substantial deficiencies in curricular representation, constructive alignment and/or outcome level. Our data allow identifying challenges to be considered by local curriculum developers or framework reviewers (e.g. non-achievement of competency levels, potential underrepresentation, lacking constructive alignment).

CONCLUSION

Our non-normative, process-related benchmarking approach provides a differentiated crosscut snapshot to compare programs in the field of others, thus revealing shortcomings in role implementation, especially for Leader/Manager and Collaborator. The synopsis of multi-site data may serve as an external reference for program self-assessment and goal-oriented curriculum development. It may also provide practical data for framework review.

In-the-Moment Feedback and Coaching: Improving R2C2 for a New Context

BACKGROUND

The R2C2, a 4-phase feedback and coaching model, builds relationships, explores reactions, determines content and coaches for change, and facilitates formal feedback conversations between clinical supervisors/preceptors and residents. Formal discussions about performance are typically based on collated information from daily encounter sheets, objective structured clinical examinations, multisource feedback, and other data. This model has not been studied in settings where brief feedback and coaching conversations occur immediately after a specific clinical experience.

OBJECTIVE

We explored how supervisors adapt the R2C2 model for in-the-moment feedback and coaching and developed a guide for its use in this context.

METHODS

Eleven purposefully selected supervisors were interviewed in 2018 to explore where they used the R2C2 model, how they adapted it for in-the-moment conversations, and phrases used corresponding to each phase that could guide design of a new R2C2 in-the-moment model.

RESULTS

Participants readily adapted the model to varied feedback situations; each of the 4 phases were relevant for conversations. Phase-specific phrases that could enable effective coaching conversations in a limited amount of time were identified. Data facilitated a revision of the original R2C2 model for in-the-moment feedback and coaching conversations and design of an accompanying trifold brochure to enable its effective use.

CONCLUSIONS

The R2C2 in-the-moment model offers a systematic approach to feedback and coaching that builds on the original model, yet addresses time constraints and the need for an iterative conversation between the reaction and content phases. The model enables supervisors to coach and co-create an action plan with residents to improve performance.

Medical students' experiences with goals of care discussions and their impact on professional identity formation

CONTEXT

Goals of care (GoC) discussions occur amongst patients, family members and clinicians in order to establish plans of care and are invaluable aspects of end-of-life care. In previous research, medical learners have reported insufficient training and emotional distress about end-of-life decision making, but most studies have focused on postgraduate trainees and have been quantitative or have evaluated specific educational interventions. None have qualitatively explored medical students' experiences with GoC discussions, their perceptions of associated hidden curricula, and the impacts of these on professional identity formation (PIF), the individualised developmental processes by which laypersons evolve to think, act and feel like, and ultimately become, medical professionals.

METHODS

Using purposive sampling at one Canadian medical school, individual semi-structured interviews were conducted with 18 medical students to explore their experiences with GoC discussions during their core internal medicine clerkship. Interviews were audiorecorded, transcribed and anonymised. Concurrently with data collection, transcripts were analysed iteratively and inductively using interpretative phenomenological analysis, a qualitative research approach that allows the rich exploration of subjective experiences.

RESULTS

Participants reported minimal support and supervision in conducting GoC discussions, which were experienced as ethically challenging, emotionally powerful encounters exemplifying tensions between formal and hidden curricula. Role modelling and institutional culture were key mechanisms through which hidden curricula were transmitted, subverting formal curricula in doing so and contributing to participants' emotional distress. Participants' coping responses were generally negative and included symptoms of burnout, the pursuit of standardisation, rationalisation, compartmentalisation and the adaptation of previously held, more idealised professional identities.

CONCLUSIONS

GoC discussions in this study were often led by inexperienced medical students and impacted negatively on their PIF. Through complex emotional processes, they struggled to reconcile earlier concepts of physician identities with newly developing ones and often reluctantly adopted suboptimal professional behaviours and attitudes. Improved education about GoC discussions is necessary for patient care and may represent concrete and specific opportunities to influence students' PIF positively.

A balancing act: The Supervisor of Training role in anaesthesia education

In this qualitative study, we report how Supervisors of Training, educational supervisors overseeing the learning of anaesthesia trainees, experience their role in practice. Using purposive sampling, we interviewed Supervisors of Training from across Australia and New Zealand. The interviews began by asking ‘what do you see as your role as a Supervisor of Training?’ then explored the response in detail. Following the technique of thematic analysis, inductive analysis occurred as data were collected until we generated a thematic structure sufficient to address our research question after 19 interviews. In the first three of the four identified themes, Supervisors of Training perceived themselves as the fulcrum of the learning environment, ‘the something in between’. These three themes were: guiding and assessing trainees; identifying, supporting, and adjudicating trainee underperformance; and mediating trainees’ relationship with the hospital. Participants perceived themselves as a broker between trainees, their colleagues, their hospital, the Australian and New Zealand College of Anaesthetists and the community to varying degrees at different times. Negotiating these competing responsibilities required Supervisors of Training to manage multiple different relationships and entailed significant emotional work. Our fourth theme, scarcity, described the imbalance between these demands and the time and resources available. The complexity of the Supervisor of Training role and the tensions between these competing demands is underappreciated. Our findings would support strategies to mitigate the administrative load and share the decision-making burden of the role and to enhance the capability of Supervisors of Training by requiring formal training for the role.

Beyond the United States Medical Licensing Examination Score: Assessing Competence for Entering Residency

Assessments of physician learners during the transition from undergraduate to graduate medical education generate information that may inform their learning and improvement needs, determine readiness to move along the medical education continuum, and predict success in their residency programs. To achieve a constructive transition for the learner, residency program, and patients, high-quality assessments should provide meaningful information regarding applicant characteristics, academic achievement, and competence that lead to a suitable match between the learner and the residency program's culture and focus.The authors discuss alternative assessment models that may correlate with resident physician clinical performance and patient care outcomes. Currently, passing the United States Medical Licensing Examination Step examinations provides one element of reliable assessment data that could inform judgments about a learner's likelihood for success in residency. Yet, learner capabilities in areas beyond those traditionally valued in future physicians, such as life experiences, community engagement, language skills, and leadership attributes, are not afforded the same level of influence when candidate selections are made.While promising new methods of screening and assessment-such as objective structured clinical examinations, holistic assessments, and competency-based assessments-have attracted increased attention in the medical education community, currently they may be expensive, be less psychometrically sound, lack a national comparison group, or be complicated to administer. Future research and experimentation are needed to establish measures that can best meet the needs of programs, faculty, staff, students, and, more importantly, patients.

Intrinsic roles in the crosshair - strategic analysis of multi-site role implementation with an adapted matrix map approach

BACKGROUND

The implementation of competency-based intrinsic roles in undergraduate medical education remains a challenge. Faculties in transition need to be provided with generalizable curricular data in order to facilitate orientation on curricular roles' representation and to decide on steps of curriculum development. Explicit and implicit representation of objectives and multi-site agreement can be viewed as status indicators for the adoption of roles. Our aim was to develop a pragmatic cross-locational approach to capture roles' developmental status in an overview and prioritize strategic recommendations.

METHODS

Based on the mapping data from six German medical faculties, the relationship between explicit and implicit curricular representation of role' objectives (weighting) and extent of programs' consent (agreement) was calculated. Data was visualized in a role-specific Matrix Map to analyse roles' implicit-explicit relation and risk-value potential. The matrix was combined with Roger's stages of innovation diffusion for differentiated interpretation of the developmental role status.

RESULTS

Entangling multi-site agreement and curricular weighting, the 4-Field-Matrix allows to assess objectives based on their current localization in a quadrant: "Disregard" (lower left) and "Progress" quadrant (upper left) reveal the diffusion period; "Potential" (lower right) and "Emphasis" quadrant (upper right) indicate the adoption period. The role patterns differ in curricular representation, progression and clarity: (1) Scholar: explicit/implicit - scattered across the matrix; most explicit objectives in "Progress". (2) Health Advocate: explicit - primarily in "Emphasis"; only role in which the explicit representation significantly exceeds the implicit. (3) Collaborator: explicit - mainly "Potential"; implicit - "Progress" or "Emphasis". (4) Professional: explicit - primarily "Potential" but also "Emphasis"; implicit - "Progress" and "Emphasis"; appears better adopted but scattered in weighting; high hidden curricula. (5) Manager: explicit and implicit - exclusively in "Potential", without signs of development. Role patterns correspond to evidences from literature. Exemplified with roles, quadrant-specific strategies and measures are suggested. Framework reviewers may gain information for discussion of critical content.

CONCLUSION

The Matrix Map enables to catch intuitively the status of intrinsic roles' profiles regarding role pattern, implicit-explicit relation and programs agreement. Thus, interpretation and informed discussions are fostered. Further target-oriented analyses and strategic developments can be conducted to enhance transparency and resource-efficiency.

Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring

Background

Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED.

Methods

The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen's simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson's correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback.

Results

There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant (r = 0.630, n = 17, p < 0.01). Further, qualitative analysis demonstrated overall management and leadership themes were more common narratives in the workplace, while more specific task-based feedback predominated in the simulation-based assessment. Both workplace-based and simulation-based narratives frequently commented on communication skills.

Conclusions

In this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression.

Medical education today: all that glitters is not gold

BACKGROUND

The medical education system based on principles advocated by Flexner and Osler has produced generations of scientifically grounded and clinically skilled physicians whose collective experiences and contributions have served medicine and patients well. Yet sweeping changes launched around the turn of the millennium have constituted a revolution in medical education. In this article, a critique is presented of the new undergraduate medical education (UME) curricula in relationship to graduate medical education (GME) and clinical practice.

DISCUSSION

Medical education has changed and will continue to change in response to scientific advances and societal needs. However, enthusiasm for reform needs to be tempered by a more measured approach to avoid unintended consequences. Movement from novice to master in medicine cannot be rushed. An argument is made for a shoring up of biomedical science in revised curricula with the beneficiaries being nascent practitioners, developing physician-scientists --and the public.

CONCLUSION

Unless there is further modification, the new integrated curricula are at risk of produce graduates deficient in the characteristics that have set physicians apart from other healthcare professionals, namely high-level clinical expertise based on a deep grounding in biomedical science and understanding of the pathologic basis of disease. The challenges for education of the best possible physicians are great but the benefits to medicine and society are enormous.

Student perspective on outcomes and process - Recommendations for implementing competency-based medical education

PURPOSE

Competency-based medical education (CBME) seeks to prepare undergraduate and postgraduate trainees for clinical practice. Its major emphasis is on outcomes, but questions about how best to reach these remain. One key issue is the need to integrate what matters most to students when setting educational goals: this is crucial if we are to design curricula that trainees understand and engage with, and that promote successful achievement of competencies.

METHOD

We interviewed medical students in years 4 and 6 of a 6-year medical degree and used thematic analysis to understand their main educational priorities and how these fit with the aims of CBME.

RESULTS

Two major themes emerged: features of content and process. For content, students wanted clear guidance on what constitutes competence, finding broad outcome statements abstract and difficult to understand as novices. They also attach critical importance to features of process such as being welcomed, included in clinical teams and being known personally - these promote motivation, understanding, and professional development.

CONCLUSIONS

We present recommendations for those designing CBME curricula to emphasize the student perspective: what kind of guidance on outcomes is required, and features of process that must not be neglected if competence is to be achieved.

Managing tensions in assessment: moving beyond either-or thinking

CONTEXT

In health professions education, assessment systems are bound to be rife with tensions as they must fulfil formative and summative assessment purposes, be efficient and effective, and meet the needs of learners and education institutes, as well as those of patients and health care organisations. The way we respond to these tensions determines the fate of assessment practices and reform. In this study, we argue that traditional 'fix-the-problem' approaches (i.e. either-or solutions) are generally inadequate and that we need alternative strategies to help us further understand, accept and actually engage with the multiple recurring tensions in assessment programmes.

METHODS

Drawing from research in organisation science and health care, we outline how the Polarity Thinking™ model and its 'both-and' approach offer ways to systematically leverage assessment tensions as opportunities to drive improvement, rather than as intractable problems. In reviewing the assessment literature, we highlight and discuss exemplars of specific assessment polarities and tensions in educational settings. Using key concepts and principles of the Polarity Thinking™ model, and two examples of common tensions in assessment design, we describe how the model can be applied in a stepwise approach to the management of key polarities in assessment.

DISCUSSION

Assessment polarities and tensions are likely to surface with the continued rise of complexity and change in education and health care organisations. With increasing pressures of accountability in times of stretched resources, assessment tensions and dilemmas will become more pronounced. We propose to add to our repertoire of strategies for managing key dilemmas in education and assessment design through the adoption of the polarity framework. Its 'both-and' approach may advance our efforts to transform assessment systems to meet complex 21st century education, health and health care needs.

Association of a Competency-Based Assessment System With Identification of and Support for Medical Residents in Difficulty

IMPORTANCE

Competency-based medical education is now established in health professions training. However, critics stress that there is a lack of published outcomes for competency-based medical education or competency-based assessment tools.

OBJECTIVE

To determine whether competency-based assessment is associated with better identification of and support for residents in difficulty.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study of secondary data from archived files on 458 family medicine residents (2006-2008 and 2010-2016) was conducted between July 5, 2016, and March 2, 2018, using a large, urban family medicine residency program in Canada.

EXPOSURES

Introduction of the Competency-Based Achievement System (CBAS).

MAIN OUTCOMES AND MEASURES

Proportion of residents (1) with at least 1 performance or professionalism flag, (2) receiving flags on multiple distinct rotations, (3) classified as in difficulty, and (4) with flags addressed by the residency program.

RESULTS

Files from 458 residents were reviewed (pre-CBAS: n = 163; 81 [49.7%] women; 90 [55.2%] aged >30 years; 105 [64.4%] Canadian medical graduates; post-CBAS: n = 295; 144 [48.8%] women; 128 [43.4%] aged >30 years; 243 [82.4%] Canadian medical graduates). A significant reduction in the proportion of residents receiving at least 1 flag during training after CBAS implementation was observed (0.38; 95% CI, 0.377-0.383), as well as a significant decrease in the numbers of distinct rotations during which residents received flags on summative assessments (0.24; 95% CI, 0.237-0.243). There was a decrease in the number of residents in difficulty after CBAS (from 0.13 [95% CI, 0.128-0.132] to 0.17 [95% CI, 0.168-0.172]) depending on the strictness of criteria defining a resident in difficulty. Furthermore, there was a significant increase in narrative documentation that a flag was discussed with the resident between the pre-CBAS and post-CBAS conditions (0.18; 95% CI, 0.178-0.183).

CONCLUSIONS AND RELEVANCE

The CBAS approach to assessment appeared to be associated with better identification of residents in difficulty, facilitating the program's ability to address learners' deficiencies in competence. After implementation of CBAS, residents experiencing challenges were better supported and their deficiencies did not recur on later rotations. A key argument for shifting to competency-based medical education is to change assessment approaches; these findings suggest that competency-based assessment may be useful.

Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education

Medical education has shifted to a competency-based paradigm, leading to calls for improved learner assessment methods and validity evidence for how assessment data are interpreted. Clinical competency committees (CCCs) use the collective input of multiple people to improve the validity and reliability of decisions made and actions taken based on assessment data. Significant heterogeneity in CCC structure and function exists across postgraduate medical education programs and specialties, and while there is no "one-size-fits-all" approach, there are ways to maximize value for learners and programs. This paper collates available evidence and the authors' experiences to provide practical tips on CCC purpose, membership, processes, and outputs. These tips can benefit programs looking to start a CCC and those that are improving their current CCC processes.

Timing of milestone competency acquisition in neurology residency: What by when?

OBJECTIVE

To determine the stage of training at which neurology residents should achieve individual elements of the Accreditation Council for Graduate Medical Education neurology Milestones and to examine the relationship between perceived importance of Milestones and the stage by which they should be achieved.

METHODS

A modified Delphi technique was used to establish consensus postgraduate year (PGY) expectations for neurology Milestone competencies across 3 geographically and administratively distinct Mayo Clinic neurology residency programs. Timing expectations were examined for relationships to perceived importance of the individual Milestones and effects of participant characteristics.

RESULTS

PGY expectations for neurology Milestone elements ranged from PGY 1.3 to PGY 4.1. Extent of rater educational seniority had no effect on PGY competency expectations. There was a moderate inverse relationship between perceived importance of the Milestone element and the PGY by which it should be achieved (r _s = -0.74, p < 0.0001).

CONCLUSIONS AND RELEVANCE

Expectations for neurology Milestone competency acquisition can be measured and may help inform individual program design, educational expectations, and future Milestone design.

Developing the Korean Association of Medical Colleges graduate outcomes of basic medical education based on "the role of Korean doctor, 2014"

The Korean Association of Medical Colleges (KAMC) developed graduate outcomes based on "The role of Korean doctor, 2014" to serve as guidelines regarding outcome-based education in Korea. The working group in this study analyzed 65 competencies proposed in "The role of Korean doctor, 2014" according to the developmental principle that certain outcomes should be demonstrated at the point of entry into the graduate medical education. We established 34 competencies as "preliminary graduate outcomes" (PGOs). The advisory committee consisted of 11 professors, who reviewed the validity of PGOs. Ultimately, a total of 19 "revised graduate outcomes" (RGOs) were selected. We modified the RGOs based on opinions from medical schools and a public hearing. In November 2017, the KAMC announced the "graduate outcomes for basic medical education," which serves as a guide for basic medical education for the 40 medical schools throughout Korea. Medical schools can expand the graduate outcomes according to their educational goals and modify them according to their own context. We believe that graduate outcomes can be a starting point for connecting basic medical education to graduate medical education.

Using the multiple mini interview as an assessment strategy within the first year of a health professions curriculum

BACKGROUND

The multiple mini-interview (MMI) is a common assessment strategy used in student selection. The MMI as an assessment strategy within a health professions curriculum, however, has not been previously studied. This study describes the integration of a 5-station MMI as part of an end-of-year capstone following the first year of a health professions curriculum. The goal of the capstone MMI was to assess professional competencies of students and to offer formative feedback to prepare students for their upcoming clinical practice experiences. The purpose of this study was to evaluate the psychometric properties of an MMI integrated into a health professions curriculum.

METHODS

Five capstone MMI stations were designed to each evaluate a single construct assessed by one rater. A principal component analysis (PCA) was used to evaluate the structure of the model and its ability to distinguish 5 separate constructs. A Multifaceted Rasch Measurement (MFRM) model assessed student performance and estimated the sources of measurement error attributed to 3 facets: student ability, rater stringency, and station difficulty. At the conclusion, students were surveyed about the capstone MMI experience.

RESULTS

The PCA confirmed the MMI reliably assessed 5 unique constructs and performance on each station was not strongly correlated with one another. The 3-facet MFRM analysis explained 58.79% of the total variance in student scores. Specifically, 29.98% of the variance reflected student ability, 20.25% reflected rater stringency, and 8.56% reflected station difficulty. Overall, the data demonstrated an acceptable fit to the MFRM model. The majority of students agreed the MMI allowed them to effectively demonstrate their communication (80.82%), critical thinking (78.77%), and collaboration skills (70.55%).

CONCLUSIONS

The MMI can be a valuable assessment strategy of professional competence within a health professions curriculum. These findings suggest the MMI is well-received by students and can produce reliable results. Future research should explore the impact of using the MMI as a strategy to monitor longitudinal competency development and inform feedback approaches.

Emotions and assessment: considerations for rater-based judgements of entrustment

CONTEXT

Assessment is subject to increasing scrutiny as medical education transitions towards a competency-based medical education (CBME) model. Traditional perspectives on the roles of assessment emphasise high-stakes, summative assessment, whereas CBME argues for formative assessment. Revisiting conceptualisations about the roles and formats of assessment in medical education provides opportunities to examine understandings and expectations of the assessment of learners. The act of the rater generating scores might be considered as an exclusively cognitive exercise; however, current literature has drawn attention to the notion of raters as measurement instruments, thereby attributing additional factors to their decision-making processes, such as social considerations and intuition. However, the literature has not comprehensively examined the influence of raters' emotions during assessment. In this narrative review, we explore the influence of raters' emotions in the assessment of learners.

METHODS

We summarise existing literature that describes the role of emotions in assessment broadly, and rater-based assessment specifically, across a variety of fields. The literature related to emotions and assessment is examined from different perspectives, including those of educational context, decision making and rater cognition. We use the concept of entrustable professional activities (EPAs) to contextualise a discussion of the ways in which raters' emotions may have meaningful impacts on the decisions they make in clinical settings. This review summarises findings from different perspectives and identifies areas for consideration for the role of emotion in rater-based assessment, and areas for future research.

CONCLUSIONS

We identify and discuss three different interpretations of the influence of raters' emotions during assessments: (i) emotions lead to biased decision making; (ii) emotions contribute random noise to assessment, and (iii) emotions constitute legitimate sources of information that contribute to assessment decisions. We discuss these three interpretations in terms of areas for future research and implications for assessment.

A competency-based approach to critical care education

The medical education environment is rapidly changing. Competency-based medical education (CBME) is a great advance, but operationalizing competencies for teaching and assessment is problematic. Entrustable professional activities (EPAs) can revitalize CBME by connecting competencies to practice, creating flexibility in programs. CBME requires and deepens the nature of workplace-based assessments. It is important to use EPAs to verify residents’ ability to care for critically ill patients unsupervised in simulation education.

Implementing competency-based medical education: What changes in curricular structure and processes are needed?

Medical educators must prepare for a number of challenges when they decide to implement a competency-based curriculum. Many of these challenges will pertain to three key aspects of implementation: organizing the structural changes that will be necessary to deliver new curricula and methods of assessment; modifying the processes of teaching and evaluation; and helping to change the culture of education so that the CBME paradigm gains acceptance. This paper focuses on nine key considerations that will support positive change in first two of these areas. Key considerations include: ensuring that educational continuity exists amongst all levels of medical education, altering how time is used in medical education, involving CBME in human health resources planning, ensuring that competent doctors work in competent health care systems, ensuring that information technology supports CBME, ensuring that faculty development is supported, ensuring that the rights and responsibilities of the learner are appropriately balanced in the workplace, preparing for the costs of change, and having appropriate leadership in order to achieve success in implementation.

Competency-based medical education and continuing professional development: A conceptualization for change

Competency-based medical education (CBME) is as important in continuing professional development (CPD) as at any other stage of a physician's career. Principles of CBME have the potential to revolutionize CPD. Transitioning to CBME-based CPD will require a cultural change to gain commitment from physicians, their employers and institutions, CPD providers, professional organizations, and medical regulators. It will require learning to be aligned with professional and workplace standards. Practitioners will need to develop the expertise to systematically examine their own clinical performance data, identify performance improvement opportunities and possibilities, and develop a plan to address areas of concern. Health care facilities and systems will need to produce data on a regular basis and to develop and train CPD educators who can work with physician groups. Stakeholders, such as medical regulatory authorities who are responsible for licensing physicians and other standard-setting bodies that credential and develop maintenance-of-certification systems, will need to change their paradigm of competency enhancement through CPD.

Implementing competency-based medical education: Moving forward

For more than 60 years, competency-based education has been proposed as an approach to education in many disciplines. In medical education, interest in CBME has grown dramatically in the last decade. This editorial introduces a series of papers that resulted from summits held in 2013 and 2016 by the International CBME Collaborators, a scholarly network whose members are interested in developing competency-based approaches to preparing the next generation of health professionals. An overview of the papers is given, as well as a summary of landmarks in the conceptual evolution and implementation of CBME. This series follows on a first collection of papers published by the International CBME Collaborators in Medical Teacher in 2010.

Core principles of assessment in competency-based medical education

The meaningful assessment of competence is critical for the implementation of effective competency-based medical education (CBME). Timely ongoing assessments are needed along with comprehensive periodic reviews to ensure that trainees continue to progress. New approaches are needed to optimize the use of multiple assessors and assessments; to synthesize the data collected from multiple assessors and multiple types of assessments; to develop faculty competence in assessment; and to ensure that relationships between the givers and receivers of feedback are appropriate. This paper describes the core principles of assessment for learning and assessment of learning. It addresses several ways to ensure the effectiveness of assessment programs, including using the right combination of assessment methods and conducting careful assessor selection and training. It provides a reconceptualization of the role of psychometrics and articulates the importance of a group process in determining trainees' progress. In addition, it notes that, to reach its potential as a driver in trainee development, quality care, and patient safety, CBME requires effective information management and documentation as well as ongoing consideration of ways to improve the assessment system.