Perspective: the ACGME toolbox: half empty or half full?

What/Why/When/Where/How Framework and Faculty Development Workshop to Improve the Utility of Narrative Evaluations for Assessing Internal Medicine Residents

Introduction

Clinical competency committees (CCCs) rely on narrative evaluations to assess resident competency. Despite the emphasis on these evaluations, their utility is frequently hindered by lack of sufficient detail for use by CCCs. Prior resources have sought to improve specificity of comments and use of evaluations by residents but not their utility for CCCs in assessing trainee performance.

Methods

We developed a 1-hour faculty development workshop focused on a newly devised framework for Department of Medicine faculty supervising internal medicine residents. The what/why/when/where/how framework highlighted key features of useful narrative evaluations: behaviors of strength and growth, contextualized observations, improvement over time, and actionable next steps. Workshop sessions were implemented at a large multisite internal medicine residency program. We assessed the workshop by measuring attendee confidence and skill in writing narrative evaluations useful for CCCs. Skill was assessed through a rubric adapted from literature on the utility of narrative evaluations.

Results

Fifty-four participants started the presurvey, and 33 completed the workshop, for a response rate of 61%. Participant confidence improved pre-, post-, and 3 months postworkshop. Total utility scores improved in mock evaluations from 12.4 to 15.5 and in real evaluations from 13.7 to 15.0, but only some subcomponent scores improved, with fewer improving in the real evaluations.

Discussion

A short workshop focusing on our framework improves confidence and utility of narrative evaluations of internal medicine residents for use by CCCs. Next steps should include developing more challenging components of narrative evaluations for continued improvement in trainee performance and faculty assessment.

Croix R, van Melle E. Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Competency in medical training: current concepts, assessment modalities, and practical challenges

In the past two decades, competency-based medical education (CBME) has rapidly become the cornerstone of medical training and accreditation programmes worldwide. It has increasingly replaced traditional time-based educational approaches which were often rigid, fragmented, and overly emphasized clinical content knowledge over practical skillsets and attitudes. CBME adoption was in the hope of better preparing medical graduates for the demands and responsibilities of real-world clinical practice. For all the supposed merits of CBME, there hitherto remains difficulties in arriving at comprehensive and practical 'competency' definitions, and actual challenges with implementation of clinical competency assessment modalities pertaining to construct validity, reliability, and applicability with the use and interpretation of evaluation metrics. Therefore, in this article, we describe the various conceptualizations of 'competency' in medical education literature and attempt to refine its usage in practice to meet the evolving needs and expectations of healthcare stakeholders, as well as incorporate emerging concepts in the medical education discourse. We herein propose that clinical 'competencies' should be defined as multi-domain clinical expertise, comprising medical knowledge, skills, attitudes and metacognitive capabilities that reflects the prevailing needs of healthcare stakeholders, and is inferred from performance evaluations of medical trainees. In order to attain 'competence', there must then be a process of integrating multi-domain competencies into meaningful professional identity formation that is commensurate with the context and stage of medical training. In addition, we review the current competency assessment modalities, including common pitfalls with their usage, and sought to provide practical strategies to mitigate the identified challenges.

Applying a video recording, video-based rating method in OSCEs

INTRODUCTION

Objective structured clinical examination (OSCE) results could be affected by low homogeneity of examiners, non-retrospectiveness of test results, and examiner-cohort effect. In China, many students participate in medical qualification examinations, and this issue is particularly significant. This study aimed to develop a video recording, video-based rating method and compare the reliability of video and on-site ratings to enhance the quality assurance of OSCEs.

METHODS

The subjects of this study were clinical students one year after graduation participating in the clinical skills portion of the National Medical Licensing Examination. The participants were from four cities in Jiangsu province. Participants were randomly allocated to on-site and video rating groups to evaluate the rating methods consistency. We verified the reliability of recording equipment and evaluability of video recording. Moreover, we compared the consistency and equivalence of the two rating methods and analyzed the impact of video recording on scores.

RESULTS

The reliability of recording equipment and evaluability of video recording were high. Evaluation consistency between experts and examiners was acceptable, and there was no difference in evaluation results (P = 0.61). There was good consistency between video and on-site rating; however, a difference between the two rating methods was detected. The scores of video-based rating group students were lower than those of all students (P < 0.00).

CONCLUSIONS

Video-based rating could be reliable and offer advantages over on-site rating. The video recording, video-based rating method could provide greater content validity based on its traceability and the ability to view details. Video recording, video-based rating offers a promising mthod for improving the effectiveness and fairness of OSCEs.

Striving to thrive or striving to survive: Professional identity constructions of medical trainees in clinical assessment activities

CONTEXT

Assessment plays a key role in competence development and the shaping of future professionals. Despite its presumed positive impacts on learning, unintended consequences of assessment have drawn increasing attention in the literature. Considering professional identities and how these can be dynamically constructed through social interactions, as in assessment contexts, our study sought to understand how assessment influences the construction of professional identities in medical trainees.

METHODS

Within social constructionism, we adopted a discursive, narrative approach to investigate the different positions trainees narrate for themselves and their assessors in clinical assessment contexts and the impact of these positions on their constructed identities. We purposively recruited 28 medical trainees (23 students and five postgraduate trainees), who took part in entry, follow-up and exit interviews of this study and submitted longitudinal audio/written diaries across nine-months of their training programs. Thematic framework and positioning analyses (focusing on how characters are linguistically positioned in narratives) were applied using an interdisciplinary teamwork approach.

RESULTS

We identified two key narrative plotlines, striving to thrive and striving to survive, across trainees' assessment narratives from 60 interviews and 133 diaries. Elements of growth, development, and improvement were identified as trainees narrated striving to thrive in assessment. Neglect, oppression and perfunctory narratives were elaborated as trainees narrated striving to survive from assessment. Nine main character tropes adopted by trainees with six key assessor character tropes were identified. Bringing these together we present our analysis of two exemplary narratives with elaboration of their wider social implications.

CONCLUSION

Adopting a discursive approach enabled us to better understand not only what identities are constructed by trainees in assessment contexts but also how they are constructed in relation to broader medical education discourses. The findings are informative for educators to reflect on, rectify and reconstruct assessment practices for better facilitating trainee identity construction.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

PURPOSE

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data.

METHOD

For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies.

RESULTS

Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale.

CONCLUSIONS

Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

A Meaningful and Actionable Professionalism Assessment: Validity Evidence for the Professionalism Mini-Evaluation Exercise (P-MEX) Across 8 Years

PURPOSE

With the growing importance of professionalism in medical education, it is imperative to develop professionalism assessments that demonstrate robust validity evidence. The Professionalism Mini-Evaluation Exercise (P-MEX) is an assessment that has demonstrated validity evidence in the authentic clinical setting. Identifying the factorial structure of professionalism assessments determines professionalism constructs that can be used to provide diagnostic and actionable feedback. This study examines validity evidence for the P-MEX, a focused and standardized assessment of professionalism, in a simulated patient setting.

METHOD

The P-MEX was administered to 275 pediatric residency applicants as part of a 3-station standardized patient encounter, pooling data over an 8-year period (2012 to 2019 residency admission years). Reliability and construct validity for the P-MEX were evaluated using Cronbach's alpha, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA).

RESULTS

Cronbach's alpha for the P-MEX was 0.91. The EFA yielded 4 factors: doctor-patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. The CFA demonstrated good model fit with a root-mean-square error of approximation of .058 and a comparative fit index of .92, confirming the reproducibility of the 4-factor structure of professionalism.

CONCLUSIONS

The P-MEX demonstrates construct validity as an assessment of professionalism, with 4 underlying subdomains in doctor-patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. These results yield new confidence in providing diagnostic and actionable subscores within the P-MEX assessment. Educators may wish to integrate the P-MEX assessment into their professionalism curricula.

Translating outcome frameworks to assessment programmes: Implications for validity

OBJECTIVES

Competency-based medical education (CBME) requires that educators structure assessment of clinical competence using outcome frameworks. Although these frameworks may serve some outcomes well (e.g. represent eventual practice), translating these into workplace-based assessment plans may undermine validity and, therefore, trustworthiness of assessment decisions due to a number of competing factors that may not always be visible or their impact knowable. Explored here is the translation process from outcome framework to formative and summative assessment plans in postgraduate medical education (PGME) in three Canadian universities.

METHODS

We conducted a qualitative study involving in-depth semi-structured interviews with leaders of PGME programmes involved in assessment and/or CBME implementation, with a focus on their assessment-based translational activities and evaluation strategies. Interviews were informed by Callon's theory of translation. Our analytical strategy involved directed content analysis, allowing us to be guided by Kane's validity framework, whilst still participating in open coding and analytical memo taking. We then engaged in axial coding to systematically explore themes across the dataset, various situations and our conceptual framework.

RESULTS

Twenty-four interviews were conducted involving 15 specialties across three universities. Our results suggest: (i) using outcomes frameworks for assessment is necessary for good assessment but are also viewed as incomplete constructs; (ii) there are a number of social and practical negotiations with competing factors that displace validity as a core influencer in assessment planning, including implementation, accreditation and technology; and (iii) validity exists as threatened, uncertain and assumed due to a number of unchecked assumptions and reliance on surrogates.

CONCLUSIONS

Translational processes in CBME involve negotiating with numerous influencing actors and institutions that, from an assessment perspective, provide challenges for assessment scientists, institutions and educators to contend with. These processes are challenging validity as a core element of assessment designs. Educators must reconcile these influences when preparing for or structuring validity arguments.

North American Society for Pediatric Gastroenterology, Hepatology, and Nutrition Position Paper on Entrustable Professional Activities: Development of Pediatric Gastroenterology, Hepatology, and Nutrition Entrustable Professional Activities

Quality training in pediatric gastroenterology, hepatology, and nutrition is essential for the future of our specialty from advancing the science through research to providing clinical care for children with gastrointestinal, hepatic and nutritional disorders. As educational theory has developed, both the American Board of Pediatrics (ABP) and the Accreditation Council for Graduate Medical Education (ACGME) have commissioned projects to better define training including core competencies, and milestones with the goal of competency-based assessment. Seeking to provide a clinical context for these competencies and milestones, the ABP commissioned a project for each pediatric subspecialty to develop entrustable professional activities (EPA) while at the same time developing EPAs that are common to all pediatric subspecialties. North American Society for Pediatric Gastroenterology, Hepatology, Nutrition (NASPGHAN) commissioned an EPA Task Force to develop the pediatric gastroenterology, hepatology and nutrition EPAs. This document serves as an introduction to EPAs, including their historical background, underlying educational theory, and the process used to develop the pediatric gastroenterology, hepatology and nutrition EPAs in the United States of America.

Implementing standardised flow: navigating operational and professional dependencies

Purpose

The purpose of this study had two aims: (1) to extend insight regarding the challenges of implementing standardised work, via care pathways, in a healthcare setting by considering interactions with other operational (i.e. resource sharing, portfolio alignment) and professional (i.e. autonomous expertise) dependencies and (2) to develop novel insights regarding a specific flow mechanism, the stroke nurse practitioner, a form of flow “pilo” or guide.

Design/methodology/approach

This was a longitudinal case study of implementing the acute stroke care pathway in a National Health Service hospital in England based on 185 hours of non-participant observations and 68 semi-structured interviews. Archival documents were also analysed.

Findings

The combined flow, operational and professional dependency lens extends operations management understanding of the challenge of implementing standardised work in healthcare. One observed practice, the process pilot role, may be particularly valuable in dealing with these dependencies but it requires specific design and continuous support, for which the authors provide some initial guidance.

Research limitations/implications

The research was a single case study and was focussed on a single care pathway. The findings require replication and extension but offer a novel set of insights into the implications of standardised work in healthcare.

Originality/value

In addition to confirming that a multidependency lens adds conceptual and practical insight to the challenges of implementing standardised work in a healthcare setting, the findings and recommendations regarding flow “pilots” are novel. The authors' analysis of this role reveals new insights regarding the need for continued improvisation in standardised work.

Can Professionalism Mini-Evaluation Exercise Scores Predict Medical Residency Performance? Validity Evidence Across Five Longitudinal Cohorts

PURPOSE

The residency admissions process is a high-stakes assessment system with the purpose of identifying applicants who best meet standards of the residency program and the medical specialty. Prior studies have found that professionalism issues contribute significantly to residents in difficulty during training. This study examines the reliability (internal structure) and predictive (relations to other variables) validity evidence for a standardized patient (SP)-based Professionalism Mini-Evaluation Exercise (P-MEX) using longitudinal data from pediatrics candidates from admission to the end of the first year of postgraduate training.

METHOD

Data from 5 cohorts from 2012 to 2016 (195 invited applicants) were analyzed from the University of Geneva (Switzerland) Pediatrics Residency Program. Generalizability theory was used to examine the reliability and variance components of the P-MEX scores, gathered across 3 cases. Correlations and mixed-effects regression analyses were used to examine the predictive utility of SP-based P-MEX scores (gathered as part of the admissions process) with rotation evaluation scores (obtained during the first year of residency).

RESULTS

Generalizability was moderate (G coefficient = 0.52). Regression analyses predicting P-MEX scores to first-year rotation evaluations indicated significant standardized effect sizes for attitude and personality (β = 0.36, P = .02), global evaluation (β = 0.27, P = .048), and total evaluation scores (β = 0.34, P = .04).

CONCLUSIONS

Validity evidence supports the use of P-MEX scores as part of the admissions process to assess professionalism. P-MEX scores provide a snapshot of an applicant's level of professionalism and may predict performance during the first year of residency.

Objective Evaluation of a Simulation Course for Residents in the Pediatric Emergency Medicine Department: Breaking Bad News

INTRODUCTION

Breaking bad news (BBN), especially in the pediatric emergency medicine department, requires significant skill and delicacy due to the acute context of a busy emergency department (ED) and the lack of prior rapport with the patients and families. Pediatric literature on breaking bad news has mostly focused on pediatric oncology and pediatric critical care, with limited literature focused on pediatric emergency medicine. Review of the literature also reveals that most existing studies solely assess the learners' self-ratings of efficacy and comfort, and far fewer studies objectively evaluate learners' actual performance using simulation. Our objectives for this study was to use an objective assessment tool to assess residents' breaking bad news skills, pre- and post-simulation training, specifically in the setting of a pediatric emergency medicine department.

METHODS

34 residents were evaluated on their performance in breaking bad news via videotaped simulation encounters before and after teaching intervention. The "Modified Breaking Bad News Assessment Scale" (mBAS) was used as the assessment tool. A paired t-test analysis was conducted to examine the mean difference in pre- and post-simulation scores in each of the five mBAS domains.

RESULTS

Breaking bad news performance score improves one to two weeks post-intervention, and was statistically significant in three of five domains.

CONCLUSION

Our study shows that breaking bad news is a teachable skill that can be improved by simulated education in the pediatric emergency medicine department. This study demonstrates the utility of simulation course in improving breaking bad news skills in the pediatric emergency medicine department. Future work in developing focused simulation curriculums is important to improve provider communication skills and patient-physician relationships.

Competency-based medical education: the discourse of infallibility

BACKGROUND

Over the last two decades, competency-based frameworks have been internationally adopted as the primary educational approach in medicine. Yet competency-based medical education (CBME) remains contested in the academic literature. We look broadly at the nature of this debate to explore how it may shape scholars' understanding of CBME, and its implications for medical education research and practice. In doing so, we deconstruct unarticulated discourses and assumptions embedded in the CBME literature.

METHODS

We assembled an archive of literature focused on CBME. The archive dates from 1996, the publication year of the first CanMEDS Physician Competency Framework. We then conducted a Foucauldian critical discourse analysis (CDA) to delineate the dominant discourses underpinning the literature. CDA examines the intersections of language, social practices, knowledge and power relations to highlight how entrenched ways of thinking influence what can or cannot be said about a topic.

FINDINGS

Detractors of CBME have advanced an array of conceptual critiques. Proponents have often responded with a recurring discursive strategy that minimises these critiques and deflects attention from the underlying concept of the competency-based approach. As part of this process, conceptual concerns are reframed as two practical problems: implementation and interpretation. Yet the assertion that these are the construct's primary concerns was often unsupported by empirical evidence. These practices contribute to a discourse of infallibility of CBME.

DISCUSSION

In uncovering the discourse of infallibility, we explore how it can silence critical voices and hinder a rigorous examination of the competency-based approach. These discursive practices strengthen CBME by constructing it as infallible in the literature. We propose re-approaching the dialogue surrounding CBME as a starting point for empirical investigation, driven by the aim to broaden scholars' understanding of its design, development and implementation in medical education.

Mixed Messages or Miscommunication? Investigating the Relationship Between Assessors' Workplace-Based Assessment Scores and Written Comments

PURPOSE

The shift toward broader, programmatic assessment has revolutionized the approaches that many take in assessing medical competence. To understand the association between quantitative and qualitative evaluations, the authors explored the relationships that exist among assessors' checklist scores, task ratings, global ratings, and written comments.

METHOD

The authors collected and analyzed, using regression analyses, data from the McMaster Modular Assessment Program. The data were from emergency medicine residents in their first or second year of postgraduate training from 2012 through 2014. Additionally, using content analysis, the authors analyzed narrative comments corresponding to the "done" and "done, but needs attention" checklist score options.

RESULTS

The regression analyses revealed that the task ratings, provided by faculty assessors, are associated with the use of the "done, but needs attention" checklist score option. Analyses also identified that the "done, but needs attention" option is associated with a narrative comment that is balanced, providing both strengths and areas for improvement. Analysis of qualitative comments revealed differences in the type of comments provided to higher- and lower-performing residents.

CONCLUSIONS

This study highlights some of the relationships that exist among checklist scores, rating scales, and written comments. The findings highlight that task ratings are associated with checklist options while global ratings are not. Furthermore, analysis of written comments supports the notion of a "hidden code" used to communicate assessors' evaluation of medical competence, especially when communicating areas for improvement or concern. This study has implications for how individuals should interpret information obtained from qualitative assessments.

Getting off the carousel: De-centring the curriculum in medical education

None

Professional Development Perceptions and Practices Among U.S. Physicians: A Cross-Specialty National Survey

PURPOSE

Professional development (PD)-both for-credit continuing medical education (CME) and informal self-directed or point-of-care learning-is vital to all physicians. The authors sought to understand physicians' PD perceptions and practices and how these vary by specialty and practice type.

METHOD

The authors administered an Internet and paper survey, from September 2015 to April 2016, to randomly sampled U.S. physicians. Survey items addressed perceived PD needs and barriers and how physicians identify knowledge/skills gaps.

RESULTS

Of 4,648 invitees, 988 (21.6%) responded. Respondents believed that they already know what they need to learn (mean 5.8 [1 = strongly disagree; 7 = strongly agree]), can answer clinical questions using available resources (5.9), and want credit for learning during patient care (5.1). They did not strongly desire help identifying learning gaps (4.0) or indicate difficulty accumulating CME credits (3.1). Most PD was done during personal time (5.5). Competencies regarding medical knowledge/skills, wellness, informatics, and practice/systems improvement were rated the highest priority, while research, teaching, and professionalism were rated the lowest. The most important sources used to identify knowledge/skills gaps were immediate patient care needs (4.1 [1 = not important; 5 = extremely important]), personal awareness (3.8), and practice updates (3.7). The most important barriers were time (3.5) and cost (2.9). Differences by specialty and practice type were generally small and not statistically significant.

CONCLUSIONS

Physicians feel confident in identifying their own learning needs, perceive medical knowledge/skills as their highest-priority need, and desire more credit for learning during patient care.

Is It Time for Entrustable Professional Activities for Residency Program Directors?

Residency program directors (PDs) play an important role in establishing and leading high-quality graduate medical education programs. However, medical educators have failed to codify the position on a national level, and PDs are often not recognized for the significant role they play. The authors of this Commentary argue that the core entrustable professional activities (EPAs) framework may be a mechanism to further this work and define the roles and responsibilities of the PD position. Based on personal observations as PDs and communications with others in the academic medicine community, the authors used work in competency-based medical education to define a list of potential EPAs for PDs. The benefits of developing these EPAs include being able to define competencies for PDs using a deconstructive process, highlighting the increasingly important role PDs play in leading high-quality graduate medical education programs, using EPAs as a framework to assess PD performance and provide feedback, allowing PDs to focus their professional development efforts on the most important areas for their work, and helping guide the PD recruitment and selection processes.

[From the Residency Training in the United States to See the Challenges and Directions of China Residency Standardized Training]

住院医师规范化培训在我国已经全面启动并逐步铺开。由于住院医师规范化培训是临床医学毕业生成长为合格临床医师的重要途径，是保证临床医师均质化、提高医疗服务质量和水平的治本之策，所以此项工作受到各方瞩目。美国的住院医生培训已有近百年的历史，形成较为系统的模式也已有近50年，是西方医学教育的典型代表和成功模式。本文旨在通过比较中美两国在培训的目标、计划、管理机构、考核和薪酬等方面的差异，对我国规培制度的现行的制度安排和发展方向进行讨论。

Factors Influencing Physicians' Selection of Continuous Professional Development Activities: A Cross-Specialty National Survey

INTRODUCTION

We sought to understand what influences physicians' decisions about participation in continuous professional development (CPD) activities, and how often physicians engage in specific CPD activities.

METHODS

From September 2015 to April 2016, we administered a survey to 4648 randomly sampled licensed US physicians. Survey items addressed perceived barriers to CPD, factors that might influence participation in four prototypical CPD activities (reading an article, or completing a local activity, online course, or far-away course), and frequency of CPD engagement.

RESULTS

Nine hundred eighty-eight (21.6%) physicians responded. The most important barriers were time (mean [SD] 3.5 [1.3], 1 = not important, 5 = extremely important) and cost (2.9 [1.3]). In prioritizing factors influencing participation in four prototypical CPD activities, topical relevance consistently had the highest average rank. Quality of content and time to complete the activity were also frequently selected. Over the past 3 years, most physicians reported having participated in patient-focused learning and self-directed learning on a weekly basis; quality improvement and local continuing medical education (CME) activities several times per year; online learning, on-site courses, and national board-related activities a few times per year; and interprofessional learning less than once per year. Physicians believed that they ought to engage more often in all of these activities except board-related activities. They would like CME credit for these activities much more often than currently obtained.

DISCUSSION

The reasons physicians select a given CPD activity vary by activity, but invariably include topic and quality of content. Physicians want CME credit for the CPD activities they are already doing.

Preparing anesthesiology faculty for competency-based medical education

PURPOSE

Competency-based medical education (CBME) is quickly becoming the dominant organizing principle for medical residency programs. As CBME requires changes in the way medical education is delivered, faculty will need to acquire new skills in teaching and assessment in order to navigate the transition. In this paper, we examine the evidence supporting best practices in faculty development, propose strategies for faculty development for CBME-based residency programs, and discuss the results of faculty development initiatives at the pioneering anesthesia CBME residency program at the University of Ottawa.

SOURCE

Review of the current literature and information from the University of Ottawa anesthesia residency program.

PRINCIPAL FINDINGS

Faculty development is critical to the success of CBME programs. Attention must be paid to the competence of faculty to teach and assess all of the CanMEDS roles. At the University of Ottawa, some faculty development initiatives were very successful, while others were hindered by factors both internal and external to the residency program. Many faculty development activities had low attendance rates.

CONCLUSIONS

Faculty development must be considered in the rollout of any new educational initiative. Experts suggest that faculty development for CBME should incorporate educational activities using multiple teaching and delivery methods, and should be offered longitudinally through the planning, development, and implementation phases of curriculum change. Additionally, these educational activities must continue until all faculty have demonstrated an acceptable level of competence. Faculty buy-in is paramount to the successful delivery of any faculty development program that is not mandatory in nature.

[Competency-based Neurosurgery Residency Programme]

A programme proposal for competency-based Neurosurgery training adapted to the specialization project is presented. This proposal has been developed by a group of neurosurgeons commissioned by the SENEC (Spanish Society of Neurosurgery) and could be modified to generate a final version that could come into force coinciding with the implementation of the specialization programme. This document aims to facilitate the test of the new programme included in the online version of our journal.

DURATION OF THE PROGRAMME

Total training period is 6 years; initial 2 years belong to the surgery specialization and remaining 4 years belong to core specialty period.

STRUCTURE OF THE PROGRAMME

It is a competency-based programmed based on the map used by the US Accreditation Council for Graduate Medical Education (ACGME) including the following domains of clinical competency: Medical knowledge, patient care, communication skills, professionalism, practice-based learning and improvement, health systems, interprofessional collaboration and professional and personal development. Subcompetencies map in the domains of Knowledge and Patient care (including surgical competencies) was adapted to the one proposed by AANS and CNS (annex 1 of the programme). A subcompetency map was also used for the specialization rotations.

INSTRUCTION METHODS

Resident's training is based on personal study (self-learning) supported by efficient use of information sources and supervised clinical practice, including bioethical instruction, clinical management, research and learning techniques.

EVALUATION METHODS

Resident evaluation proposal includes, among other instruments, theoretical knowledge tests, objective and structured evaluation of the level of clinical competency with real or standardised patients, global competency scales, 360-degree evaluation, clinical record audits, milestones for residents progress and self-assessment (annex 2). Besides, residents periodically assess the teaching commitment of the department's neurosurgeons and other professors participating in rotations, and annually assess the overall operation of the programme. Results of evaluations are registered, together with other relevant data, in the Resident's Book.

PROGRAMME'S NATIONAL COMMITTEE

The creation of a Programme Committee directly attached to the SENEC (National Commission) that, aside from generating a final version of the programme, monitors its implementation (level of adherence and operation in the different departments), assumes the creation of test banks and the centralized administration of knowledge tests (in the middle of the residency and/or at the end of it) and centralizes information collected by tutors that could be used for re-accreditation of the services, is proposed.

Implementation of competency-based medical education: are we addressing the concerns and challenges?

CONTEXT

Competency-based medical education (CBME) has emerged as a core strategy to educate and assess the next generation of physicians. Advantages of CBME include: a focus on outcomes and learner achievement; requirements for multifaceted assessment that embraces formative and summative approaches; support of a flexible, time-independent trajectory through the curriculum; and increased accountability to stakeholders with a shared set of expectations and a common language for education, assessment and regulation.

OBJECTIVES

Despite the advantages of CBME, numerous concerns and challenges to the implementation of CBME frameworks have been described, including: increased administrative requirements; the need for faculty development; the lack of models for flexible curricula, and inconsistencies in terms and definitions. Additionally, there are concerns about reductionist approaches to assessment in CBME, lack of good assessments for some competencies, and whether CBME frameworks include domains of current importance. This study will outline these issues and discuss the responses of the medical education community.

METHODS

The concerns and challenges expressed are primarily categorised as: (i) those related to practical, administrative and logistical challenges in implementing CBME frameworks, and (ii) those with more conceptual or theoretical bases. The responses of the education community to these issues are then summarised.

CONCLUSIONS

The education community has begun to address the challenges involved in implementing CBME. Models and guidance exist to inform implementation strategies across the continuum of education, and focus on the more efficient use of resources and technology, and the use of milestones and entrustable professional activities-based frameworks. Inconsistencies in CBME definitions and frameworks remain a significant obstacle. Evolution in assessment approaches from in vitro task-based methods to in vivo integrated approaches is responsive to many of the theoretical and conceptual concerns about CBME, but much work remains to be done to bring rigour and quality to work-based assessment.

Examiners and content and site: Oh My! A national organization's investigation of score variation in large-scale performance assessments

Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada's National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.

Development of the Canadian Physiotherapy Assessment of Clinical Performance: A New Tool to Assess Physiotherapy Students' Performance in Clinical Education

PURPOSE

To develop the first draft of a Canadian tool to assess physiotherapy (PT) students' performance in clinical education (CE). Phase 1: to gain consensus on the items within the new tool, the number and placement of the comment boxes, and the rating scale; Phase 2: to explore the face and content validity of the draft tool.

METHODS

Phase 1 used the Delphi method; Phase 2 used cognitive interviewing methods with recent graduates and clinical instructors (CIs) and detailed interviews with clinical education and measurement experts.

RESULTS

Consensus was reached on the first draft of the new tool by round 3 of the Delphi process, which was completed by 21 participants. Interviews were completed with 13 CIs, 6 recent graduates, and 7 experts. Recent graduates and CIs were able to interpret the tool accurately, felt they could apply it to a recent CE experience, and provided suggestions to improve the draft. Experts provided salient advice.

CONCLUSIONS

The first draft of a new tool to assess PT students in CE, the Canadian Physiotherapy Assessment of Clinical Performance (ACP), was developed and will undergo further development and testing, including national consultation with stakeholders. Data from Phase 2 will contribute to developing an online education module for CIs and students.

Breaking bad news is a teachable skill in pediatric residents: A feasibility study of an educational intervention

OBJECTIVE

Patients and physicians identify communication of bad news as a skill in need of improvement. Our objectives were to measure change in performance of first-year pediatric residents in the delivery of bad news after an educational intervention and to measure if changes in performance were sustained over time.

METHODS

Communication skills of 29 residents were assessed via videotaped standardized patient (SP) encounters at 3 time points: baseline, immediately post-intervention, and 3 months post-intervention. Educational intervention used was the previously published "GRIEV_ING Death Notification Protocol."

RESULTS

The intraclass correlation coefficient demonstrated substantial inter-rater agreement with the assessment tool. Performance scores significantly improved from baseline to immediate post-intervention. Performance at 3 months post-intervention showed no change in two subscales and small improvement in one subscale.

CONCLUSIONS

We concluded that breaking bad news is a complex and teachable skill that can be developed in pediatric residents. Improvement was sustained over time, indicating the utility of this educational intervention.

PRACTICE IMPLICATIONS

This study brings attention to the need for improved communication training, and the feasibility of an education intervention in a large training program. Further work in development of comprehensive communication curricula is necessary in pediatric graduate medical education programs.

Concurrent Validity Between a Shared Curriculum, the Internal Medicine In-Training Examination, and the American Board of Internal Medicine Certifying Examination

BACKGROUND

A core objective of residency education is to facilitate learning, and programs need more curricula and assessment tools with demonstrated validity evidence.

OBJECTIVE

We sought to demonstrate concurrent validity between performance on a widely shared, ambulatory curriculum (the Johns Hopkins Internal Medicine Curriculum), the Internal Medicine In-Training Examination (IM-ITE), and the American Board of Internal Medicine Certifying Examination (ABIM-CE).

METHODS

A cohort study of 443 postgraduate year (PGY)-3 residents at 22 academic and community hospital internal medicine residency programs using the curriculum through the Johns Hopkins Internet Learning Center (ILC). Total and percentile rank scores on ILC didactic modules were compared with total and percentile rank scores on the IM-ITE and total scores on the ABIM-CE.

RESULTS

The average score on didactic modules was 80.1%; the percentile rank was 53.8. The average IM-ITE score was 64.1% with a percentile rank of 54.8. The average score on the ABIM-CE was 464. Scores on the didactic modules, IM-ITE, and ABIM-CE correlated with each other (P < .05). Residents completing greater numbers of didactic modules, regardless of scores, had higher IM-ITE total and percentile rank scores (P < .05). Resident performance on modules covering back pain, hypertension, preoperative evaluation, and upper respiratory tract infection was associated with IM-ITE percentile rank.

CONCLUSIONS

Performance on a widely shared ambulatory curriculum is associated with performance on the IM-ITE and the ABIM-CE.

Neuropathology training worldwide-evolution and comparisons

Training of neuropathologists varies worldwide. Systems range from highly organized specialist and subspecialist education with national certification, to regulated training with diploma recognition, to informal apprenticeships in neurological hospitals and no formal recognition. This overview compiles and summarizes the history of regulated training systems, the status of neuropathology within various countries' medical systems and the manner in which neuropathologists are trained. Anecdotal evidence suggests that countries with regulated systems of neuropathology training and an active professional organization are more likely to have an adequate supply of diagnostic specialists and a vibrant research community. The different training systems reflect the style of medical services delivery in the respective countries. In general, the existence of formal neuropathology training systems occurs only in countries with relatively high levels of per capita health expenditures, reflecting the development of medical specialization overall. Evolving diagnostic technologies and major international research endeavors, whose goals are to understand structure and function of the human brain, demand that neuropathology training is more than simply diagnostic histopathology.

Teaching and Assessing Colorectal Surgery Residents in the Age of ACGME Competencies: Pieces of the Whole

Educators have struggled with teaching and evaluation of the six Accreditation Council for Graduate Medical Education (ACGME) core competencies since their introduction in 1999. In addition, many authors have questioned the construct validity of the competencies. Concern has also arisen regarding the educational effects of the competencies and the subsequent limitation of resident duty hours, the combination of which have forced unprecedented changes in American graduate medical education. This article attempts to present an understanding of how these events have had direct and indirect effects on the education of residents in colon and rectal surgery, and to provide a framework for educators in colon and rectal surgery to adapt in their curricula.

Workplace-based assessment: raters' performance theories and constructs

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

Exploring the impact of mental workload on rater-based assessments

When appraising the performance of others, assessors must acquire relevant information and process it in a meaningful way in order to translate it effectively into ratings, comments, or judgments about how well the performance meets appropriate standards. Rater-based assessment strategies in health professional education, including scale and faculty development strategies aimed at improving them have generally been implemented with limited consideration of human cognitive and perceptual limitations. However, the extent to which the task assigned to raters aligns with their cognitive and perceptual capacities will determine the extent to which reliance on human judgment threatens assessment quality. It is well recognized in medical decision making that, as the amount of information to be processed increases, judges may engage mental shortcuts through the application of schemas, heuristics, or the adoption of solutions that satisfy rather than optimize the judge's needs. Further, these shortcuts may fundamentally limit/bias the information perceived or processed. Thinking of the challenges inherent in rater-based assessments in an analogous way may yield novel insights regarding the limits of rater-based assessment and may point to greater understanding of ways in which raters can be supported to facilitate sound judgment. This paper presents an initial exploration of various cognitive and perceptual limitations associated with rater-based assessment tasks. We hope to highlight how the inherent cognitive architecture of raters might beneficially be taken into account when designing rater-based assessment protocols.

History and practice of competency-based assessment

CONTEXT

In response to historical trends in expectations of doctors, the goals of medical education are increasingly framed in terms of global competencies. The language of these competencies has tended to adopt a prescriptive, rather than descriptive, approach. However, despite widespread agreement on the importance of competency-based education and more than two decades of study, this effort has not generated a dependable set of assessment tools.

DISCUSSION

Because models of competency are legislated, rather than shaped by scholarly consideration of empirical data, it is unlikely that such models directly reflect actual human behaviour. Efforts to measure clinical behaviours could benefit from increased clarity in three related conceptual areas. Firstly, the language of educational constructs should be framed in terms of data-based hypotheses, rather than in terms of intuitively plausible abilities. Secondly, these constructs should be specified in terms of the situations to which they are relevant, rather than as global personal characteristics. Finally, the resources required to measure these constructs should be rigorously established because a common resource-based metric would allow for rational selection of assessment methods. Specific methods to establish each of these objectives are discussed.

CONCLUSIONS

The political process of negotiating educational objectives should not be confused with the scientific work of establishing coherent and interpretable patterns of behaviour. Although the two activities can complement one another, each has its own distinct methods and style of discourse. It is thus critical to maintain boundaries between these two approaches to defining professional performance.

Assessing a method to limit influence of standardized tests on clerkship grades

BACKGROUND

Grading committees give excessive weight to standardized-examination scores.

PURPOSE

Understanding that biases are often ingrained in grading processes, we sought to assess the influence of a structured grading policy in limiting this effect.

METHODS

All 7 clerkship grading committees derived students' clinical scores while blinded to examination scores. Scores were combined to yield a final rank order, which was used to derive grade cutoffs. Logit regression was performed to assess the contribution of clinical and examination scores to final grades. Results were compared to a similar analysis where committees were not blinded to examination scores.

RESULTS

In contrast to prior findings, grading committees consistently assigned greater weight to clinical-performance scores in assigning final grades when blinded to examination scores.

CONCLUSIONS

Grading committees may be unaware of the extent to which they discount clinical assessments when they are at odds with the results of standardized examinations. This can be addressed with a procedure that blinds grading committees to examination scores.

Rater-based assessments as social judgments: rethinking the etiology of rater errors

BACKGROUND

Measurement errors are a limitation of using rater-based assessments that are commonly attributed to rater errors. Solutions targeting rater subjectivity have been largely unsuccessful.

METHOD

This critical review examines investigations of rater idiosyncrasy from impression formation literatures to ask new questions for the parallel problem in rater-based assessments.

RESULTS

Raters may form categorical judgments about ratees as part of impression formation. Although categorization can be idiosyncratic, raters tend to consistently construct one of a few possible interpretations of each ratee. If raters naturally form categorical judgments, an assessment system requiring ordinal or interval ratings may inadvertently introduce conversion errors due to translation techniques unique to each rater.

CONCLUSIONS

Potential implications of raters forming differing categorizations of ratees combined with the use of rating scales to collect categorical judgments on measurement outcomes in rater-based assessments are explored.

Demystifying the Millennial student: a reassessment in measures of character and engagement in professional education

The characteristic profile of Millennial Generation students, driving many educational reforms, can be challenged by research in a number of fields including cognition, learning style, neurology, and psychology. This evidence suggests that the current aggregate view of the Millennial student may be less than accurate. Statistics show that Millennial students are considerably diverse in backgrounds, personalities, and learning styles. Data are presented regarding technological predilection, multitasking, reading, critical thinking, professional behaviors, and learning styles, which indicate that students in the Millennial Generation may not be as homogenous in fundamental learning strategies and attitudes as is regularly proposed. Although their common character traits have implications for instruction, no available evidence demonstrates that these traits impact their fundamental process of learning. Many curricular strategies have been implemented to address alleged changes in the manner by which Millennial students learn. None has clearly shown superior outcomes in academic accomplishments or developing expertise for graduating students and concerns persist related to the successful engagement of Millennial students in the process of learning. Four factors for consideration in general curricular design are proposed to address student engagement and optimal knowledge acquisition for 21st century learners.

Duty hour recommendations and implications for meeting the ACGME core competencies: views of residency directors

OBJECTIVE

To describe the views of residency program directors regarding the effect of the 2010 duty hour recommendations on the 6 core competencies of graduate medical education.

METHODS

US residency program directors in internal medicine, pediatrics, and general surgery were e-mailed a survey from July 8 through July 20, 2010, after the 2010 Accreditation Council for Graduate Medical Education (ACGME) duty hour recommendations were published. Directors were asked to rate the implications of the new recommendations for the 6 ACGME core competencies as well as for continuity of inpatient care and resident fatigue.

RESULTS

Of 719 eligible program directors, 464 (65%) responded. Most program directors believe that the new ACGME recommendations will decrease residents' continuity with hospitalized patients (404/464 [87%]) and will not change (303/464 [65%]) or will increase (26/464 [6%]) resident fatigue. Additionally, most program directors (249-363/464 [53%-78%]) believe that the new duty hour restrictions will decrease residents' ability to develop competency in 5 of the 6 core areas. Surgery directors were more likely than internal medicine directors to believe that the ACGME recommendations will decrease residents' competency in patient care (odds ratio [OR], 3.9; 95% confidence interval [CI], 2.5-6.3), medical knowledge (OR, 1.9; 95% CI, 1.2-3.2), practice-based learning and improvement (OR, 2.7; 95% CI, 1.7-4.4), interpersonal and communication skills (OR, 1.9; 95% CI, 1.2-3.0), and professionalism (OR, 2.5; 95% CI, 1.5-4.0).

CONCLUSION

Residency program directors' reactions to ACGME duty hour recommendations demonstrate a marked degree of concern about educating a competent generation of future physicians in the face of increasing duty hour standards and regulation.