1
|
Birman NA, Vashdi DR, Miller-Mor Atias R, Riskin A, Zangen S, Litmanovitz I, Sagi D. Unveiling the paradoxes of implementing post graduate competency based medical education programs. MEDICAL TEACHER 2024:1-8. [PMID: 38803298 DOI: 10.1080/0142159x.2024.2356826] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2023] [Accepted: 05/14/2024] [Indexed: 05/29/2024]
Abstract
PURPOSE Competency-based medical education (CBME) has gained prominence as an innovative model for post-graduate medical education, yet its implementation poses significant challenges, especially with regard to its sustainability. Drawing on paradox theory, we suggest that revealing the paradoxes underlying these challenges may contribute to our understanding of post graduate competency-based medical education (PGCBME) implementation processes and serve as a first-step in enhancing better implementation. Thus, the purpose of the current study is to identify the paradoxes associated with PGCBME implementation. METHOD A qualitative study was conducted, as part of a larger action research, using in-depth semi-structured interviews with fellows and educators in eight Neonatal wards. RESULTS Analysis revealed that the PGCBME program examined in this study involves three different levels of standardization, each serving as one side of paradoxical tensions; (1) a paradox between the need for standardized assessment tools and for free-flow flexible assessment tools, (2) a paradox between the need for a standardized implementation process across all wards and the need for unique implementation protocols in each ward; and 3) a paradox between the need for a standardized meaning of competency proficiency and the need for flexible and personal competency achievement indicators. CONCLUSIONS Implementing PGCBME programs involves many challenges, some of which are paradoxical, i.e. two contradictory challenges in which solving one challenge exacerbates another. Revealing these paradoxes is important in navigating them successfully.
Collapse
Affiliation(s)
- Noa A Birman
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Dana R Vashdi
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Rotem Miller-Mor Atias
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Arieh Riskin
- Technion Israel Institute of Technology, The Ruth and Bruce Rappaport Faculty of Medicine, Haifa, Israel
| | - Shmuel Zangen
- Ben- Gurion University of the Negev, Faculty of Health Sciences, Be'er-Sheva, Israel
| | - Ita Litmanovitz
- Tel Aviv University, Faculty of Medicine & Health Sciences, Tel-Aviv, Israel
| | - Doron Sagi
- The Israel Center for Medical Simulation, Sheba Medical Center, Tel-Hashomer, Ramat-Gan, Israel
| |
Collapse
|
2
|
van der Gulden R, Thoonen BPA, Heeneman S, Muris JWM, Sagasser MH, Timmerman AA, Scherpbier-de Haan ND. How to use polarity thinking TM to manage tensions between accountability and learner agency when using a multipurpose portfolio? EDUCATION FOR PRIMARY CARE 2024:1-10. [PMID: 38762767 DOI: 10.1080/14739879.2024.2335610] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2023] [Accepted: 03/24/2024] [Indexed: 05/20/2024]
Abstract
Portfolios are often implemented to target multiple purposes, e.g. assessment, accountability and/or self-regulated learning. However, in educational practice, it appears to be difficult to combine different purposes in one portfolio, as interdependencies between the purposes can cause tensions. This paper explored directions to manage tensions that are inextricably linked to multipurpose portfolio use. We used a systems thinking methodology, that was based on the polarity thinkingTM framework. This framework provides a step-by-step approach to chart a polarity map® that can help to balance the tensions present in specific settings. We followed the steps of the framework to chart a polarity map for multipurpose portfolio use. Based on literature and our prior research, we selected one overarching polarity: accountability and learner agency. This polarity seems responsible for multiple tensions related to multipurpose portfolio use. We formulated values (potential benefits) and fears (tensions that can arise) of the two poles of this polarity. Then, we organised a session with stakeholders who work with the portfolio of the Dutch General Practice speciality programme. Together we formulated action steps and early warnings that can help to balance accountability and learner agency during multipurpose portfolio use. In addition to previous recommendations concerning portfolio use, we advocate that it is important to create a shared frame of reference between all involved with the multipurpose portfolio. During this process, the acknowledgement and discussion of tensions related to multipurpose portfolio use are vital.
Collapse
Affiliation(s)
- R van der Gulden
- Department of Primary and Community Care, Radboud University Medical Center, Nijmegen, the Netherlands
| | - B P A Thoonen
- Department of Primary and Community Care, Radboud University Medical Center, Nijmegen, the Netherlands
| | - S Heeneman
- Department of Pathology, Maastricht University, Maastricht, the Netherlands
| | - J W M Muris
- Department of Family Medicine, Maastricht University, Maastricht, the Netherlands
| | - M H Sagasser
- Network of GP Specialty Training Institutes, Utrecht, the Netherlands
| | - A A Timmerman
- Department of Family Medicine, Maastricht University, Maastricht, the Netherlands
| | - N D Scherpbier-de Haan
- Department of General Practice and Elderly Care Medicine, University Medical Centre Groningen, Groningen, the Netherlands
| |
Collapse
|
3
|
Braund H, Dalgarno N, O’Dell R, Taylor DR. Making assessment a team sport: a qualitative study of facilitated group feedback in internal medicine residency. CANADIAN MEDICAL EDUCATION JOURNAL 2024; 15:14-26. [PMID: 38827914 PMCID: PMC11139793 DOI: 10.36834/cmej.75250] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/05/2024]
Abstract
Purpose Competency-based medical education relies on feedback from workplace-based assessment (WBA) to direct learning. Unfortunately, WBAs often lack rich narrative feedback and show bias towards Medical Expert aspects of care. Building on research examining interactive assessment approaches, the Queen's University Internal Medicine residency program introduced a facilitated, team-based assessment initiative ("Feedback Fridays") in July 2017, aimed at improving holistic assessment of resident performance on the inpatient medicine teaching units. In this study, we aim to explore how Feedback Fridays contributed to formative assessment of Internal Medicine residents within our current model of competency-based training. Method A total of 53 residents participated in facilitated, biweekly group assessment sessions during the 2017 and 2018 academic year. Each session was a 30-minute facilitated assessment discussion done with one inpatient team, which included medical students, residents, and their supervising attending. Feedback from the discussion was collected, summarized, and documented in narrative form in electronic WBA forms by the program's assessment officer for the residents. For research purposes, verbatim transcripts of feedback sessions were analyzed thematically. Results The researchers identified four major themes for feedback: communication, intra- and inter-personal awareness, leadership and teamwork, and learning opportunities. Although feedback related to a broad range of activities, it showed strong emphasis on competencies within the intrinsic CanMEDS roles. Additionally, a clear formative focus in the feedback was another important finding. Conclusions The introduction of facilitated team-based assessment in the Queen's Internal Medicine program filled an important gap in WBA by providing learners with detailed feedback across all CanMEDS roles and by providing constructive recommendations for identified areas for improvement.
Collapse
Affiliation(s)
- Heather Braund
- Office of Professional Development and Educational Scholarship, Ontario, Canada
- Faculty of Education, Queen’s University, Ontario, Canada
| | - Nancy Dalgarno
- Office of Professional Development and Educational Scholarship, Ontario, Canada
- Department of Biomedical and Molecular Sciences, Faculty of Health Sciences, Queen’s University, Ontario, Canada
| | - Rachel O’Dell
- Department of Internal Medicine, Faculty of Health Sciences, Queen’s University, Ontario, Canada
| | - David R Taylor
- Academy for Teachers and Educators, Department of Medicine, Queen’s University, Ontario, Canada
| |
Collapse
|
4
|
Cady P, Heykoop C. Strengthening health system leadership in practice. Healthc Manage Forum 2024; 37:128-132. [PMID: 37977152 PMCID: PMC11044504 DOI: 10.1177/08404704231209945] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2023]
Abstract
The field of health leadership is shifting rapidly, and there is an opportunity to learn with health leaders about what is needed to support health leadership education, research, and practice. In 2022, to augment student feedback and faculty praxis, Royal Roads University conducted 12 virtual interviews with senior health system leaders across various settings to learn how health leaders can better respond to emerging and future leadership needs and priorities facing health systems. Findings from this study informed the development of a health-specific elective for the Master of Arts in Leadership, Health Specialization program entitled Considerations for Health Systems Renewal. This elective explores the following topics that emerged from this research study: (1) an orientation to possibility; (2) emerging strategic human resource concerns; (3) healthcare innovation; (4) relational and social systems leadership; (5) polarity thinking; (6) trauma-informed leadership; and (7) Canadian healthcare networks. In this article, we share our research process and findings to arrive at these recommendations.
Collapse
Affiliation(s)
- Phil Cady
- Royal Roads University, Victoria, British Columbia, Canada
| | - Cheryl Heykoop
- Royal Roads University, Victoria, British Columbia, Canada
| |
Collapse
|
5
|
Vaccari E, Moonen-van Loon J, Van der Vleuten C, Hunt P, McManus B. Marking parties for marking written assessments: A spontaneous community of practice. MEDICAL TEACHER 2024; 46:573-579. [PMID: 37783205 DOI: 10.1080/0142159x.2023.2262102] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/04/2023]
Abstract
In programmes of assessment with both high and low-stakes assessments, the inclusion of open-ended long answer questions in the high-stakes examination can contribute to driving deeper learning among students. However, in larger institutions, this would generate a seemingly insurmountable marking workload. In this study, we use a focused ethnographic approach to explore how such a marking endeavour can be tackled efficiently and pragmatically. In marking parties, examiners come together to individually mark student papers. This study focuses on marking parties for two separate tasks assessing written clinical communication in medical school finals at Southampton, UK. Data collected included field notes from 21.3 h of marking parties, details of demographics and clinical and educational experience of examiners, examiners' written answers to an open-ended post-marking party questionnaire, an in-depth interview and details of the actual marks assigned during the marking parties. In a landscape of examiners who are busy clinicians and rarely interact with each other educationally, marking parties represent a spontaneous and sustainable community of practice, with functions extending beyond the mere marking of exams. These include benchmarking, learning, managing biases and exam development. Despite the intensity of the work, marking parties built camaraderie and were considered fun and motivating.
Collapse
Affiliation(s)
- Emma Vaccari
- Faculty of Medicine, University of Southampton, Southampton, UK
| | - Joyce Moonen-van Loon
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Cees Van der Vleuten
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Paula Hunt
- Faculty of Medicine, University of Southampton, Southampton, UK
| | - Bruce McManus
- Faculty of Medicine, University of Southampton, Southampton, UK
| |
Collapse
|
6
|
Poretti MG, Monti M, Fadda M. Exploring clinical teachers' beliefs about teaching in a newly established medical school in Southern Switzerland. BMC MEDICAL EDUCATION 2024; 24:330. [PMID: 38519900 PMCID: PMC10960412 DOI: 10.1186/s12909-024-05299-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Accepted: 03/12/2024] [Indexed: 03/25/2024]
Abstract
Academic social contexts and educational beliefs may influence teaching practices and teaching intentions. Insight into such beliefs represents an important source of information for medical schools to improve the quality of teaching and to guide content of faculty development programs. The aim of this study was to explore beliefs about teaching among prospective clinical teachers at a newly established medical school in Southern Switzerland and to estimate the relationship between these beliefs and intention to commit to teaching in the newly established medical curriculum using Fishbein's integrative model of behaviour prediction. We targeted a purposive sample of physicians working in hospital departments potentially involved in the clinical immersion of medical students enrolled in the program. We designed a cross-sectional quantitative study using an online questionnaire. The questionnaire included both items developed by the authors and items belonging to a previously validated questionnaire. Participants showed a high intention to commit to teaching in the newly established curriculum. We found that self-efficacy beliefs and two sets of behavioural beliefs, namely perceived importance and expectations, had a positive correlation with the intention to commit. On the other hand, we pointed out a number of normative beliefs that in the long run could hinder the maintenance of this commitment, which should be addressed both at the levels of both faculty development and institutional policy rules. Our study also highlights the importance of reinforcing teachers' perceived self-efficacy in providing clinical supervision, strengthening their perceived importance of the clinical supervision, and incentivising the commitment of teachers by ensuring they have protected time to devote to this activity.
Collapse
Affiliation(s)
- Marilù Guigli Poretti
- Ente Ospedaliero Cantonale, Area Formazione Medica e Ricerca, Direzione Generale, Lugano, Switzerland.
- Medical Education Unit, Faculty of Biomedical Sciences, Università della Svizzera italiana, Lugano, Switzerland.
| | - Matteo Monti
- Medical Education Unit, Faculty of Biology and Medicine, University of Lausanne, Lausanne, Switzerland
- Division of Internal Medicine, Lausanne University Hospital, Lausanne, Switzerland
| | - Marta Fadda
- Institute of Public Health, Faculty of Biomedical Sciences, Università della Svizzera italiana, Lugano, Switzerland
| |
Collapse
|
7
|
Braund H, Patel V, Dalgarno N, Mann S. Exploring residents' perceptions of competency-based medical education across Canada: A national survey study. MEDEDPUBLISH 2024; 14:2. [PMID: 38487752 PMCID: PMC10933567 DOI: 10.12688/mep.19247.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/20/2023] [Indexed: 03/17/2024] Open
Abstract
Background: As competency-based medical education (CBME) is implemented across Canada, little is known about residents' perceptions of this model. This study examined how Canadian residents understand CBME and their lived experiences with implementation. Methods: We administered a survey in 2018 with Likert-type and open-ended questions to 375 residents across Canada, of whom 270 were from traditional programs ("pre-CBME") and 105 were in a CBME program. We used the Mann-Whitney test to examine differences across samples, and analyzed qualitative data thematically. Results: Three themes were identified across both groups: program outcome concerns, changes, and emotional responses. In relation to program concerns, both groups were concerned about the administrative burden, challenges with the assessment process, and feedback quality. Only pre-CBME residents were concerned about faculty engagement and buy-in. In terms of changes, both groups discussed a more formalized assessment process with mixed reactions. Residents in the pre-CBME sample reported greater concerns for faculty time constraints, assessment completion, and quality of learning experiences, whilst those in CBME programs reported being more proactive in their learning and greater selfreflection. Residents expressed strong emotional narrative responses including greater stress and frustration in a CBME environment. Conclusion: Findings demonstrate that residents have mixed feelings and experiences regarding CBME. Their positive experiences align with the aim of developing more self-directed learners. However, the concerns suggest the need to address specific shortcomings to increase buy-in, while the emotional responses associated with CBME may require a cultural shift within residency programs to guard against burnout.
Collapse
Affiliation(s)
- Heather Braund
- Professional Development & Educational Scholarship, Queen's University, Kingston, Ontario, K7L 1B9,, Canada
| | - Vivesh Patel
- Faculty of Health Sciences, Queen's University, Kingston, Ontario, K7L 2Y1, Canada
| | - Nancy Dalgarno
- Professional Development & Educational Scholarship, Queen's University, Kingston, Ontario, K7L 1B9,, Canada
| | - Steve Mann
- Department of Surgery, Queen's University, Kingston, Ontario, K7L 2V7, Canada
| |
Collapse
|
8
|
Wood TJ, Daniels VJ, Pugh D, Touchie C, Halman S, Humphrey-Murto S. Implicit versus explicit first impressions in performance-based assessment: will raters overcome their first impressions when learner performance changes? ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023:10.1007/s10459-023-10302-2. [PMID: 38010576 DOI: 10.1007/s10459-023-10302-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Accepted: 11/12/2023] [Indexed: 11/29/2023]
Abstract
First impressions can influence rater-based judgments but their contribution to rater bias is unclear. Research suggests raters can overcome first impressions in experimental exam contexts with explicit first impressions, but these findings may not generalize to a workplace context with implicit first impressions. The study had two aims. First, to assess if first impressions affect raters' judgments when workplace performance changes. Second, whether explicitly stating these impressions affects subsequent ratings compared to implicitly-formed first impressions. Physician raters viewed six videos where learner performance either changed (Strong to Weak or Weak to Strong) or remained consistent. Raters were assigned two groups. Group one (n = 23, Explicit) made a first impression global rating (FIGR), then scored learners using the Mini-CEX. Group two (n = 22, Implicit) scored learners at the end of the video solely with the Mini-CEX. For the Explicit group, in the Strong to Weak condition, the FIGR (M = 5.94) was higher than the Mini-CEX Global rating (GR) (M = 3.02, p < .001). In the Weak to Strong condition, the FIGR (M = 2.44) was lower than the Mini-CEX GR (M = 3.96 p < .001). There was no difference between the FIGR and the Mini-CEX GR in the consistent condition (M = 6.61, M = 6.65 respectively, p = .84). There were no statistically significant differences in any of the conditions when comparing both groups' Mini-CEX GR. Therefore, raters adjusted their judgments based on the learners' performances. Furthermore, raters who made their first impressions explicit showed similar rater bias to raters who followed a more naturalistic process.
Collapse
Affiliation(s)
- Timothy J Wood
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada.
| | - Vijay J Daniels
- Department of Medicine, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Canada
| | - Debra Pugh
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
- Medical Council of Canada, Ottawa, Canada
| | - Claire Touchie
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
| | - Samantha Halman
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
| | - Susan Humphrey-Murto
- Faculty of Medicine, University of Ottawa, 850 Peter Morand Crescent, Ottawa, ON, K1G-5Z3, Canada
- Department of Medicine, The Ottawa Hospital, Ottawa, Canada
| |
Collapse
|
9
|
Pearce J, Chiavaroli N, Tavares W. On the use and abuse of metaphors in assessment. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:1333-1345. [PMID: 36729196 DOI: 10.1007/s10459-022-10203-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Accepted: 12/29/2022] [Indexed: 06/18/2023]
Abstract
This paper is motivated by a desire to advance assessment in the health professions through encouraging the judicious and productive use of metaphors. Through five specific examples (pixels, driving lesson/test, jury deliberations, signal processing, and assessment as a toolbox), we interrogate how metaphors are being used in assessment to consider what value they add to understanding and implementation of assessment practices. By unpacking these metaphors in action, we probe each metaphor's rationale and function, the gains each metaphor makes, and explore the unintended meanings they may carry. In summarizing common uses of metaphors, we elucidate how there may be both advantages and/or disadvantages. Metaphors can play important roles in simplifying, complexifying, communicating, translating, encouraging reflection, and convincing. They may be powerfully rhetorical, leading to intended consequences, actions, and other pragmatic outcomes. Although metaphors can be extremely helpful, they do not constitute thorough critique, justified evidence or argumentation. We argue that although metaphors have utility, they must be carefully considered if they are to serve assessment needs in intended ways. We should pay attention to how metaphors may be misinterpreted, what they ignore or unintentionally signal, and perhaps mitigate against this with anticipated corrections or nuanced qualifications. Failure to do so may lead to implementing practices that miss underlying and relevant complexities for assessment science and practice. Using metaphors requires careful attention with respect to their role, contributions, benefits and limitations. We highlight the value that comes from critiquing metaphors, and demonstrate the care required to ensure their continued utility.
Collapse
Affiliation(s)
- Jacob Pearce
- Tertiary Education, Australian Council for Educational Research, Camberwell, Australia.
| | - Neville Chiavaroli
- Tertiary Education, Australian Council for Educational Research, Camberwell, Australia
| | - Walter Tavares
- Department of Health and Society and Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON., Canada
| |
Collapse
|
10
|
Kusurkar RA, Orsini C, Somra S, Artino AR, Daelmans HE, Schoonmade LJ, van der Vleuten C. The Effect of Assessments on Student Motivation for Learning and Its Outcomes in Health Professions Education: A Review and Realist Synthesis. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:1083-1092. [PMID: 37146237 PMCID: PMC10453393 DOI: 10.1097/acm.0000000000005263] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/07/2023]
Abstract
PURPOSE In health professions education (HPE), the effect of assessments on student motivation for learning and its consequences have been largely neglected. This is problematic because assessments can hamper motivation and psychological well-being. The research questions guiding this review were: How do assessments affect student motivation for learning in HPE? What outcomes does this lead to in which contexts? METHOD In October 2020, the authors searched PubMed, Embase, APA PsycInfo, ERIC, CINAHL, and Web of Science Core Collection for "assessments" AND "motivation" AND "health professions education/students." Empirical papers or literature reviews investigating the effect of assessments on student motivation for learning in HPE using quantitative, qualitative, or mixed methods from January 1, 2010, to October 29, 2020, were included. The authors chose the realist synthesis method for data analysis to study the intended and unintended consequences of this complex topic. Assessments were identified as stimulating autonomous or controlled motivation using sensitizing concepts from self-determination theory and data on context-mechanism-outcome were extracted. RESULTS Twenty-four of 15,291 articles were ultimately included. Assessments stimulating controlled motivation seemed to have negative outcomes. An example of an assessment that stimulates controlled motivation is one that focuses on factual knowledge (context), which encourages studying only for the assessment (mechanism) and results in surface learning (outcome). Assessments stimulating autonomous motivation seemed to have positive outcomes. An example of an assessment that stimulates autonomous motivation is one that is fun (context), which through active learning (mechanism) leads to higher effort and better connection with the material (outcome). CONCLUSIONS These findings indicate that students strategically learned what was expected to appear in assessments at the expense of what was needed in practice. Therefore, health professions educators should rethink their assessment philosophy and practices and introduce assessments that are relevant to professional practice and stimulate genuine interest in the content.
Collapse
Affiliation(s)
- Rashmi A. Kusurkar
- R.A. Kusurkar is professor and research programme leader, Research in Education, Amsterdam University Medical Centers location Vrije Universiteit Amsterdam, professor and research programme leader, LEARN! Research Institute for Learning and Education, Faculty of Psychology and Education, VU University Amsterdam, and professor and research programme leader, Amsterdam Public Health, Quality of Care, Amsterdam, the Netherlands; ORCID: http://orcid.org/0000-0002-9382-0379
| | - Cesar Orsini
- C. Orsini is associate professor in medical education, Norwich Medical School, University of East Anglia, Norwich, United Kingdom, and Researcher in Health Professions Education, Faculty of Dentistry, Universidad de Los Andes, Santiago, Chile; ORCID: http://orcid.org/0000-0002-5226-3625
| | - Sunia Somra
- S. Somra was research assistant, Research in Education, Amsterdam University Medical Centers location Vrije Universiteit Amsterdam, Amsterdam, the Netherlands, at the time of this study
| | - Anthony R. Artino
- A.R. Artino Jr is professor and associate dean for evaluation and educational research, School of Medicine & Health Sciences, George Washington University, Washington, DC; ORCID: http://orcid.org/0000-0003-2661-7853
| | - Hester E.M. Daelmans
- H.E.M. Daelmans is director of the master of medicine programme, Faculty of Medicine Vrije Universiteit Amsterdam, Amsterdam, the Netherlands
| | - Linda J. Schoonmade
- L.J. Schoonmade is information specialist at the medical library, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands; ORCID: https://orcid.org/0000-0002-2407-5977
| | - Cees van der Vleuten
- C. van der Vleuten is professor, School of Health Professions Education, University of Maastricht, Maastricht, the Netherlands; ORCID: http://orcid.org/0000-0001-6802-3119
| |
Collapse
|
11
|
Holmboe ES, Osman NY, Murphy CM, Kogan JR. The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:S37-S49. [PMID: 37071705 DOI: 10.1097/acm.0000000000005251] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors' goal is to help improve and guide innovation in assessment by catalyzing further conversations.
Collapse
Affiliation(s)
- Eric S Holmboe
- E.S. Holmboe is chief, Research, Milestones Development and Evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0003-0108-6021
| | - Nora Y Osman
- N.Y. Osman is associate professor of medicine, Harvard Medical School, and director of undergraduate medical education, Brigham and Women's Hospital Department of Medicine, Boston, Massachusetts; ORCID: https://orcid.org/0000-0003-3542-1262
| | - Christina M Murphy
- C.M. Murphy is a fourth-year medical student and president, Medical Student Government at Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0003-3966-5264
| | - Jennifer R Kogan
- J.R. Kogan is associate dean, Student Success and Professional Development, and professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0001-8426-9506
| |
Collapse
|
12
|
Martin L, Blissett S, Johnston B, Tsang M, Gauthier S, Ahmed Z, Sibbald M. How workplace-based assessments guide learning in postgraduate education: A scoping review. MEDICAL EDUCATION 2023; 57:394-405. [PMID: 36286100 DOI: 10.1111/medu.14960] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Revised: 09/16/2022] [Accepted: 10/21/2022] [Indexed: 06/16/2023]
Abstract
INTRODUCTION Competency-based medical education (CBME) led to the widespread adoption of workplace-based assessment (WBA) with the promise of achieving assessment for learning. Despite this, studies have illustrated tensions between the summative and formative role of WBA which undermine learning goals. Models of workplace-based learning (WBL) provide insight, however, these models excluded WBA. This scoping review synthesizes the primary literature addressing the role of WBA to guide learning in postgraduate medical education, with the goal of identifying gaps to address in future studies. METHODS The search was applied to OVID Medline, Web of Science, ERIC and CINAHL databases, articles up to September 2020 were included. Titles and abstracts were screened by two reviewers, followed by a full text review. Two members independently extracted and analysed quantitative and qualitative data using a descriptive-analytic technique rooted in Billett's four premises of WBL. Themes were synthesized and discussed until consensus. RESULTS All 33 papers focused on the perception of learning through WBA. The majority applied qualitative methodology (70%), and 12 studies (36%) made explicit reference to theory. Aligning with Billett's first premise, results reinforce that learning always occurs in the workplace. WBA helped guide learning goals and enhanced feedback frequency and specificity. Billett's remaining premises provided an important lens to understand how tensions that existed in WBL have been exacerbated with frequent WBA. As individuals engage in both work and WBA, they are slowly transforming the workplace. Culture and context frame individual experiences and the perceived authenticity of WBA. Finally, individuals will have different goals, and learn different things, from the same experience. CONCLUSION Analysing WBA literature through the lens of WBL theory allows us to reframe previously described tensions. We propose that future studies attend to learning theory, and demonstrate alignment with philosophical position, to advance our understanding of assessment-for-learning in the workplace.
Collapse
Affiliation(s)
- Leslie Martin
- Department of Medicine, McMaster University, Hamilton, Ontario, Canada
| | - Sarah Blissett
- Department of Medicine, Western University, London, Ontario, Canada
| | - Bronte Johnston
- McMaster Education Research, Innovation, and Theory Program, McMaster University, Hamilton, Ontario, Canada
| | - Michael Tsang
- Department of Medicine, McMaster University, Hamilton, Ontario, Canada
| | - Stephen Gauthier
- Department of Medicine, Queens University, Kingston, Ontario, Canada
| | - Zeeshan Ahmed
- Department of Medicine, Ottawa University, Ottawa, Ontario, Canada
| | - Matthew Sibbald
- Department of Medicine, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
13
|
Schafer KR, Sood L, King CJ, Alexandraki I, Aronowitz P, Cohen M, Chretien K, Pahwa A, Shen E, Williams D, Hauer KE. The Grade Debate: Evidence, Knowledge Gaps, and Perspectives on Clerkship Assessment Across the UME to GME Continuum. Am J Med 2023; 136:394-398. [PMID: 36632923 DOI: 10.1016/j.amjmed.2023.01.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Accepted: 01/03/2023] [Indexed: 01/10/2023]
Affiliation(s)
- Katherine R Schafer
- Department of Internal Medicine, Wake Forest University School of Medicine, Winston-Salem, NC.
| | - Lonika Sood
- Elson S. Floyd College of Medicine, Washington State University, Spokane
| | - Christopher J King
- Division of Hospital Medicine, Department of Medicine, University of Colorado School of Medicine, Aurora
| | | | | | - Margot Cohen
- Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia
| | | | - Amit Pahwa
- Johns Hopkins University School of Medicine, Baltimore, Md
| | - E Shen
- Department of Internal Medicine, Wake Forest University School of Medicine, Winston-Salem, NC
| | - Donna Williams
- Department of Internal Medicine, Wake Forest University School of Medicine, Winston-Salem, NC
| | | |
Collapse
|
14
|
Westein MPD, Koster AS, Daelmans HEM, Bouvy ML, Kusurkar RA. How progress evaluations are used in postgraduate education with longitudinal supervisor-trainee relationships: a mixed method study. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:205-222. [PMID: 36094680 PMCID: PMC9992254 DOI: 10.1007/s10459-022-10153-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/30/2021] [Accepted: 08/07/2022] [Indexed: 06/15/2023]
Abstract
The combination of measuring performance and giving feedback creates tension between formative and summative purposes of progress evaluations and can be challenging for supervisors. There are conflicting perspectives and evidence on the effects supervisor-trainee relationships have on assessing performance. The aim of this study was to learn how progress evaluations are used in postgraduate education with longitudinal supervisor-trainee relationships. Progress evaluations in a two-year community-pharmacy specialization program were studied with a mixed-method approach. An adapted version of the Canadian Medical Education Directives for Specialists (CanMEDS) framework was used. Validity of the performance evaluation scores of 342 trainees was analyzed using repeated measures ANOVA. Semi-structured interviews were held with fifteen supervisors to investigate their response processes, the utility of the progress evaluations, and the influence of supervisor-trainee relationships. Time and CanMEDS roles affected the three-monthly progress evaluation scores. Interviews revealed that supervisors varied in their response processes. They were more committed to stimulating development than to scoring actual performance. Progress evaluations were utilized to discuss and give feedback on trainee development and to add structure to the learning process. A positive supervisor-trainee relationship was seen as the foundation for feedback and supervisors preferred the roles of educator, mentor, and coach over the role of assessor. We found that progress evaluations are a good method for directing feedback in longitudinal supervisor-trainee relationships. The reliability of scoring performance was low. We recommend progress evaluations to be independent of formal assessments in order to minimize roles-conflicts of supervisors.
Collapse
Affiliation(s)
- Marnix P D Westein
- Department of Pharmaceutical Sciences, Utrecht University, Universiteitsweg 99, 3584, CG, Utrecht, The Netherlands.
- Research in Education, Faculty of Medicine Vrije Universiteit, Amsterdam, The Netherlands.
- The Royal Dutch Pharmacists Association (KNMP), The Hague, The Netherlands.
| | - A S Koster
- Department of Pharmaceutical Sciences, Utrecht University, Universiteitsweg 99, 3584, CG, Utrecht, The Netherlands
| | - H E M Daelmans
- Programme Director Master of Medicine, Faculty of Medicine Vrije Universiteit, Amsterdam, The Netherlands
| | - M L Bouvy
- Department of Pharmaceutical Sciences, Utrecht University, Universiteitsweg 99, 3584, CG, Utrecht, The Netherlands
| | - R A Kusurkar
- Research in Education, Faculty of Medicine Vrije Universiteit, Amsterdam, The Netherlands
| |
Collapse
|
15
|
Hauer KE, Williams PM, Byerley JS, Swails JL, Barone MA. Blue Skies With Clouds: Envisioning the Future Ideal State and Identifying Ongoing Tensions in the UME-GME Transition. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:162-170. [PMID: 35947473 DOI: 10.1097/acm.0000000000004920] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The transition from medical school to residency in the United States consumes large amounts of time for students and educators in undergraduate and graduate medical education (UME, GME), and it is costly for both students and institutions. Attempts to improve the residency application and Match processes have been insufficient to counteract the very large number of applications to programs. To address these challenges, the Coalition for Physician Accountability charged the Undergraduate Medical Education to Graduate Medical Education Review Committee (UGRC) with crafting recommendations to improve the system for the UME-GME transition. To guide this work, the UGRC defined and sought stakeholder input on a "blue-skies" ideal state of this transition. The ideal state views the transition as a system to support a continuum of professional development and learning, thus serving learners, educators, and the public, and engendering trust among them. It also supports the well-being of learners and educators, promotes diversity, and minimizes bias. This manuscript uses polarity thinking to analyze 3 persistent key tensions in the system that require ongoing management. First, the formative purpose of assessment for learning and growth is at odds with the use of assessment data for ranking and sorting candidates. Second, the function of residents as learners can conflict with their role as workers contributing service to health care systems. Third, the current residency Match process can position the desire for individual choice-among students and their programs-against the workforce needs of the profession and the public. This Scholarly Perspective presents strategies to balance the upsides and downsides inherent to these tensions. By articulating the ideal state of the UME-GME transition and anticipating tensions, educators and educational organizations can be better positioned to implement UGRC recommendations to improve the transition system.
Collapse
Affiliation(s)
- Karen E Hauer
- K.E. Hauer is associate dean for competency assessment and professional standards and professor, Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California; ORCID: http://orcid.org/0000-0002-8812-4045
| | - Pamela M Williams
- P.M. Williams is associate dean for student affairs and professor, Department of Family Medicine, Uniformed Services University of the Health Sciences, F. Edward Hébert School of Medicine, Bethesda, Maryland
| | - Julie S Byerley
- J.S. Byerley was interim dean, University of North Carolina Adams School of Dentistry, and professor of pediatrics, University of North Carolina School of Medicine, Chapel Hill, North Carolina, at the time of writing and is now president and dean, Geisinger Commonwealth School of Medicine, Scranton, Pennsylvania
| | - Jennifer L Swails
- J.L. Swails is residency program director, codirector of interprofessional education, and associate professor, Department of Medicine, McGovern Medical School, University of Texas Health Science Center, Houston, Texas; ORCID: http://orcid.org/0000-0002-6102-831X
| | - Michael A Barone
- M.A. Barone is vice president, competency-based assessment, NBME, Philadelphia, Pennsylvania, and adjunct associate professor of pediatrics, Johns Hopkins University School of Medicine, Baltimore, Maryland; ORCID: http://orcid.org/0000-0002-4724-784X
| |
Collapse
|
16
|
Brauer S, Kettunen J, Levy A, Merenmies J, Kulmala P. The educational paradigm shift-a phenomenographic study of medical teachers' experiences of practices. BMC MEDICAL EDUCATION 2023; 23:29. [PMID: 36641444 PMCID: PMC9840161 DOI: 10.1186/s12909-023-04013-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Accepted: 01/09/2023] [Indexed: 06/17/2023]
Abstract
BACKGROUND This paper proposes a novel approach to the development of competence-oriented higher education, a national transformation aimed at harmonising and digitising undergraduate medical and dental education in Finland. METHODS We apply phenomenography as a viable qualitative method for medical education research. To better understand medical teachers' expectations towards the change in the educational paradigm, we need to study teachers' experiences of the current practices in undergraduate medical and dental education. The phenomenographic approach facilitates solid links between research, educational development, and change. RESULTS The phenomenographic study maps the qualitatively different ways in which medical teachers experience undergraduate medical and dental education practices. The answers reflect the changing educational paradigm in medical schools, suggesting practical implications for further development of medical and dental education and training. Core content analysis is preferred instructional scaffold for both teachers and students to prioritise the extensive medical education objectives. The change towards competence-based orientation is in progress and national co-operation accelerates its impact. CONCLUSION There is an obvious need to enrich the content of the current curriculum with national guidelines that aim for congruence in assessment and objectives. Our results suggest an assessment application for the theoretical concepts presented and promote the competence orientation of education throughout the curricula of medical and dental undergraduate education. Moreover, our results contribute to current European discourses on competence-based approaches in higher education. Up-to-date pedagogical faculty development programmes are a key prerequisite for teacher empowerment and future orientation in teaching and learning for healthcare professions.
Collapse
Affiliation(s)
- Sanna Brauer
- Faculty of Medicine, University of Oulu, Oulu, Finland
| | - Jaana Kettunen
- Finnish Institute for Educational Research, University of Jyväskylä, Jyväskylä, Finland
| | - Anna Levy
- Faculty of Medicine, University of Oulu, Oulu, Finland
| | - Jussi Merenmies
- University of Helsinki, Helsinki University Hospital, Helsinki, Finland
| | - Petri Kulmala
- Faculty of Medicine, University of Oulu, Oulu, Finland
| |
Collapse
|
17
|
Jamieson J, Gibson S, Hay M, Palermo C. Teacher, Gatekeeper, or Team Member: supervisor positioning in programmatic assessment. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2022:10.1007/s10459-022-10193-9. [PMID: 36469231 DOI: 10.1007/s10459-022-10193-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 11/27/2022] [Indexed: 06/17/2023]
Abstract
Competency-based assessment is undergoing an evolution with the popularisation of programmatic assessment. Fundamental to programmatic assessment are the attributes and buy-in of the people participating in the system. Our previous research revealed unspoken, yet influential, cultural and relationship dynamics that interact with programmatic assessment to influence success. Pulling at this thread, we conducted secondary analysis of focus groups and interviews (n = 44 supervisors) using the critical lens of Positioning Theory to explore how workplace supervisors experienced and perceived their positioning within programmatic assessment. We found that supervisors positioned themselves in two of three ways. First, supervisors universally positioned themselves as a Teacher, describing an inherent duty to educate students. Enactment of this position was dichotomous, with some supervisors ascribing a passive and disempowered position onto students while others empowered students by cultivating an egalitarian teaching relationship. Second, two mutually exclusive positions were described-either Gatekeeper or Team Member. Supervisors positioning themselves as Gatekeepers had a duty to protect the community and were vigilant to the detection of inadequate student performance. Programmatic assessment challenged this positioning by reorientating supervisor rights and duties which diminished their perceived authority and led to frustration and resistance. In contrast, Team Members enacted a right to make a valuable contribution to programmatic assessment and felt liberated from the burden of assessment, enabling them to assent power shifts towards students and the university. Identifying supervisor positions revealed how programmatic assessment challenged traditional structures and ideologies, impeding success, and provides insights into supporting supervisors in programmatic assessment.
Collapse
Affiliation(s)
- Janica Jamieson
- Monash University, Melbourne, Australia.
- School of Medical and Health Sciences, Edith Cowan University, 270 Joondalup Drive, Joondalup, WA, 6027, Australia.
| | | | | | | |
Collapse
|
18
|
Merritt C, Pusic M, Wolff M, Cico SJ, Santen SA. The Case for Core Competency and Competent Corps: Using Polarity Management to Illuminate Tensions in Training. J Grad Med Educ 2022; 14:650-654. [PMID: 36591425 PMCID: PMC9765911 DOI: 10.4300/jgme-d-22-00199.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Affiliation(s)
- Chris Merritt
- Chris Merritt, MD, MPH, MHPE, is Director, Medical Education Fellowship, Assistant Dean for Faculty Development, and Associate Professor of Emergency Medicine and Pediatrics, Alpert Medical School, Brown University
| | - Martin Pusic
- Martin Pusic, MD, MA, PhD, is Associate Physician, Division of Emergency Medicine, Boston Children's Hospital, and Associate Professor of Pediatrics and of Emergency Medicine, Harvard Medical School
| | - Margaret Wolff
- Margaret Wolff, MD, MHPE, is Associate Professor of Emergency Medicine and Pediatrics, University of Michigan Medical School
| | - Stephen J. Cico
- Stephen J. Cico, MD, MEd, is Assistant Dean for Graduate Medical Education, Designated Institutional Official, and Professor of Emergency Medicine, University of Central Florida College of Medicine
| | - Sally A. Santen
- Sally A. Santen, MD, PhD, is Senior Associate Dean of Assessment, Evaluation, and Scholarship and Professor of Emergency Medicine, Virginia Commonwealth University School of Medicine, and Professor, Departments of Emergency Medicine and Medical Education, University of Cincinnati
| |
Collapse
|
19
|
Tackett S, Steinert Y, Whitehead CR, Reed DA, Wright SM. Blind spots in medical education: how can we envision new possibilities? PERSPECTIVES ON MEDICAL EDUCATION 2022; 11:365-370. [PMID: 36417161 PMCID: PMC9684906 DOI: 10.1007/s40037-022-00730-y] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Revised: 09/07/2022] [Accepted: 09/15/2022] [Indexed: 05/29/2023]
Abstract
As human beings, we all have blind spots. Most obvious are our visual blind spots, such as where the optic nerve meets the retina and our inability to see behind us. It can be more difficult to acknowledge our other types of blind spots, like unexamined beliefs, assumptions, or biases. While each individual has blind spots, groups can share blind spots that limit change and innovation or even systematically disadvantage certain other groups. In this article, we provide a definition of blind spots in medical education, and offer examples, including unfamiliarity with the evidence and theory informing medical education, lack of evidence supporting well-accepted and influential practices, significant absences in our scholarly literature, and the failure to engage patients in curriculum development and reform. We argue that actively helping each other see blind spots may allow us to avoid pitfalls and take advantage of new opportunities for advancing medical education scholarship and practice. When we expand our collective field of vision, we can also envision more "adjacent possibilities," future states near enough to be considered but not so distant as to be unimaginable. For medical education to attend to its blind spots, there needs to be increased participation among all stakeholders and a commitment to acknowledging blind spots even when that may cause discomfort. Ultimately, the better we can see blind spots and imagine new possibilities, the more we will be able to adapt, innovate, and reform medical education to prepare and sustain a physician workforce that serves society's needs.
Collapse
Affiliation(s)
- Sean Tackett
- Division of General Internal Medicine, Johns Hopkins Bayview Medical Center, Baltimore, MD, USA.
| | - Yvonne Steinert
- Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada
| | - Cynthia R Whitehead
- Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, Canada
- Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Darcy A Reed
- Division of Community Internal Medicine, Mayo Clinic, Rochester, MN, USA
- College of Medicine and Science, Mayo Clinic, Rochester, MN, USA
| | - Scott M Wright
- Division of General Internal Medicine, Johns Hopkins Bayview Medical Center, Baltimore, MD, USA
| |
Collapse
|
20
|
Ryan MS, Holmboe ES, Chandra S. In Reply to Ibrahim and Abdel-Razig. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1581. [PMID: 36287721 DOI: 10.1097/acm.0000000000004923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Affiliation(s)
- Michael S Ryan
- Professor and vice chair of education, Department of Pediatrics, Children's Hospital of Richmond at Virginia Commonwealth University, Richmond, Virginia;
| | - Eric S Holmboe
- Chief research, milestone development, and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0003-0108-6021
| | - Subani Chandra
- Associate professor, vice chair of education, and residency program director, Department of Internal Medicine, Columbia University Vagelos College of Physicians and Surgeons, New York, New York; ORCID: https://orcid.org/0000-0001-5162-994X
| |
Collapse
|
21
|
Phinney LB, Fluet A, O'Brien BC, Seligman L, Hauer KE. Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1511-1520. [PMID: 35703235 DOI: 10.1097/acm.0000000000004774] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation. METHOD This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system. RESULTS Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning. CONCLUSIONS Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.
Collapse
Affiliation(s)
- Lauren B Phinney
- L.B. Phinney is a first-year internal medicine resident, Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California
| | - Angelina Fluet
- A. Fluet is a fourth-year medical student, University of California, San Francisco School of Medicine, San Francisco, California
| | - Bridget C O'Brien
- B.C. O'Brien is professor of medicine and education scientist, Department of Medicine and Center for Faculty Educators, University of California, San Francisco School of Medicine, San Francisco, California
| | - Lee Seligman
- L. Seligman is a second-year internal medicine resident, Department of Medicine, New York-Presbyterian Hospital, Columbia University Irving Medical Center, New York, New York
| | - Karen E Hauer
- K.E. Hauer is associate dean for competency assessment and professional standards and professor, Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California
| |
Collapse
|
22
|
Trier J, Turnnidge J, McGuire CS, Côté J, Dagnone JD. It's a 'two-way street': resident perspectives of effective coaching relationships in the clinical learning environment. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:5-12. [PMID: 35875451 PMCID: PMC9297249 DOI: 10.36834/cmej.72940] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
BACKGROUND Coaching has gained traction in postgraduate medical education to enhance interactions between residents and clinical teachers, but these relationships present unique challenges and tensions. In order to realize the promises of coaching in medical education, we must understand how coaching relationships can be enacted to optimize resident development. The purpose of this study was to explore residents' perceptions of key characteristics for effective clinical teacher-resident (CT-R) coaching relationships. METHODS We conducted four focus groups and eight interviews with residents at a Canadian academic center. Using a social constructionist approach, focus groups and interviews were audio recorded, transcribed verbatim, and thematically analyzed. RESULTS Residents described three main characteristics that contributed to effective CT-R coaching relationships: safe, meaningful, and collaborative. Residents emphasized that these characteristics needed to be bidirectional in nature to be most effective, in that both the resident and clinical teacher embodied these characteristics. CONCLUSIONS Residents identified that effective coaching relationships were shaped not only by clinical teacher behaviours, but importantly, the quality of the interpersonal relationship that was fostered. Thus, it is imperative to consider the bidirectional nature of the CT-R coaching relationship when striving to enhance resident development.
Collapse
Affiliation(s)
- Jessica Trier
- Department of Physical Medicine and Rehabilitation, Queen’s University, Ontario, Canada
- Providence Care Hospital, Ontario, Canada
| | - Jennifer Turnnidge
- Office of Professional Development and Educational Scholarship, Queen’s University, Ontario, Canada
- School of Kinesiology and Health Studies, Queen’s University, Ontario, Canada
| | - Cailie S McGuire
- School of Kinesiology and Health Studies, Queen’s University, Ontario, Canada
| | - Jean Côté
- School of Kinesiology and Health Studies, Queen’s University, Ontario, Canada
| | - J Damon Dagnone
- Department of Emergency Medicine, Queen’s University, Ontario, Canada
- Kingston Health Sciences Centre, Ontario, Canada
| |
Collapse
|
23
|
Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments. J Gen Intern Med 2022; 37:2200-2207. [PMID: 35710663 PMCID: PMC9296736 DOI: 10.1007/s11606-022-07509-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Accepted: 03/24/2022] [Indexed: 10/18/2022]
Abstract
BACKGROUND Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding. OBJECTIVE In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks. DESIGN Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel. PARTICIPANTS A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor. MAIN MEASURES Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors. KEY RESULTS IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty. CONCLUSIONS To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.
Collapse
|
24
|
Westein MPD, Koster AS, Daelmans HEM, Collares CF, Bouvy ML, Kusurkar RA. Validity evidence for summative performance evaluations in postgraduate community pharmacy education. CURRENTS IN PHARMACY TEACHING & LEARNING 2022; 14:701-711. [PMID: 35809899 DOI: 10.1016/j.cptl.2022.06.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/15/2021] [Revised: 05/30/2022] [Accepted: 06/09/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework. METHODS The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process. RESULTS A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director. CONCLUSIONS The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.
Collapse
Affiliation(s)
- Marnix P D Westein
- Department of Pharmaceutical Sciences, Utrecht University, Royal Dutch Pharmacists Association (KNMP), Research in Education, Faculty of Medicine Vrije Universiteit, Amsterdam, the Netherlands.
| | - Andries S Koster
- Department of Pharmaceutical Sciences, Utrecht University, Utrecht, the Netherlands.
| | - Hester E M Daelmans
- Master's programme of Medicine, Faculty of Medicine Vrije Universiteit, Amsterdam, the Netherlands.
| | - Carlos F Collares
- Maastricht University Faculty of Health Medicine and Life Sciences, Maastricht, the Netherlands.
| | - Marcel L Bouvy
- Department of Pharmaceutical Sciences, Utrecht University, Utrecht, the Netherlands.
| | - Rashmi A Kusurkar
- Research in Education, Faculty of Medicine Vrije Universiteit, Amsterdam, the Netherlands.
| |
Collapse
|
25
|
Khalife R, Gupta M, Gonsalves C, Park YS, Riddle J, Tekian A, Horsley T. Patient involvement in assessment of postgraduate medical learners: A scoping review. MEDICAL EDUCATION 2022; 56:602-613. [PMID: 34981565 DOI: 10.1111/medu.14726] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/03/2021] [Revised: 11/18/2021] [Accepted: 12/27/2021] [Indexed: 06/14/2023]
Abstract
CONTEXT Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent. METHODS We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment. RESULTS We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise. CONCLUSION Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.
Collapse
Affiliation(s)
- Roy Khalife
- Department of Medicine (Hematology), The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada
| | - Manika Gupta
- Department of Medicine (Hematology), The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada
| | - Carol Gonsalves
- Department of Medicine (Hematology), The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada
| | - Yoon Soo Park
- Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Janet Riddle
- Department of Medical Education, University of Illinois College of Medicine at Chicago, Chicago, Illinois, USA
| | - Ara Tekian
- Department of Medical Education, University of Illinois College of Medicine at Chicago, Chicago, Illinois, USA
| | - Tanya Horsley
- Research Unit, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada
| |
Collapse
|
26
|
Valentine N, Durning SJ, Shanahan EM, van der Vleuten C, Schuwirth L. The pursuit of fairness in assessment: Looking beyond the objective. MEDICAL TEACHER 2022; 44:353-359. [PMID: 35104191 DOI: 10.1080/0142159x.2022.2031943] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.
Collapse
Affiliation(s)
- Nyoli Valentine
- Prideaux Discipline of Clinical Education, Flinders University, Bedford Park, Australia
| | - Steven J Durning
- Center for Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
| | | | - Cees van der Vleuten
- Department of Educational Development and Research, Maastricht University, Maastricht, Netherlands
| | - Lambert Schuwirth
- Prideaux Discipline of Clinical Education, Flinders University, Bedford Park, Australia
| |
Collapse
|
27
|
Deng YM, Wu HW, Liao HE. Utilization Intention of Community Pharmacy Service under the Dual Threats of Air Pollution and COVID-19 Epidemic: Moderating Effects of Knowledge and Attitude toward COVID-19. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:ijerph19063744. [PMID: 35329431 PMCID: PMC8954536 DOI: 10.3390/ijerph19063744] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Revised: 03/09/2022] [Accepted: 03/16/2022] [Indexed: 02/05/2023]
Abstract
The utilization of pharmacy services in response to the threat of COVID-19 infection remains unclear in areas suffering from air pollution, and little is known regarding the effects of knowledge and attitude (KA) toward COVID-19 on this preventive behavior. This study aimed to explore how the residents perceived and reacted to the new threats of the epidemic and how KA may affect the correlation. Based on the health belief model (HBM), this research took the pharmacy service utilization (PSU) as an example to explain the preventive behavior. The samples were 375 respondents recruited from five districts near the industrial parks. T-test, ANOVA, and regression analyses of SPSS 22.0 were used to analyze the data. Test results show that self-efficacy was the strongest predictor, followed by the net perceived benefit. KA moderated the association of perceived threat and PSU intention. The levels of air pollution of a district may not be a good predictor for the preventive behavior against COVID-19.
Collapse
Affiliation(s)
- Yueen-Mei Deng
- Department of Healthcare Management, Asia University, No.500, Lioufeng Rd., Taichung 41354, Taiwan
- Correspondence: (Y.-M.D.); (H.-E.L.); Tel.: +886-919-038978 (Y.-M.D.)
| | - Hong-Wei Wu
- Department of Pharmacy, Tajen University, No.2, Wexin Rd., Yampu 906, Taiwan;
| | - Hung-En Liao
- Department of Healthcare Management, Asia University, No.500, Lioufeng Rd., Taichung 41354, Taiwan
- Correspondence: (Y.-M.D.); (H.-E.L.); Tel.: +886-919-038978 (Y.-M.D.)
| |
Collapse
|
28
|
Castanelli DJ, Weller JM, Molloy E, Bearman M. Trust, power and learning in workplace-based assessment: The trainee perspective. MEDICAL EDUCATION 2022; 56:280-291. [PMID: 34433230 PMCID: PMC9292503 DOI: 10.1111/medu.14631] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/01/2021] [Revised: 08/09/2021] [Accepted: 08/17/2021] [Indexed: 05/22/2023]
Abstract
For trainees to participate meaningfully in workplace-based assessment (WBA), they must have trust in their assessor. However, the trainee's dependent position complicates such trust. Understanding how power and trust influence WBAs may help us make them more effective learning opportunities. We conducted semi-structured interviews with 17 postgraduate anaesthesia trainees across Australia and New Zealand. Sensitised by notions of power, we used constructivist grounded theory methodology to examine trainees' experiences with trusting their supervisors in WBAs. In our trainee accounts, we found that supervisors held significant power to mediate access to learning opportunities and influence trainee progress in training. All episodes where supervisors could observe trainees, from simply working together to formal WBAs, were seen to generate assessment information with potential consequences. In response, trainees actively acquiesced to a deferential role, which helped them access desirable expertise and minimise the risk of reputational harm. Trainees granted trust based on how they anticipated a supervisor would use the power inherent in their role. Trainees learned to ration exposure of their authentic practice to supervisors in proportion to their trust in them. Trainees were more trusting and open to learning when supervisors used their power for the trainee's benefit and avoided WBAs with supervisors they perceived as less trustworthy. If assessment for learning is to flourish, then the trainee-supervisor power dynamic must evolve. Enhancing supervisor behaviour through reflection and professional development to better reward trainee trust would invite more trainee participation in assessment for learning. Modifying the assessment system design to nudge the power balance towards the trainee may also help. Modifications could include designated formative and summative assessments or empowering trainees to select which assessments count towards progress decisions. Attending to power and trust in WBA may stimulate progress towards the previously aspirational goal of assessment for learning in the workplace.
Collapse
Affiliation(s)
- Damian J. Castanelli
- School of Clinical Sciences at Monash HealthMonash UniversityClaytonVictoriaAustralia
- Department of Anaesthesia and Perioperative MedicineMonash HealthClaytonVictoriaAustralia
- Centre for Research and Assessment in Digital Learning (CRADLE)Deakin UniversityGeelongVictoriaAustralia
| | - Jennifer M. Weller
- Centre for Medical and Health Sciences Education, School of MedicineUniversity of AucklandAucklandNew Zealand
- Department of AnaesthesiaAucklandAucklandNew Zealand
| | - Elizabeth Molloy
- Department of Medical Education, Melbourne Medical SchoolUniversity of MelbourneMelbourneVictoriaAustralia
| | - Margaret Bearman
- Centre for Research and Assessment in Digital Learning (CRADLE)Deakin UniversityGeelongVictoriaAustralia
| |
Collapse
|
29
|
Ziv T, Wamsley M, Lai CJ, Griffiths EP, Maxey A, Kryzhanovskaya IE, Teherani A. A is for advocacy: How introducing student advocacy assessment impacts longitudinal integrated clerkship students and clinical supervisors. MEDICAL TEACHER 2022; 44:149-157. [PMID: 34433360 DOI: 10.1080/0142159x.2021.1967903] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
To improve health, physicians are increasingly called to advocate. Yet medical schools have only recently focused on health advocacy skill-building. Limited work to date addresses assessing medical student advocacy on behalf of patients. We describe how students and clinical supervisors (CS) in two urban longitudinal integrated clerkships (LIC) experience patient advocacy and how introducing a new advocacy assessment impacts them. Using a thematic approach, we analyzed transcripts of focus groups during 2018-2019. Seventeen of 24 (71%) students and 15 of 21 (71%) CS participated in the focus groups. We describe how students perceive their advocacy role as they accompany the patient, amplify their voice, and facilitate connection. The rationale for advocacy assessment includes that it (1) adds a novel dimension to the written and verbal assessment, (2) drives student learning, (3) aligns with the institutional goal to promote equity, and (4) impacts CS teaching and clinical practice. Challenges are the ambiguity of expectations, pressure to 'perform,' and a moral overlay to advocacy assessment. Findings demonstrate how educational alliances between students and CS and longitudinal relationships between LIC students and patients offer a constructive opportunity for advocacy assessment. We describe suggestions to hone and expand the reach of advocacy assessment.
Collapse
Affiliation(s)
- Tali Ziv
- Kaiser Permanente, Oakland, CA, USA
- University of California San Francisco School of Medicine, San Francisco, CA, USA
| | - Maria Wamsley
- University of California San Francisco School of Medicine, San Francisco, CA, USA
| | - Cindy J Lai
- University of California San Francisco School of Medicine, San Francisco, CA, USA
| | | | - Alexandra Maxey
- University of California San Francisco School of Medicine, San Francisco, CA, USA
| | | | - Arianne Teherani
- University of California San Francisco School of Medicine, San Francisco, CA, USA
| |
Collapse
|
30
|
Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education. Br J Anaesth 2022; 128:691-699. [PMID: 35027168 DOI: 10.1016/j.bja.2021.12.012] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2021] [Revised: 11/23/2021] [Accepted: 12/10/2021] [Indexed: 01/01/2023] Open
Abstract
BACKGROUND Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training. METHODS The ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences. RESULTS We analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test-retest reliability was high (intraclass correlation coefficient [ICC2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent. CONCLUSIONS This study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.
Collapse
|
31
|
Kaushal B, Singh D, Magoon R, Kashav R. Pedagogy-andragogyheutagogy: Towards transformative educational epistemologies. J Anaesthesiol Clin Pharmacol 2022; 38:497-498. [PMID: 36505193 PMCID: PMC9728418 DOI: 10.4103/joacp.joacp_270_20] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Revised: 07/06/2020] [Accepted: 01/04/2021] [Indexed: 12/15/2022] Open
Affiliation(s)
- Brajesh Kaushal
- Department of Anaesthesia, Gandhi Medical College and Hamidia Hospital, Bhopal, Madhya Pradesh, India
| | - Daljeet Singh
- Department of Anaesthesia, Atal Bihari Vajpayee Institute of Medical Sciences (ABVIMS) and Dr. Ram Manohar Lohia Hospital, Baba Kharak Singh Marg, New Delhi, India
| | - Rohan Magoon
- Department of Cardiac Anaesthesia, Atal Bihari Vajpayee Institute of Medical Sciences (ABVIMS) and Dr. Ram Manohar Lohia Hospital, Baba Kharak Singh Marg, New Delhi, India,Address for correspondence: Dr. Rohan Magoon, Department of Cardiac Anaesthesia, Atal Bihari Vajpayee Institute of Medical Sciences (ABVIMS) and Dr. Ram Manohar Lohia Hospital, Baba Kharak Singh Marg, New Delhi - 110001, India. E-mail:
| | - Ramesh Kashav
- Department of Cardiac Anaesthesia, Atal Bihari Vajpayee Institute of Medical Sciences (ABVIMS) and Dr. Ram Manohar Lohia Hospital, Baba Kharak Singh Marg, New Delhi, India
| |
Collapse
|
32
|
Rotthoff T, Kadmon M, Harendza S. It does not have to be either or! Assessing competence in medicine should be a continuum between an analytic and a holistic approach. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:1659-1673. [PMID: 33779895 PMCID: PMC8610945 DOI: 10.1007/s10459-021-10043-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/02/2020] [Accepted: 03/07/2021] [Indexed: 06/12/2023]
Abstract
Assessing competence is a tremendous challenge in medical education. There are two contrasting approaches in competence assessment: an analytic approach that aims to precisely measure observable constituents and facets of competence and a holistic approach that focuses on a comprehensive assessment of competences in complex real situations reflecting actual performance. We would like to contribute to the existing discourse about medical competence and its assessment by proposing an approach that can provide orientation for the development of competence-based assessment concepts in undergraduate and postgraduate medical education. The approach follows Kane's framework of an "argument-based approach" to validity and is based on insights into task complexity, testing and learning theories as well as the importance of the learning environment. It describes a continuum from analytic to holistic approaches to assess the constituents and facets of competence to performance. We conclude that the complexity of a task should determine the selection of the assessment and suggest to use this approach to reorganize and adapt competence assessment.
Collapse
Affiliation(s)
- Thomas Rotthoff
- Medical Didactics and Educational Research, DEMEDA, Medical Faculty, University of Augsburg, Universitätsstrasse 2, 86159, Augsburg, Germany.
| | - Martina Kadmon
- Medical Education Sciences, DEMEDA, Medical Faculty, University of Augsburg, Augsburg, Germany
| | - Sigrid Harendza
- III. Department of Medicine, University Hospital Hamburg-Eppendorf, Hamburg, Germany
| |
Collapse
|
33
|
Anderson HL, Kurtz J, West DC. Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S164-S174. [PMID: 34406132 DOI: 10.1097/acm.0000000000004366] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
PURPOSE Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions. METHOD The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders. RESULTS In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain. CONCLUSIONS Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.
Collapse
Affiliation(s)
- Hannah L Anderson
- H.L. Anderson is research associate, Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0002-9435-1535
| | - Joshua Kurtz
- J. Kurtz is a first-year resident, Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - Daniel C West
- D.C. West is professor of pediatrics, The Perelman School of Medicine at the University of Pennsylvania, and associate chair for education and senior director of medical education, Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0002-0909-4213
| |
Collapse
|
34
|
Pearce J, Tavares W. A philosophical history of programmatic assessment: tracing shifting configurations. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:1291-1310. [PMID: 33893881 DOI: 10.1007/s10459-021-10050-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/23/2020] [Accepted: 04/09/2021] [Indexed: 06/12/2023]
Abstract
Programmatic assessment is now well entrenched in medical education, allowing us to reflect on when it first emerged and how it evolved into the form we know today. Drawing upon the intellectual tradition of historical epistemology, we provide a philosophically-oriented historiographical study of programmatic assessment. Our goal is to trace its relatively short historical trajectory by describing shifting configurations in its scene of inquiry-focusing on questions, practices, and philosophical presuppositions. We identify three historical phases: emergence, evolution and entrenchment. For each, we describe the configurations of the scene; examine underlying philosophical presuppositions driving changes; and detail upshots in assessment practice. We find that programmatic assessment emerged in response to positivist 'turmoil' prior to 2005, driven by utility considerations and implicit pragmatist undertones. Once introduced, it evolved with notions of diversity and learning being underscored, and a constructivist ontology developing at its core. More recently, programmatic assessment has become entrenched as its own sub-discipline. Rich narratives have been emphasised, but philosophical underpinnings have been blurred. We hope to shed new light on current assessment practices in the medical education community by interrogating the history of programmatic assessment from this philosophical vantage point. Making philosophical presuppositions explicit highlights the perspectival nature of aspects of programmatic assessment, and suggest reasons for perceived benefits as well as potential tensions, contradictions and vulnerabilities in the approach today. We conclude by offering some reflections on important points to emerge from our historical study, and suggest 'what next' for programmatic assessment in light of this endeavour.
Collapse
Affiliation(s)
- J Pearce
- Tertiary Education (Assessment), Australian Council for Educational Research, 19 Prospect Hill Road, Camberwell, VIC, 3124, Australia.
| | - W Tavares
- The Wilson Centre and Post-MD Education. University Health Network and University of Toronto, Toronto, ON, Canada
| |
Collapse
|
35
|
Ginsburg S, Watling CJ, Schumacher DJ, Gingerich A, Hatala R. Numbers Encapsulate, Words Elaborate: Toward the Best Use of Comments for Assessment and Feedback on Entrustment Ratings. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S81-S86. [PMID: 34183607 DOI: 10.1097/acm.0000000000004089] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The adoption of entrustment ratings in medical education is based on a seemingly simple premise: to align workplace-based supervision with resident assessment. Yet it has been difficult to operationalize this concept. Entrustment rating forms combine numeric scales with comments and are embedded in a programmatic assessment framework, which encourages the collection of a large quantity of data. The implicit assumption that more is better has led to an untamable volume of data that competency committees must grapple with. In this article, the authors explore the roles of numbers and words on entrustment rating forms, focusing on the intended and optimal use(s) of each, with a focus on the words. They also unpack the problematic issue of dual-purposing words for both assessment and feedback. Words have enormous potential to elaborate, to contextualize, and to instruct; to realize this potential, educators must be crystal clear about their use. The authors set forth a number of possible ways to reconcile these tensions by more explicitly aligning words to purpose. For example, educators could focus written comments solely on assessment; create assessment encounters distinct from feedback encounters; or use different words collected from the same encounter to serve distinct feedback and assessment purposes. Finally, the authors address the tyranny of documentation created by programmatic assessment and urge caution in yielding to the temptation to reduce words to numbers to make them manageable. Instead, they encourage educators to preserve some educational encounters purely for feedback, and to consider that not all words need to become data.
Collapse
Affiliation(s)
- Shiphra Ginsburg
- S. Ginsburg is professor of medicine, Department of Medicine, Sinai Health System and Faculty of Medicine, University of Toronto, scientist, Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, Canada, and Canada Research Chair in Health Professions Education; ORCID: http://orcid.org/0000-0002-4595-6650
| | - Christopher J Watling
- C.J. Watling is professor and director, Centre for Education Research and Innovation, Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada; ORCID: https://orcid.org/0000-0001-9686-795X
| | - Daniel J Schumacher
- D.J. Schumacher is associate professor of pediatrics, Cincinnati Children's Hospital Medical Center and University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452
| | - Andrea Gingerich
- A. Gingerich is assistant professor, Northern Medical Program, University of Northern British Columbia, Prince George, British Columbia, Canada; ORCID: https://orcid.org/0000-0001-5765-3975
| | - Rose Hatala
- R. Hatala is professor, Department of Medicine, and director, Clinical Educator Fellowship, Center for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada; ORCID: https://orcid.org/0000-0003-0521-2590
| |
Collapse
|
36
|
Kinnear B, Warm EJ, Caretta-Weyer H, Holmboe ES, Turner DA, van der Vleuten C, Schumacher DJ. Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S56-S63. [PMID: 34183603 DOI: 10.1097/acm.0000000000004108] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.
Collapse
Affiliation(s)
- Benjamin Kinnear
- B. Kinnear is associate professor of internal medicine and pediatrics, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-0052-4130
| | - Eric J Warm
- E.J. Warm is professor of internal medicine and program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6088-2434
| | - Holly Caretta-Weyer
- H. Caretta-Weyer is assistant professor of emergency medicine, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| | - Eric S Holmboe
- E.S. Holmboe is chief, research, milestones development and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0003-0108-6021
| | - David A Turner
- D.A. Turner is vice president, Competency-Based Medical Education, American Board of Pediatrics, Chapel Hill, North Carolina
| | - Cees van der Vleuten
- C. van der Vleuten is professor of education, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands; ORCID: https://orcid.org/0000-0001-6802-3119
| | - Daniel J Schumacher
- D.J. Schumacher is associate professor of pediatrics, Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452
| |
Collapse
|
37
|
Seligman L, Abdullahi A, Teherani A, Hauer KE. From Grading to Assessment for Learning: A Qualitative Study of Student Perceptions Surrounding Elimination of Core Clerkship Grades and Enhanced Formative Feedback. TEACHING AND LEARNING IN MEDICINE 2021; 33:314-325. [PMID: 33228392 DOI: 10.1080/10401334.2020.1847654] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Problem: Medical students perceive honors grading during core clerkships as unfair and inequitable, and negatively impacting their learning and wellbeing. Eliminating honors grading, a powerful extrinsic motivator, and emphasizing formative feedback may address these problems and promote intrinsic motivation and learning. However, it is unknown how transitioning from honors to pass/fail grading with enhanced formative feedback in the core clerkship year may affect student learning experiences, wellbeing, and perceptions of the learning environment. Intervention: Core clerkship grading was transitioned from honors/pass/fail to pass/fail at one US medical school. In addition, the requirement for students to obtain formative supervisor feedback was formalized to twice per week. Context: This qualitative study utilized semi-structured interviews to explore the perceptions among core clerkship students of learning and assessment. Interview questions addressed motivation, wellbeing, learning behaviors, team dynamics, feedback, and student and supervisor attitudes regarding assessment changes. The authors analyzed data inductively using thematic analysis informed by sensitizing concepts related to theories of motivation (goal orientation theory and self-determination theory). Impact: Eighteen students participated, including five with experience in both honors-eligible and pass/fail clerkships. The authors identified three major themes in students' descriptions of the change in approach to assessment: student engagement in clerkships, wellbeing, and recognition of learning context. Student engagement subthemes included intrinsic motivation for patient care rather than performing; sense of agency over learning, including ability to set learning priorities, seek and receive feedback, take learning risks, and disagree with supervisors, and collaborative relationships with peers and team members. Positive wellbeing was characterized by low stress, sense of authenticity with team members, prioritized physical health, and attention to personal life. Learning context subthemes included recognition of variability of clerkship contexts with pass/fail grading mitigating fairness and equity concerns, support of the grading change from residents and some attendings, and implications surrounding future stress and residency selection. Lessons Learned: Students perceive a transition from honors grading to pass/fail with increased feedback as supporting their engagement in learning, intrinsic motivation, and wellbeing. Drivers of wellbeing appear to include students' feelings of control, achieved through the ability to seek learning opportunities, teaching, and constructive feedback without the perceived need to focus on impressing others. Ongoing evaluation of the consequences of this shift in assessment is needed.
Collapse
Affiliation(s)
- Lee Seligman
- Internal Medicine, NewYork-Presbyterian Hospital, Columbia University Irving Medical Center, New York, New York, USA
| | - Abdikarin Abdullahi
- University of California, San Francisco School of Medicine, San Francisco, California, USA
| | - Arianne Teherani
- University of California, San Francisco School of Medicine, San Francisco, California, USA
- Department of Medicine, University of California, San Francisco, California, USA
| | - Karen E Hauer
- University of California, San Francisco School of Medicine, San Francisco, California, USA
- Department of Medicine, University of California, San Francisco, California, USA
| |
Collapse
|
38
|
Valentine N, Durning S, Shanahan EM, Schuwirth L. Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:713-738. [PMID: 33123837 DOI: 10.1007/s10459-020-10002-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Accepted: 10/19/2020] [Indexed: 06/11/2023]
Abstract
Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.
Collapse
Affiliation(s)
- Nyoli Valentine
- Prideaux Health Professions Education, Flinders University, Bedford Park 5042, SA, Australia.
| | - Steven Durning
- Center for Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
| | - Ernst Michael Shanahan
- Prideaux Health Professions Education, Flinders University, Bedford Park 5042, SA, Australia
| | - Lambert Schuwirth
- Prideaux Health Professions Education, Flinders University, Bedford Park 5042, SA, Australia
| |
Collapse
|
39
|
van der Meulen MW, Arah OA, Heeneman S, Oude Egbrink MGA, van der Vleuten CPM, Lombarts KMJMH. When Feedback Backfires: Influences of Negative Discrepancies Between Physicians' Self and Assessors' Scores on Their Subsequent Multisource Feedback Ratings. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2021; 41:94-103. [PMID: 34009839 DOI: 10.1097/ceh.0000000000000347] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
INTRODUCTION With multisource feedback (MSF) physicians might overrate their own performance compared with scores received from assessors. However, there is limited insight into how perceived divergent feedback affects physicians' subsequent performance scores. METHODS During 2012 to 2018, 103 physicians were evaluated twice by 684 peers, 242 residents, 999 coworkers, and themselves in three MSF performance domains. Mixed-effect models quantified associations between the outcome variable "score changes" between first and second MSF evaluations, and the explanatory variable "negative discrepancy score" (number of items that physicians rated themselves higher compared with their assessors' scores) at the first MSF evaluation. Whether associations differed across assessor groups and across a physician's years of experience as a doctor was analyzed too. RESULTS Forty-nine percent of physicians improved their total MSF score at the second evaluation, as assessed by others. Number of negative discrepancies was negatively associated with score changes in domains "organization and (self)management" (b = -0.02; 95% confidence interval [CI], -0.03 to -0.02; SE = 0.004) and "patient-centeredness" (b = -0.03; 95% CI, -0.03 to -0.02; SE = 0.004). For "professional attitude," only negative associations between score changes and negative discrepancies existed for physicians with more than 6-year experience (b6-10yearsofexperience = -0.03; 95% CI, -0.05 to -0.003; SE = 0.01; b16-20yearsofexperience = -0.03; 95% CI, -0.06 to -0.004; SE = 0.01). DISCUSSION The extent of performance improvement was less for physicians confronted with negative discrepancies. Performance scores actually declined when physicians overrated themselves on more than half of the feedback items. PA score changes of more experienced physicians confronted with negative discrepancies and were affected more adversely. These physicians might have discounted feedback due to having more confidence in own performance. Future work should investigate how MSF could improve physicians' performance taking into account physicians' confidence.
Collapse
Affiliation(s)
- Mirja W van der Meulen
- Dr. van der Meulen: is PhD Candidate, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands, and Professional Performance and Compassionate Care Research Group, Department of Medical Psychology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands. Dr. Arah: is professor, Department of Epidemiology, University of California, Los Angeles (UCLA), Los Angeles, the United States of America. Dr. Heeneman: is professor, Department of Pathology, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, the Netherlands. Dr. oude Egbrink: is professor, Department of Physiology, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. Dr. van der Vleuten: is professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. Dr. Lombarts: is professor, Professional Performance and Compassionate Care Research Group, Department of Medical Psychology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | | | | | | | | | | |
Collapse
|
40
|
Padmore JS, Andolsek KM, Iobst WF, Poulin LJ, Hogan SO, Richard KM. Navigating Academic Law in Competency Decisions. J Grad Med Educ 2021; 13:102-108. [PMID: 33936542 PMCID: PMC8078072 DOI: 10.4300/jgme-d-20-00963.1] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/31/2022] Open
Affiliation(s)
- Jamie S. Padmore
- Jamie S. Padmore, DM, is Professor and Senior Associate Dean for Medical Education, Georgetown University Medical Center, and Vice President, Academic Affairs, and Designated Institutional Official, MedStar Health
| | - Kathryn M. Andolsek
- Kathryn M. Andolsek, MD, MPH, is Professor, Department of Family Medicine and Community Health, Assistant Dean, Premedical Education, and Senior Fellow, Center for Study of Aging and Human Development, Duke University School of Medicine
| | - William F. Iobst
- William F. Iobst, MD, FACP, is Senior Scholar, Department of Research, Milestones Development, and Evaluation, Accreditation Council for Graduate Medical Education (ACGME), and Emeritus Professor of Medicine, Geisinger Commonwealth School of Medicine
| | - Lauren J. Poulin
- Lauren J. Poulin, PhD, MPP, is Milestones Administrator, Milestones Department, ACGME
| | - Sean O. Hogan
- Sean O. Hogan, PhD, is Director, Outcomes Research and Evaluation, ACGME
| | - Kerry M. Richard
- Kerry M. Richard, Esq, is Vice President and Senior Deputy General Counsel, MedStar Health
| |
Collapse
|
41
|
Fainstad TL, McClintock AH, Yarris LM. Bias in assessment: name, reframe, and check in. CLINICAL TEACHER 2021; 18:449-453. [PMID: 33787001 DOI: 10.1111/tct.13351] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2021] [Revised: 02/19/2021] [Accepted: 03/11/2021] [Indexed: 11/28/2022]
Abstract
Cognitive bias permeates almost every learner assessment in medical education. Assessment bias has the potential to affect a learner's education, future career and sense of self-worth. Decades of data show that there is little educators can do to overcome bias in learner assessments. Using in-group favouritism as an example, we offer an evidence-based, three-step solution to understand and move forward with cognitive bias in assessment: (1) Name: a simple admission about the presence of inherent bias in assessment, (2) Reframe: a rephrasing of assessment language to shed light on the assessor's subjectivity and (3) Check-in: a chance to ensure learner understanding and open lines of bidirectional communication. This process is theory-informed and based on decades of educational, sociological and psychological literature; we offer it as a logical first step towards a much-needed paradigm shift towards addressing bias in learner assessment.
Collapse
Affiliation(s)
- Tyra L Fainstad
- Department of Medicine, Division of General Internal Medicine, University of Colorado, Aurora, CO, USA
| | - Adelaide H McClintock
- Department of Medicine, Division of General Internal Medicine, University of Washington, Seattle, WA, USA
| | | |
Collapse
|
42
|
McDonald JA, Lai CJ, Lin MYC, O'Sullivan PS, Hauer KE. "There Is a Lot of Change Afoot": A Qualitative Study of Faculty Adaptation to Elimination of Tiered Grades With Increased Emphasis on Feedback in Core Clerkships. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:263-270. [PMID: 32909998 DOI: 10.1097/acm.0000000000003730] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
PURPOSE Medical schools responding to challenges with fairness, equity, learning environments, and student wellness have reconsidered clerkship grades. This study used the Concerns-Based Adoption Model (CBAM) to explore how faculty responded to a change in the approach to assessment from focus on grades toward focus on feedback. METHOD This qualitative study used an inductive approach to analyze data from semistructured interviews with teaching faculty and education leaders at University of California, San Francisco, School of Medicine 6 months following the elimination of traditional tiered clerkship grades. Participants were recruited in 2019 using purposive sampling. Interview questions addressed participants' perceptions of the new approach to assessment and grading that emphasizes assessment for learning, including advantages and concerns. The authors analyzed data using thematic analysis informed by sensitizing concepts from CBAM. RESULTS Nineteen faculty participated (11 medicine, 8 surgery). Faculty expressed optimism about the impact of the change on their clinical educator roles and sought faculty development to enhance feedback and assessment skills. Perceived benefits to students addressed learning and wellness, with concern for students' motivation and professional development. Faculty shared uncertainty about their roles and larger systemic consequences involving the residency match and institutional reputation. Participants acknowledged that traditional grading is imperfect, yet felt a departure from this system represented significant culture change. CONCLUSIONS Faculty perceptions of the assessment and grade change in 2 large departments demonstrated tensions around grading. Their descriptions of change as an ongoing process aligned with CBAM and required them to consider new approaches and perspectives. While discourse about assessment and grading frequently focuses on learners, these findings support institutional consideration of the impact of changes in assessment on teaching faculty.
Collapse
MESH Headings
- Clinical Clerkship/standards
- Clinical Clerkship/statistics & numerical data
- Clinical Competence/statistics & numerical data
- Education, Medical/standards
- Education, Medical/statistics & numerical data
- Education, Medical, Undergraduate/standards
- Education, Medical, Undergraduate/statistics & numerical data
- Educational Measurement/methods
- Faculty, Medical/statistics & numerical data
- Faculty, Medical/trends
- Feedback
- Female
- Humans
- Leadership
- Learning/physiology
- Male
- Perception
- Qualitative Research
- San Francisco/epidemiology
- Schools, Medical/organization & administration
- Students, Medical/psychology
- Surveys and Questionnaires
- Thematic Apperception Test/statistics & numerical data
- Universities/organization & administration
Collapse
Affiliation(s)
- Jordan A McDonald
- J.A. McDonald is a third-year medical student, University of California, San Francisco, School of Medicine, San Francisco, California
| | - Cindy J Lai
- C.J. Lai is director of internal medicine clerkships and professor, Department of Medicine, University of California, San Francisco, School of Medicine, San Francisco, California
| | - Matthew Y C Lin
- M.Y.C. Lin is director of surgery clerkships and associate professor, Department of Surgery, University of California, San Francisco, School of Medicine, San Francisco, California
| | - Patricia S O'Sullivan
- P.S. O'Sullivan is director of research and development in medical education and professor, Department of Medicine and Department of Surgery, University of California, San Francisco, School of Medicine, San Francisco, California
| | - Karen E Hauer
- K.E. Hauer is associate dean for competency assessment and professional standards and professor, Department of Medicine, University of California, San Francisco, School of Medicine, San Francisco, California
| |
Collapse
|
43
|
Skipper M, Nøhr SB, Engeström Y. The change laboratory in medical education: Two examples of tackling contradictory challenges. MEDICAL EDUCATION 2021; 55:93-100. [PMID: 32722852 DOI: 10.1111/medu.14298] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/15/2020] [Revised: 07/08/2020] [Accepted: 07/20/2020] [Indexed: 06/11/2023]
Abstract
CONTEXT Medical education and workplace learning is bound to develop through tensions between providing high quality patient care and providing training of the future specialist healthcare workforce. This paper on the Change Laboratory and the theoretical framework supporting it, shows examples on how to explore inherent and contradictory tensions in medical education and healthcare and use them as a driving force for change. We argue that the traditional tools and theories for change and fixing tensions are inadequate and therefore suggest an alternative strategy found in Cultural-Historical Activity Theory (CHAT) and the Change Laboratory method. METHODS The Change Laboratory intervention method builds on the theoretical framework of CHAT and specifically the theory of expansive learning. The Change Laboratory intervention method uses well-defined steps for participants in collaboration with researchers/facilitators to co-construct and develop new ways of going about their work practice. RESULTS Drawing on our own research on implementing the Change Laboratory intervention method we present two case examples of interventions in respectively a Finnish surgical unit and a Danish paediatric outpatient clinic. CONCLUSIONS The Change Laboratory intervention offers ways to systematically leverage tensions in medical education and thus could be effective in developing and designing organisational and professional change. It is not a quick fix solution as participators must be motivated and engaged in uncovering inherent contradictions in their activity systems (workplace) and get familiar with the concepts and theory underlying the intervention and its procedures. Profound knowledge and transformative agency emerges when participants and facilitators/researchers are given the time and opportunity to analyse both historical practice, current data on practice, and organisational issues collaboratively in order to envision and redesign their practice and learning environment.
Collapse
Affiliation(s)
- Mads Skipper
- Ringgold Standard Institution, Centre for Health Sciences Education, Aarhus University, Aarhus, Denmark
- Postgraduate Medical Education Region North, Viborg, Denmark
| | - Susanne Backman Nøhr
- Ringgold Standard Institution, Centre for Health Sciences Education, Aarhus University, Aarhus, Denmark
- Department for Postgraduate Education, Ringgold Standard Institution, Aalborg University Hospital, Aalborg, Denmark
| | - Yrjö Engeström
- Faculty of Educational Sciences, Ringgold Standard Institution, CRADLE, University of Helsinki, Helsinki, Finland
| |
Collapse
|
44
|
Young JQ, Sugarman R, Schwartz J, O'Sullivan PS. Faculty and Resident Engagement With a Workplace-Based Assessment Tool: Use of Implementation Science to Explore Enablers and Barriers. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:1937-1944. [PMID: 32568853 DOI: 10.1097/acm.0000000000003543] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
PURPOSE Implementation of workplace-based assessment programs has encountered significant challenges. Faculty and residents alike often have a negative view of these programs as "tick-box" or "jump through the hoops" exercises. A number of recommendations have been made to address these challenges. To understand the experience with a workplace-based assessment tool that follows many of these recommendations, the authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with the tool. METHOD The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) is a direct observation tool designed to assess resident performance during a psychiatric medication management visit. From August 2017 to February 2018, the P-SCO was implemented in the outpatient continuity clinics for second- and third-year residents at Zucker Hillside Hospital/Northwell Health. In February and March 2019, the authors conducted semistructured interviews of participating faculty and residents. Interview guides based on the CFIR were used to capture the enablers and barriers to engagement. Interview transcripts were independently coded. Codes were then organized into themes relevant to the domains of the CFIR. RESULTS Ten faculty and 10 residents were interviewed. Overall, participants had a positive experience with the P-SCO. Enabling factors for faculty and residents included the ongoing training, design features of the P-SCO, predisposing beliefs, dedicated faculty time, and the perception that the P-SCO improved verbal feedback quality. Barriers for faculty included checklist length and discomfort with feedback that threatens identity, and barriers for residents included faculty variability in timeliness and quality of feedback and minimal review of the feedback after initial receipt. CONCLUSIONS This study demonstrates that the negative experience of faculty and residents with workplace-based assessment tools shown in prior studies can be overcome, at least in part, when specific implementation strategies are pursued. The findings provide guidance for future research and implementation efforts.
Collapse
Affiliation(s)
- John Q Young
- J.Q. Young is professor of psychiatry and vice chair for education, Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, and Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York
| | - Rebekah Sugarman
- R. Sugarman is a research assistant, Department of Psychiatry, Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York
| | - Jessica Schwartz
- J. Schwartz is a resident, Department of Psychiatry, Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York
| | - Patricia S O'Sullivan
- P.S. O'Sullivan is professor, Department of Medicine, and director of research and development in medical education, University of California, San Francisco, School of Medicine, San Francisco, California
| |
Collapse
|
45
|
Lucey CR, Hauer KE, Boatright D, Fernandez A. Medical Education's Wicked Problem: Achieving Equity in Assessment for Medical Learners. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:S98-S108. [PMID: 32889943 DOI: 10.1097/acm.0000000000003717] [Citation(s) in RCA: 63] [Impact Index Per Article: 15.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Despite a lack of intent to discriminate, physicians educated in U.S. medical schools and residency programs often take actions that systematically disadvantage minority patients. The approach to assessment of learner performance in medical education can similarly disadvantage minority learners. The adoption of holistic admissions strategies to increase the diversity of medical training programs has not been accompanied by increases in diversity in honor societies, selective residency programs, medical specialties, and medical school faculty. These observations prompt justified concerns about structural and interpersonal bias in assessment. This manuscript characterizes equity in assessment as a "wicked problem" with inherent conflicts, uncertainty, dynamic tensions, and susceptibility to contextual influences. The authors review the underlying individual and structural causes of inequity in assessment. Using an organizational model, they propose strategies to achieve equity in assessment and drive institutional and systemic improvement based on clearly articulated principles. This model addresses the culture, systems, and assessment tools necessary to achieve equitable results that reflect stated principles. Three components of equity in assessment that can be measured and evaluated to confirm success include intrinsic equity (selection and design of assessment tools), contextual equity (the learning environment in which assessment occurs), and instrumental equity (uses of assessment data for learner advancement and selection and program evaluation). A research agenda to address these challenges and controversies and demonstrate reduction in bias and discrimination in medical education is presented.
Collapse
Affiliation(s)
- Catherine R Lucey
- C.R. Lucey is executive vice dean/vice dean for education and professor of medicine, University of California, San Francisco, School of Medicine, San Francisco, California
| | - Karen E Hauer
- K.E. Hauer is professor of medicine, University of California, San Francisco, School of Medicine, San Francisco, California
| | - Dowin Boatright
- D. Boatright is assistant professor of emergency medicine, Yale University School of Medicine, New Haven, Connecticut
| | - Alicia Fernandez
- A. Fernandez is professor of medicine, University of California, San Francisco, School of Medicine, San Francisco, California
| |
Collapse
|
46
|
Validity of entrustment scales within anesthesiology residency training. Can J Anaesth 2020; 68:53-63. [PMID: 33083924 DOI: 10.1007/s12630-020-01823-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2019] [Revised: 07/09/2020] [Accepted: 07/12/2020] [Indexed: 10/23/2022] Open
Abstract
INTRODUCTION Competency-based medical education requires robust assessment in authentic clinical environments. Using work-based assessments, entrustment scales have emerged as a means of describing a trainee's ability to perform competently. Nevertheless, psychometric properties of entrustment-based assessment are relatively unknown, particularly in anesthesiology. This study assessed the generalizability and extrapolation evidence for entrustment scales within a program of assessment during anesthesiology training. METHODS Entrustment scores were collected during the first seven blocks of training for three resident cohorts. Entrustment scores were assessed during daily evaluations using a Clinical Case Assessment Tool (CCAT) within the preoperative, intraoperative, and postoperative setting. The reliability of the entrustment scale was estimated using generalizability theory. Spearman's correlations measured the relationship between median entrustment scores and percentiles scores on the Anesthesia Knowledge Test (AKT)-1 and AKT-6, mean Objective Structured Clinical Examination (OSCE) scores, and rankings of performance by the Clinical Competence Committee (CCC). RESULTS Analyses were derived from 2,309 CCATs from 35 residents. The reliability or generalizability (G) coefficient of the entrustment scale was 0.73 (95% confidence interval [CI], 0.70 to 0.76), and the internal consistency was 0.86 (95% CI, 0.84 to 0.88). Intraoperative entrustment scores significantly correlated with the AKT-6 (rho = 0.51, P = 0.01), mean OSCE (rho = 0.45, P = 0.04), and CCC performance rankings (rho = 0.52, P = 0.006). CONCLUSION As part of an assessment program, entrustment scales used early during anesthesiology training showed evidence of validity. Intraoperative entrustment scores had good reliability and showed acceptable internal consistency. Interpreting entrustment scores in this setting may constitute a valuable adjunct complementing traditional summative evaluations.
Collapse
|
47
|
Huwendiek S, Jung D, Schirlo C, Huber P, Balestra G, Guttormsen S, Berendonk C. The introduction of a standardised national licensing exam as a driver of change in medical education: A qualitative study from Switzerland. MEDICAL TEACHER 2020; 42:1163-1170. [PMID: 32772611 DOI: 10.1080/0142159x.2020.1798911] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
INTRODUCTION Only a few studies have described the impacts, strengths and needs for further development of national licensing exams (NLE). To gain such insights regarding the Swiss NLE, which includes a multiple-choice and a standardised clinical skills exam, we explored the perceptions of involved experts and stakeholders. METHODS We explored participants' perceptions in four focus group discussions. The interviews were recorded, transcribed verbatim and qualitatively analysed using a thematic analysis approach. RESULTS The analysis resulted in five perceived impacts, two strengths and two needs for further developments of the NLE. Perceived impacts were (1) steering students' learning behaviour, (2) supporting teachers and assessors to align teaching to competencies, (3) elevating the importance of the Swiss Catalogue of Learning Objectives, (4) setting incentives for the further development of curricula, and (5) fostering the collaboration between the faculties of medicine. Perceived strengths were the blend of assessment formats, including their competency-based orientation, and the collaborative development approach. Perceived needs lay in the NLE's further development to sustain its fit for purpose and in incentives for people involved. CONCLUSION According to our study, this NLE had, and has, notable impacts on medical education in Switzerland. Our insights can be useful for others planning a similar undertaking.
Collapse
Affiliation(s)
- Sören Huwendiek
- Institute for Medical Education, Department for Assessment and Evaluation, University of Bern, Bern, Switzerland
| | | | - Christian Schirlo
- Head Structure and Development, Office Vice President Medicine, University of Zurich, Zürich, Switzerland
| | - Philippe Huber
- Department of Rehabilitation and Geriatrics, Geneva University Hospitals, Geneva, Switzerland
| | | | - Sissel Guttormsen
- Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Christoph Berendonk
- Institute for Medical Education, Department for Assessment and Evaluation, University of Bern, Bern, Switzerland
| |
Collapse
|
48
|
Gumuchian ST, Pal NE, Young M, Danoff D, Plotnick LH, Cummings BA, Gomez-Garibello C, Dory V. Learner handover: Perspectives and recommendations from the front-line. PERSPECTIVES ON MEDICAL EDUCATION 2020; 9:294-301. [PMID: 32809189 PMCID: PMC7550510 DOI: 10.1007/s40037-020-00601-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2023]
Abstract
INTRODUCTION Current medical education models increasingly rely on longitudinal assessments to document learner progress over time. This longitudinal focus has re-kindled discussion regarding learner handover-where assessments are shared across supervisors, rotations, and educational phases, to support learner growth and ease transitions. The authors explored the opinions of, experiences with, and recommendations for successful implementation of learner handover among clinical supervisors. METHODS Clinical supervisors from five postgraduate medical education programs at one institution completed an online questionnaire exploring their views regarding learner handover, specifically: potential benefits, risks, and suggestions for implementation. Survey items included open-ended and numerical responses. The authors used an inductive content analysis approach to analyze the open-ended questionnaire responses, and descriptive and correlational analyses for numerical data. RESULTS Seventy-two participants completed the questionnaire. Their perspectives varied widely. Suggested benefits of learner handover included tailored learning, improved assessments, and enhanced patient safety. The main reported risk was the potential for learner handover to bias supervisors' perceptions of learners, thereby affecting the validity of future assessments and influencing the learner's educational opportunities and well-being. Participants' suggestions for implementation focused on who should be involved, when and for whom it should occur, and the content that should be shared. DISCUSSION The diverse opinions of, and recommendations for, learner handover highlight the necessity for handover to maximize learning potential while minimizing potential harms. Supervisors' suggestions for handover implementation reveal tensions between assessment-of and for-learning.
Collapse
Affiliation(s)
| | - Nicole E Pal
- Institute of Health Sciences Education, McGill University, Montréal, Québec, Canada
| | - Meredith Young
- Institute of Health Sciences Education, McGill University, Montréal, Québec, Canada
- Department of Medicine, McGill University, Montréal, Québec, Canada
| | - Deborah Danoff
- Institute of Health Sciences Education, McGill University, Montréal, Québec, Canada
| | - Laurie H Plotnick
- Institute of Health Sciences Education, McGill University, Montréal, Québec, Canada
- Department of Pediatrics, McGill University, and Montreal Children's Hospital, McGill University Health Centre, Montréal, Québec, Canada
| | - Beth-Ann Cummings
- Institute of Health Sciences Education, McGill University, Montréal, Québec, Canada
- Department of Medicine, McGill University, Montréal, Québec, Canada
| | - Carlos Gomez-Garibello
- Institute of Health Sciences Education, McGill University, Montréal, Québec, Canada
- Department of Medicine, McGill University, Montréal, Québec, Canada
| | - Valérie Dory
- Department of General Practice, Université de Liège, Liège, Belgium.
| |
Collapse
|
49
|
Holmboe ES, Durning SJ. Understanding the social in diagnosis and error: a family of theories known as situativity to better inform diagnosis and error. Diagnosis (Berl) 2020; 7:161-164. [DOI: 10.1515/dx-2020-0080] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022]
Affiliation(s)
- Eric S. Holmboe
- Accreditation Council for Graduate Medical Education , Chicago , IL , USA
| | - Steven J. Durning
- Uniformed Services University of the Health Sciences , Bethesda , MD , USA
| |
Collapse
|
50
|
Schumacher DJ, Dornoff E, Carraccio C, Busari J, van der Vleuten C, Kinnear B, Kelleher M, Sall DR, Warm E, Martini A, Holmboe E. The Power of Contribution and Attribution in Assessing Educational Outcomes for Individuals, Teams, and Programs. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:1014-1019. [PMID: 31833856 DOI: 10.1097/acm.0000000000003121] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Recent discussions have brought attention to the utility of contribution analysis for evaluating the effectiveness and outcomes of medical education programs, especially for complex initiatives such as competency-based medical education. Contribution analysis focuses on the extent to which different entities contribute to an outcome. Given that health care is provided by teams, contribution analysis is well suited to evaluating the outcomes of care delivery. Furthermore, contribution analysis plays an important role in analyzing program- and system-level outcomes that inform program evaluation and program-level improvements for the future. Equally important in health care, however, is the role of the individual. In the overall contribution of a team to an outcome, some aspects of this outcome can be attributed to individual team members. For example, a recently discharged patient with an unplanned return to the emergency department to seek care may not have understood the discharge instructions given by the nurse or may not have received any discharge guidance from the resident physician. In this example, if it is the nurse's responsibility to provide discharge instructions, that activity is attributed to him or her. This and other activities attributed to different individuals (e.g., nurse, resident) combine to contribute to the outcome for the patient. Determining how to tease out such attributions is important for several reasons. First, it is physicians, not teams, that graduate and are granted certification and credentials for medical practice. Second, incentive-based payment models focus on the quality of care provided by an individual. Third, an individual can use data about his or her performance on the team to help drive personal improvement. In this article, the authors explored how attribution and contribution analyses can be used in a complimentary fashion to discern which outcomes can and should be attributed to individuals, which to teams, and which to programs.
Collapse
Affiliation(s)
- Daniel J Schumacher
- D.J. Schumacher is associate professor of pediatrics, Cincinnati Children's Hospital Medical Center and University of Cincinnati College of Medicine, Cincinnati, Ohio. E. Dornoff is a medical student, University of Cincinnati College of Medicine, Cincinnati, Ohio. C. Carraccio is vice president of competency-based assessment, American Board of Pediatrics, Chapel Hill, North Carolina. J. Busari is consultant pediatrician and associate professor of medical education, Maastricht University, Maastricht, the Netherlands. C. van der Vleuten is professor of education, Department of Educational Development and Research, Faculty of Health, Medicine, and Life Sciences, and scientific director, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. B. Kinnear is assistant professor of pediatrics and internal medicine, Cincinnati Children's Hospital Medical Center and University of Cincinnati College of Medicine, Cincinnati, Ohio. M. Kelleher is assistant professor of pediatrics and internal medicine, Cincinnati Children's Hospital Medical Center and University of Cincinnati College of Medicine, Cincinnati, Ohio. D.R. Sall is assistant professor of internal medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio. E. Warm is professor of medicine and internal medicine program director, University of Cincinnati College of Medicine, Cincinnati, Ohio. A. Martini is a clinical research coordinator, Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio. E. Holmboe is senior vice president for milestones development and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois
| | | | | | | | | | | | | | | | | | | | | |
Collapse
|