1
|
Chiel LE, Marcus CH, Gomez AR, Williams K, Buraks O, Fishman M, Angerman J, Winn AS. Peer evaluations in an anonymous and open system: Intern reactions and evaluation analysis. MEDICAL TEACHER 2024; 46:919-924. [PMID: 38065679 DOI: 10.1080/0142159x.2023.2287985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Accepted: 11/22/2023] [Indexed: 06/27/2024]
Abstract
PURPOSE Feedback and evaluation from peers is fundamental to trainees' professional development but may be uncomfortable to provide non-anonymously. We aimed to understand resident perception of anonymous and open written evaluation systems and to analyze evaluations in each of these systems. MATERIALS AND METHOD We compared two years of intern peer evaluations at a large United States-based pediatric residency program - the 2018-2019 and 2019-2020 years during which intern peer evaluations were anonymous and open, respectively. We electronically surveyed interns about their perceptions of peer evaluations and analyzed four aspects of the evaluations themselves: (1) orientation, (2) caliber, (3) Likert-scale, and (4) word count. RESULTS 40 (78%) and 38 (75%) interns participated in the survey in the anonymous and open years, respectively. Respondents reported being more likely to avoid writing constructive comments in the open year. There were more high caliber comments in the open year. Likert-scale ratings of peers were lower in the open year. Word count was longer in the open year. CONCLUSIONS While interns expressed more discomfort evaluating peers in an open evaluation system, they wrote longer and more high caliber comments in an open system than in an anonymous system. Residency programs should consider professional development in writing peer evaluation.
Collapse
Affiliation(s)
- Laura E Chiel
- Division of Pulmonary Medicine, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Carolyn H Marcus
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Amanda R Gomez
- Division of Gastroenterology, Hepatology, and Nutrition, Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago and Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Kathryn Williams
- Biostatistics and Research Design Center, Institutional Centers for Clinical and Translational Research, Boston Children's Hospital, Boston, Massachusetts, USA
| | | | - Michael Fishman
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Jessica Angerman
- Boston Combined Residency Program at Boston Children's Hospital and Health Informatics at Northwestern University, Evanston, Illinois, USA
| | - Ariel S Winn
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
2
|
Van Ostaeyen S, Embo M, Rotsaert T, De Clercq O, Schellens T, Valcke M. A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles. PERSPECTIVES ON MEDICAL EDUCATION 2023; 12:584-593. [PMID: 38144672 PMCID: PMC10742175 DOI: 10.5334/pme.1050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/14/2023] [Accepted: 11/08/2023] [Indexed: 12/26/2023]
Abstract
Introduction Competency-based education requires high-quality feedback to guide students' acquisition of competencies. Sound assessment and feedback systems, such as ePortfolios, are needed to facilitate seeking and giving feedback during clinical placements. However, it is unclear whether the written feedback comments in ePortfolios are of high quality and aligned with the current competency focus. Therefore, this study investigates the quality of written feedback comments in ePortfolios of healthcare students, as well as how these feedback comments align with the CanMEDS roles. Methods A qualitative textual analysis was conducted. 2,349 written feedback comments retrieved from the ePortfolios of 149 healthcare students (specialist medicine, general practice, occupational therapy, speech therapy and midwifery) were analysed retrospectively using deductive content analysis. Two structured categorisation matrices, one based on four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and another one on the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional), guided the analysis. Results The minority of the feedback comments (n = 352; 14.9%) could be considered of high quality because they met all four quality criteria. Most feedback comments were of moderate quality and met only two to three quality criteria. Regarding the CanMEDS roles, the Medical Expert role was most frequently represented in the feedback comments, as opposed to the roles Leader and Health Advocate. Discussion The results highlighted that providing high-quality feedback is challenging. To respond to these challenges, it is recommended to set up individual and continuous feedback training.
Collapse
Affiliation(s)
- Sofie Van Ostaeyen
- Department of Educational Sciences at Ghent University in Belgium, Belgium
| | - Mieke Embo
- Department of Nursing and Midwifery at the University of Antwerp, Belgium
- Department of Educational Sciences at Ghent University and in the Expertise Network Health and Care at the Artevelde University of Applied Sciences in Belgium, Belgium
| | - Tijs Rotsaert
- Department of Educational Sciences at Ghent University in Belgium, Belgium
| | - Orphée De Clercq
- Language and Translation Technology Team at Ghent University in Belgium, Belgium
| | - Tammy Schellens
- Department of Educational Sciences at Ghent University in Belgium, Belgium
| | - Martin Valcke
- Department of Educational Sciences at Ghent University in Belgium, Belgium
| |
Collapse
|
3
|
Van Ostaeyen S, De Langhe L, De Clercq O, Embo M, Schellens T, Valcke M. Automating the Identification of Feedback Quality Criteria and the CanMEDS Roles in Written Feedback Comments Using Natural Language Processing. PERSPECTIVES ON MEDICAL EDUCATION 2023; 12:540-549. [PMID: 38144670 PMCID: PMC10742245 DOI: 10.5334/pme.1056] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/14/2023] [Accepted: 10/03/2023] [Indexed: 12/26/2023]
Abstract
Introduction Manually analysing the quality of large amounts of written feedback comments is time-consuming and demands extensive resources and human effort. Therefore, this study aimed to explore whether a state-of-the-art large language model (LLM) could be fine-tuned to identify the presence of four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional) in written feedback comments. Methods A set of 2,349 labelled feedback comments of five healthcare educational programs in Flanders (Belgium) (specialistic medicine, general practice, midwifery, speech therapy and occupational therapy) was split into 12,452 sentences to create two datasets for the machine learning analysis. The Dutch BERT models BERTje and RobBERT were used to train four multiclass-multilabel classification models: two to identify the four feedback quality criteria and two to identify the seven CanMEDS roles. Results The classification models trained with BERTje and RobBERT to predict the presence of the four feedback quality criteria attained macro average F1-scores of 0.73 and 0.76, respectively. The F1-score of the model predicting the presence of the CanMEDS roles trained with BERTje was 0.71 and 0.72 with RobBERT. Discussion The results showed that a state-of-the-art LLM is able to identify the presence of the four feedback quality criteria and the CanMEDS roles in written feedback comments. This implies that the quality analysis of written feedback comments can be automated using an LLM, leading to savings of time and resources.
Collapse
Affiliation(s)
| | - Loic De Langhe
- Language and Translation Technology Team at Ghent University, Belgium
| | - Orphée De Clercq
- Language and Translation Technology Team at Ghent University, Belgium
| | - Mieke Embo
- Department of Educational Sciences at Ghent University and in the Expertise Network Health and Care at the Artevelde University of Applied Sciences, Belgium
| | - Tammy Schellens
- Department of Educational Sciences at Ghent University, Belgium
| | - Martin Valcke
- Department of Educational Sciences at Ghent University, Belgium
| |
Collapse
|
4
|
Anderson LM, Rowland K, Edberg D, Wright KM, Park YS, Tekian A. An Analysis of Written and Numeric Scores in End-of-Rotation Forms from Three Residency Programs. PERSPECTIVES ON MEDICAL EDUCATION 2023; 12:497-506. [PMID: 37929204 PMCID: PMC10624145 DOI: 10.5334/pme.41] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/21/2022] [Accepted: 10/24/2023] [Indexed: 11/07/2023]
Abstract
Introduction End-of-Rotation Forms (EORFs) assess resident progress in graduate medical education and are a major component of Clinical Competency Committee (CCC) discussion. Single-institution studies suggest EORFs can detect deficiencies, but both grades and comments skew positive. In this study, we sought to determine whether the EORFs from three programs, including multiple specialties and institutions, produced useful information for residents, program directors, and CCCs. Methods Evaluations from three programs were included (Program 1, Institution A, Internal Medicine: n = 38; Program 2, Institution A, Anesthesia: n = 9; Program 3, Institution B, Anesthesia: n = 11). Two independent researchers coded each written comment for relevance (specificity and actionability) and orientation (praise or critical) using a standardized rubric. Numeric scores were analyzed using descriptive statistics. Results 4869 evaluations were collected from the programs. Of the 77,434 discrete numeric scores, 691 (0.89%) were considered "below expected level." 71.2% (2683/3767) of the total written comments were scored as irrelevant, while 3217 (85.4%) of total comments were scored positive and 550 (14.6%) were critical. When combined, 63.2% (n = 2379) of comments were scored positive and irrelevant while 6.5% (n = 246) were scored critical and relevant. Discussion <1% of comments indicated below average performance; >70% of comments scored irrelevant. Critical, relevant comments were least frequently observed, consistent across all 3 programs. The low rate of constructive feedback and the high rate of irrelevant comments are inadequate for a CCC to make informed decisions. The consistency of these findings across programs, specialties, and institutions suggests both local and systemic changes should be considered.
Collapse
Affiliation(s)
- Lauren M. Anderson
- Department of Family and Preventive Medicine, Rush University, Chicago, Illinois, US
| | - Kathleen Rowland
- Department of Family and Preventive Medicine, Rush University, Chicago, Illinois, US
| | - Deborah Edberg
- Department of Family and Preventive Medicine, Rush University, Chicago, Illinois, US
| | - Katherine M. Wright
- Department of Family & Community Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, US
| | - Yoon Soo Park
- Department of Medical Education, University of Illinois Chicago, Chicago, Illinois, US
| | - Ara Tekian
- Department of Medical Education, University of Illinois Chicago, Chicago, Illinois, US
| |
Collapse
|
5
|
Figg BJ, Bolen O, Wagner MJ, Hicks T, Santen SA. Resident Physician Interactions and Engagement With Written Assessments of Performance. Fam Med 2023; 55:103-106. [PMID: 36689448 PMCID: PMC10614538 DOI: 10.22454/fammed.2022.587346] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
BACKGROUND AND OBJECTIVES Despite the requirements of the Accreditation Council for Graduate Medical Education (ACGME) to provide feedback, assessments are often not meeting the needs of resident learners. The objective of this study was to explore residents' approach to reviewing, interpreting, and incorporating the feedback provided in written faculty assessments. METHODS We conducted semistructured interviews with 14 family medicine residents. We used line-by-line iterative coding of the transcripts through the constant comparative method to identify themes and reach a consensus. RESULTS The study revealed the following themes: (1) residents value the narrative portion of assessments over numerical ratings, (2) performance reflection and reaction are part of the feedback process, (3) residents had difficulty incorporating formal assessments as many did not provide actionable feedback. CONCLUSIONS Residents reported that narrative feedback gives more insight to performance and leads to actionable changes in behaviors. Programs should consider education for both faculty and residents on the usefulness, importance, and purpose of the ACGME Milestones in order to accurately determine resident competency and provide a summative assessment. Until the purpose of the ACGME Milestones is realized and utilized, it should be noted that the comment portion of evaluations will likely be the focus of the resident's interaction with their assessments.
Collapse
Affiliation(s)
- Bethany J. Figg
- Central Michigan University Medical Education PartnersSaginaw, MI
| | - Olivia Bolen
- Central Michigan University Medical Education PartnersSaginaw, MI
| | - Mary Jo Wagner
- Central Michigan University Medical Education PartnersSaginaw, MI
| | | | | |
Collapse
|
6
|
Chakroun M, Dion VR, Ouellet K, Graillon A, Désilets V, Xhignesse M, St-Onge C. Narrative Assessments in Higher Education: A Scoping Review to Identify Evidence-Based Quality Indicators. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1699-1706. [PMID: 35612917 DOI: 10.1097/acm.0000000000004755] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE Narrative comments are increasingly used in assessment to document trainees' performance and to make important decisions about academic progress. However, little is known about how to document the quality of narrative comments, since traditional psychometric analysis cannot be applied. The authors aimed to generate a list of quality indicators for narrative comments, to identify recommendations for writing high-quality narrative comments, and to document factors that influence the quality of narrative comments used in assessments in higher education. METHOD The authors conducted a scoping review according to Arksey & O'Malley's framework. The search strategy yielded 690 articles from 6 databases. Team members screened abstracts for inclusion and exclusion, then extracted numerical and qualitative data based on predetermined categories. Numerical data were used for descriptive analysis. The authors completed the thematic analysis of qualitative data with iterative discussions until they achieved consensus for the interpretation of the results. RESULTS After the full-text review of 213 selected articles, 47 were included. Through the thematic analysis, the authors identified 7 quality indicators, 12 recommendations for writing quality narratives, and 3 factors that influence the quality of narrative comments used in assessment. The 7 quality indicators are (1) describes performance with a focus on particular elements (attitudes, knowledge, skills); (2) provides a balanced message between positive elements and elements needing improvement; (3) provides recommendations to learners on how to improve their performance; (4) compares the observed performance with an expected standard of performance; (5) provides justification for the mark/score given; (6) uses language that is clear and easily understood; and (7) uses a nonjudgmental style. CONCLUSIONS Assessors can use these quality indicators and recommendations to write high-quality narrative comments, thus reinforcing the appropriate documentation of trainees' performance, facilitating solid decision making about trainees' progression, and enhancing the impact of narrative feedback for both learners and programs.
Collapse
Affiliation(s)
- Molk Chakroun
- M. Chakroun is a PhD student, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Québec, Canada; ORCID: https://orcid.org/0000-0002-0518-1782
| | - Vincent R Dion
- V.R. Dion was research assistant, Paul Grand'Maison de la Société des médecins de l'Université de Sherbrooke Research Chair in Medical Education, at the time of this work, and is now a first-year medical student, Université de Sherbrooke, Sherbrooke, Québec, Canada
| | - Kathleen Ouellet
- K. Ouellet is research coordinator, Centre de pédagogie et des sciences de la santé, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Québec, Canada; ORCID: https://orcid.org/0000-0001-9829-151X
| | - Ann Graillon
- A. Graillon is associate professor, Department of Pediatrics, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Québec, Canada; ORCID: https://orcid.org/0000-0003-3677-7113
| | - Valérie Désilets
- V. Désilets is associate professor, Department of Pediatrics, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Québec, Canada; ORCID: https://orcid.org/0000-0001-7399-119X
| | - Marianne Xhignesse
- M. Xhignesse is full professor, Department of Family and Emergency Medicine, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Québec, Canada; ORCID: https://orcid.org/0000-0002-3257-5912
| | - Christina St-Onge
- C. St-Onge is full professor, Department of Medicine, Faculty of Medicine and Health Sciences, Université de Sherbrooke, and holds the Paul Grand'Maison de la Société des médecins de l'Université de Sherbrooke Research Chair in Medical Education, Sherbrooke, Québec, Canada; ORCID: https://orcid.org/0000-0001-5313-0456
| |
Collapse
|
7
|
Menchetti I, Eagles D, Ghanem D, Leppard J, Fournier K, Cheung WJ. Gender differences in emergency medicine resident assessment: A scoping review. AEM EDUCATION AND TRAINING 2022; 6:e10808. [PMID: 36189450 PMCID: PMC9513437 DOI: 10.1002/aet2.10808] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/26/2022] [Revised: 09/05/2022] [Accepted: 09/06/2022] [Indexed: 05/26/2023]
Abstract
Background Growing literature within postgraduate medical education demonstrates that female resident physicians experience gender bias throughout their training and future careers. This scoping review aims to describe the current body of literature on gender differences in emergency medicine (EM) resident assessment. Methods We conducted a scoping review which adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. We included research involving resident physicians or fellows in EM (population and context), which focused on the impact of gender on assessments (concept). We searched seven databases from the databases' inception to April 4, 2022. Two reviewers independently screened citations, completed full-text review, and abstracted data. A third reviewer resolved any discrepancies. Results A total of 667 unique citations were identified; 10 studies were included, and all were conducted within the United States. Four studies reported differences in EM resident assessments attributable to gender within workplace-based assessments (qualitative comments and quantitative scores) by both attending physicians and nonphysicians. Six studies investigating clinical competency committee scores, procedural scores, and simulation-based assessments did not report any significant differences attributable to gender. Conclusions This scoping review found that gender bias exists within EM resident assessment most notably at the level of narrative comments typically received via workplace-based assessments. As female EM residents receive higher rates of negative or critical comments and discordant feedback documented on assessment, these findings raise concern about added barriers female EM residents may face while progressing through residency and the impact on their clinical and professional development.
Collapse
Affiliation(s)
| | - Debra Eagles
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
- School of Epidemiology and Public HealthUniversity of OttawaOttawaOntarioCanada
| | - Dana Ghanem
- Faculty of MedicineUniversity of OttawaOttawaOntarioCanada
| | - Jennifer Leppard
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| | - Karine Fournier
- Health Sciences LibraryUniversity of OttawaOttawaOntarioCanada
| | - Warren J. Cheung
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
- Royal College of Physicians and Surgeons of CanadaOttawaOntarioCanada
| |
Collapse
|
8
|
Gordon LB, Zelaya-Floyd M, White P, Hallen S, Varaklis K, Tavakolikashi M. Interprofessional bedside rounding improves quality of feedback to resident physicians. MEDICAL TEACHER 2022; 44:907-913. [PMID: 35373712 DOI: 10.1080/0142159x.2022.2049735] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE Obtaining high quality feedback in residency education is challenging, in part due to limited opportunities for faculty observation of authentic clinical work. This study reviewed the impact of interprofessional bedside rounds ('iPACE™') on the length and quality of faculty narrative evaluations of residents as compared to usual inpatient teaching rounds. METHODS Narrative comments from faculty evaluations of Internal Medicine (IM) residents both on usual teaching service as well as the iPACE™ service (spanning 2017-2020) were reviewed and coded using a deductive content analysis approach. RESULTS Six hundred ninety-two narrative evaluations by 63 attendings of 103 residents were included. Evaluations of iPACE™ residents were significantly longer than those of residents on usual teams (109 vs. 69 words, p < 0.001). iPACE™ evaluations contained a higher average occurrence of direct observations of patient/family interactions (0.72 vs. 0.32, p < 0.001), references to interprofessionalism (0.17 vs. 0.05, p < 0.001), as well as specific (3.21 vs. 2.26, p < 0.001), actionable (1.01 vs. 0.69, p < 0.001), and corrective feedback (1.2 vs. 0.88, p = 0.001) per evaluation. CONCLUSIONS This study suggests that the iPACE™ model, which prioritizes interprofessional bedside rounds, had a positive impact on the quantity and quality of feedback, as measured via narrative comments on weekly evaluations.
Collapse
Affiliation(s)
- Lesley B Gordon
- Tufts University School of Medicine, Boston, MA, USA
- Department of Medicine, Maine Medical Center, Portland, ME, USA
| | | | - Patricia White
- Department of Medical Education, Maine Medical Center, Portland, ME, USA
| | - Sarah Hallen
- Tufts University School of Medicine, Boston, MA, USA
- Division of Geriatrics, Maine Medical Center, Portland, ME, USA
| | - Kalli Varaklis
- Tufts University School of Medicine, Boston, MA, USA
- Department of Medical Education, Maine Medical Center, Portland, ME, USA
- Department of Obstetrics and Gynecology, Maine Medical Center, Portland, ME, USA
| | - Motahareh Tavakolikashi
- Department of Medical Education, Maine Medical Center, Portland, ME, USA
- Department of System Science and Industrial Engineering, Binghamton University, Binghamton, NY, USA
| |
Collapse
|
9
|
Maranich AM, Hemmer PA, Uijtdehaage S, Battista A. ACGME Milestones in the Real World: A Qualitative Study Exploring Response Process Evidence. J Grad Med Educ 2022; 14:201-209. [PMID: 35463179 PMCID: PMC9017262 DOI: 10.4300/jgme-d-21-00546.1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/22/2021] [Revised: 10/02/2021] [Accepted: 01/05/2022] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Since the Accreditation Council for Graduate Medical Education (ACGME) introduced the Milestones in 2013, the body of validity evidence supporting their use has grown, but there is a gap with regard to response process. OBJECTIVE The purpose of this study is to qualitatively explore validity evidence pertaining to the response process of individual Clinical Competency Committee (CCC) members when assigning Milestone ratings to a resident. METHODS Using a constructivist paradigm, we conducted a thematic analysis of semi-structured interviews with 8 Transitional Year (TY) CCC members from 4 programs immediately following a CCC meeting between November and December 2020. Participants were queried about their response process in their application of Milestone assessment. Analysis was iterative, including coding, constant comparison, and theming. RESULTS Participant interviews identified an absence of formal training and a perception that Milestones are a tool for resident assessment without recognizing their role in program evaluation. In describing their thought process, participants reported comparing averaged assessment data to peers and time in training to generate Milestone ratings. Meaningful narrative comments, when available, differentiated resident performance from peers. When assessment data were absent, participants assumed an average performance. CONCLUSIONS Our study found that the response process used by TY CCC members was not always consistent with the dual purpose of the Milestones to improve educational outcomes at the levels of residents and the program.
Collapse
Affiliation(s)
- Ashley M. Maranich
- All authors are with the Uniformed Services University
- Ashley M. Maranich, MD, is Assistant Dean for Clinical Sciences and Associate Professor of Pediatrics
| | - Paul A. Hemmer
- All authors are with the Uniformed Services University
- Paul A. Hemmer, MD, MPH, is Professor of Medicine and Professor of Health Professions Education
| | - Sebastian Uijtdehaage
- All authors are with the Uniformed Services University
- Sebastian Uijtdehaage, PhD, is Professor of Medicine and Professor of Health Professions Education, and Associate Editor, Journal of Graduate Medical Education
| | - Alexis Battista
- All authors are with the Uniformed Services University
- Alexis Battista, PhD, is Assistant Professor of Medicine
| |
Collapse
|
10
|
Van Melle E, Hall AK, Schumacher DJ, Kinnear B, Gruppen L, Thoma B, Caretta-Weyer H, Cooke LJ, Frank JR. Capturing outcomes of competency-based medical education: The call and the challenge. MEDICAL TEACHER 2021; 43:794-800. [PMID: 34121596 DOI: 10.1080/0142159x.2021.1925640] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?
Collapse
Affiliation(s)
- Elaine Van Melle
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Family Medicine, Queen's University, Kingston, Canada
| | - Andrew K Hall
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Emergency Medicine, Queen's University, Kingston,Canada
| | - Daniel J Schumacher
- Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Benjamin Kinnear
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Larry Gruppen
- Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI, USA
| | - Brent Thoma
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Canada
| | - Holly Caretta-Weyer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, USA
| | - Lara J Cooke
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Clinical Neurosciences, Division of Neurology, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Jason R Frank
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
| |
Collapse
|
11
|
Edgar L, Jones MD, Harsy B, Passiment M, Hauer KE. Better Decision-Making: Shared Mental Models and the Clinical Competency Committee. J Grad Med Educ 2021; 13:51-58. [PMID: 33936533 PMCID: PMC8078083 DOI: 10.4300/jgme-d-20-00850.1] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/31/2020] [Revised: 11/13/2020] [Accepted: 12/01/2020] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Shared mental models (SMMs) help groups make better decisions. Clinical competency committees (CCCs) can benefit from the development and use of SMMs in their decision-making as a way to optimize the quality and consistency of their decisions. OBJECTIVE We reviewed the use of SMMs for decision making in graduate medical education, particularly their use in CCCs. METHODS In May 2020, the authors conducted a narrative review of the literature related to SMMs. This review included the SMM related to teams, team functioning, CCCs, and graduate medical education. RESULTS The literature identified the general use of SMMs, SMMs in graduate medical education, and strategies for building SMMs into the work of the CCC. Through the use of clear communication and guidelines, and a shared understanding of goals and expectations, CCCs can make better decisions. SMMs can be applied to Milestones, resident performance, assessment, and feedback. CONCLUSIONS To ensure fair and robust decision-making, the CCC must develop and maintain SMMs through excellent communication and understanding of expectations among members.
Collapse
Affiliation(s)
- Laura Edgar
- Laura Edgar, EdD, CAE, is Vice President, Milestones Development, Accreditation Council for Graduate Medical Education (ACGME)
| | - M. Douglas Jones
- M. Douglas Jones Jr, MD, is Professor of Pediatrics, University of Colorado School of Medicine
| | - Braden Harsy
- Braden Harsy, MA, is Milestones Administrator, ACGME
| | - Morgan Passiment
- Morgan Passiment, MS, is Director, Institutional Outreach and Collaboration, ACGME
| | - Karen E. Hauer
- Karen E. Hauer, MD, PhD, is Associate Dean, Competency Assessment and Professional Standards, and Professor of Medicine, University of California, San Francisco
| |
Collapse
|
12
|
Affiliation(s)
- Andem Ekpenyong
- Andem Ekpenyong, MD, MHPE, is Associate Professor, Department of Internal Medicine, Rush University Medical Center
| | - Marygrace Zetkulic
- Marygrace Zetkulic, MD, is Program Director, Internal Medicine, Hackensack University Medical Center
| | - Laura Edgar
- Laura Edgar, EdD, CAE, is Vice President, Milestones Development, Accreditation Council for Graduate Medical Education (ACGME)
| | - Eric S. Holmboe
- Eric S. Holmboe, MD, MACP, FRCP, is Chief Research, Milestone Development, and Evaluation Officer, ACGME
| |
Collapse
|
13
|
Branfield Day L, Miles A, Ginsburg S, Melvin L. Resident Perceptions of Assessment and Feedback in Competency-Based Medical Education: A Focus Group Study of One Internal Medicine Residency Program. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:1712-1717. [PMID: 32195692 DOI: 10.1097/acm.0000000000003315] [Citation(s) in RCA: 50] [Impact Index Per Article: 12.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
PURPOSE As key participants in the assessment dyad, residents must be engaged with the process. However, residents' experiences with competency-based medical education (CBME), and specifically with entrustable professional activity (EPA)-based assessments, have not been well studied. The authors explored junior residents' perceptions regarding the implementation of EPA assessment and feedback initiatives in an internal medicine program. METHOD From May to November 2018, 5 focus groups were conducted with 28 first-year internal medicine residents from the University of Toronto, exploring their experiences with facilitators and barriers to EPA-based assessments in the first years of the CBME initiative. Residents were exposed to EPA-based feedback tools from early in residency. Themes were identified using constructivist grounded theory to develop a framework to understand the resident perception of EPA assessment and feedback initiatives. RESULTS Residents' discussions reflected a growth mindset orientation, as they valued the idea of meaningful feedback through multiple low-stakes assessments. However, in practice, feedback seeking was onerous. While the quantity of feedback had increased, the quality had not; some residents felt it had worsened, by reducing it to a form-filling exercise. The assessments were felt to have increased daily workload with consequent disrupted workflow and to have blurred the lines between formative and summative assessment. CONCLUSIONS Residents embraced the driving principles behind CBME, but their experience suggested that changes are needed for CBME in the study site program to meet its goals. Efforts may be needed to reconcile the tension between assessment and feedback and to effectively embed meaningful feedback into CBME learning environments.
Collapse
Affiliation(s)
- Leora Branfield Day
- L. Branfield Day is a fourth-year chief medical resident, internal medicine training program, Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Amy Miles
- A. Miles is a fourth-year resident, geriatric medicine subspecialty training program, Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Shiphra Ginsburg
- S. Ginsburg is staff physician, Division of Respirology, Mount Sinai Hospital and University Health Network, professor, Department of Medicine, Faculty of Medicine, University of Toronto, and scientist, Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, Canada
| | - Lindsay Melvin
- L. Melvin is assistant professor, Department of Medicine, Faculty of Medicine, University of Toronto, and staff physician, Division of General Internal Medicine, University Health Network, Toronto, Ontario, Canada
| |
Collapse
|
14
|
Feinsilber D, Siripala DS, Mears KA. Review of Cognitive Biases in ACGME Milestones Training Assessments in Post-graduate Medical Education Programs. Cureus 2019; 11:e5518. [PMID: 31523586 PMCID: PMC6721883 DOI: 10.7759/cureus.5518] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022] Open
Abstract
The course of study for young physicians for post-graduate training is an exciting and life-changing opportunity, one that is filled with the relentless optimism of intellectual discovery and personal growth and development. The American Council for Graduate Medical Education (ACGME) is a non-profit private council that evaluates and accredits internship, residency, and fellowship programs. The role of the ACGME is to oversee curriculums, training environments, and specialty evaluation standards to ensure satisfactory competency leading to board eligibility and certification in the respected field of study. The ACGME has the monumental task of guiding educational standards that are designed to both protect the public welfare and further educational programs. Many educational standards are objective, such as quantitative performance on examinations, involvement in research, and involvement in systems development and quality improvement. However, key clinical performance measures are based on prior training and experience. Over the last several years, studies examining rates of abuse and discrimination during post-graduate medical training in both the United States and Canadian studies, which have reported alarmingly high rates of 50%. With the increasing utility and availability of social media, such issues have become more transparent to the public. A plethora of studies has been conducted, examining physician biases towards patients, practice changes, insurance company regulations, and evolving healthcare systems. However, a significant amount of evaluation is merited when examining individual institutional cultures and the educational environments that harbor them. We wish to examine the role of ever-evolving specialty-specific ACGME-instituted educational milestones in Internal Medicine and Opthalmology in the context of potential cognitive biases and their implementation within post-graduate training programs.
Collapse
Affiliation(s)
- Doron Feinsilber
- Hematology/Oncology, Medical College of Wisconsin/Froedert Cancer Center, Milwaukee, USA
| | - Duminda S Siripala
- Nephrology, University of Pittsburgh Medical Center Altoona, Altoona, USA
| | - Katrina A Mears
- Ophthalmology, Retina Consultants of Southwest Florida/National Ophthalmic Research Institute, Fort Myers, USA
| |
Collapse
|