1
|
Almendros A, Steagall PV, Lun SC, Speelman J, Giuliano A. Challenges of the Calgary-Cambridge Consultation Guide in Veterinary Multicultural and Multilingual Scenarios and the Role of Veterinary Translators. Animals (Basel) 2024; 14:2270. [PMID: 39123795 PMCID: PMC11310950 DOI: 10.3390/ani14152270] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2024] [Revised: 07/22/2024] [Accepted: 07/31/2024] [Indexed: 08/12/2024] Open
Abstract
The Calgary-Cambridge Guide is a widely recognised framework for teaching communication skills to healthcare professionals that has become a cornerstone of communication training programs in medicine and other healthcare fields. In the context of veterinary medicine, its integration into communication training programs has become an asset improving communication, education, interaction, and quality of service, enhancing the veterinary-client-patient relationship (VCPR). In veterinary medicine, however, a more challenging consultation dynamic involves the veterinarian, the owner, and the animal. The addition of a veterinary assistant that acts as an interpreter or translator is common in Hong Kong where the native language (Cantonese) coexists with English when consultations are led by non-native language speakers. This addition converts this commonly dyadic model into a triadic communication model. The addition of an assistant interpreter influences the way consultations are conducted, how information is conveyed, and how interpersonal cues and empathy are delivered. In this report we depict challenges applying the Calgary-Cambridge Guide in multicultural and multilingual veterinary medical centres in Hong Kong and highlight the role of veterinary supporting staff in these scenarios, specifically veterinary assistant interpreters.
Collapse
Affiliation(s)
- Angel Almendros
- Department of Veterinary Clinical Sciences, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong SAR, China
- Veterinary Medical Centre, City University of Hong Kong, Hong Kong SAR, China
| | - Paulo V. Steagall
- Department of Veterinary Clinical Sciences, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong SAR, China
- Centre of Animal Health and Welfare, City University of Hong Kong, Hong Kong SAR, China
| | - Suen Caesar Lun
- Department of Linguistics and Translation, College of Liberal Arts and Social Sciences, City University of Hong Kong, Hong Kong SAR, China
| | | | - Antonio Giuliano
- Department of Veterinary Clinical Sciences, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong SAR, China
- Veterinary Medical Centre, City University of Hong Kong, Hong Kong SAR, China
| |
Collapse
|
2
|
Sahi N, Humphrey-Murto S, Brennan EE, O'Brien M, Hall AK. Current use of simulation for EPA assessment in emergency medicine. CAN J EMERG MED 2024; 26:179-187. [PMID: 38374281 DOI: 10.1007/s43678-024-00649-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Accepted: 01/12/2024] [Indexed: 02/21/2024]
Abstract
OBJECTIVE Approximately five years ago, the Royal College emergency medicine programs in Canada implemented a competency-based paradigm and introduced the use of Entrustable Professional Activities (EPAs) for assessment of units of professional activity to assess trainees. Many competency-based medical education (CBME) based curricula, involve assessing for entrustment through observations of EPAs. While EPAs are frequently assessed in clinical settings, simulation is also used. This study aimed to characterize the use of simulation for EPA assessment. METHODS A study interview guide was jointly developed by all study authors and followed best practices for survey development. A national interview was conducted with program directors or assistant program directors across all the Royal College emergency medicine programs across Canada. Interviews were conducted over Microsoft Teams, interviews were recorded and transcribed, using Microsoft Teams transcribing service. Sample transcripts were analyzed for theme development. Themes were then reviewed by co-authors to ensure they were representative of the participants' views. RESULTS A 64.7% response rate was achieved. Simulation has been widely adopted by EM training programs. All interviewees demonstrated support for the use of simulation for EPA assessment for many reasons, however, PDs acknowledged limitations and thematic analysis revealed certain themes and tensions for using simulation for EPA assessment. Thematic analysis revealed six major themes: widespread support for the use of simulation for EPA assessment, concerns regarding the potential for EPA assessment to become a "tick- box" exercise, logistical barriers limiting the use of simulation for EPA assessment, varied perceptions about the authenticity of using simulation for EPA assessment, the potential for simulation for EPA assessment to compromise learner psychological safety, and suggestions for the optimization of use of simulation for EPA assessment. CONCLUSIONS Our findings offer insight for other programs and specialties on how simulation for EPA assessment can best be utilized. Programs should use these findings when considering using simulation for EPA assessment.
Collapse
Affiliation(s)
- Nidhi Sahi
- Department of Innovation in Medical Education (DIME), University of Ottawa, Ottawa, ON, Canada.
| | - Susan Humphrey-Murto
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
- Tier 2 Research Chair in Medical Education and Fellowship Director, Medical Education Research, University of Ottawa, Ottawa, ON, Canada
| | - Erin E Brennan
- Department of Emergency Medicine, Queen's University, Kingston, ON, Canada
| | - Michael O'Brien
- Emergency Medicine, The Ottawa Hospital, Ottawa, ON, Canada
- Department of Innovation in Medical Education, University of Ottawa, Ottawa, ON, Canada
| | - Andrew K Hall
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| |
Collapse
|
3
|
Sikon JR, Hess EP, Woods R. Simulation for assessment: ready for prime time? CAN J EMERG MED 2024; 26:137-138. [PMID: 38436909 DOI: 10.1007/s43678-024-00661-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/05/2024]
Affiliation(s)
- Joseph R Sikon
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, USA.
| | - Erik P Hess
- Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, USA
| | - Rob Woods
- Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Canada
| |
Collapse
|
4
|
Allen L, Hall AK, Braund H, Chaplin T. Catalytic effect of multisource feedback for trauma team captains: a mixed-methods prospective study. BMJ Open 2023; 13:e068732. [PMID: 37221034 DOI: 10.1136/bmjopen-2022-068732] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/25/2023] Open
Abstract
OBJECTIVES To evaluate the impact and feasibility of multisource feedback compared with traditional feedback for trauma team captains (TTCs). DESIGN A mixed-methods, non-randomised prospective study. SETTING A level one trauma centre in Ontario, Canada. PARTICIPANTS Postgraduate medical residents in emergency medicine and general surgery participating as TTCs. Selection was based on a convenience sampling method. INTERVENTION Postgraduate medical residents participating as TTCs received either multisource feedback or standard feedback following trauma cases. MAIN OUTCOME MEASURES TTCs completed questionnaires designed to measure the self-reported intention to change practice (catalytic effect), immediately following a trauma case and 3 weeks later. Secondary outcomes included measures of perceived benefit, acceptability, and feasibility from TTCs and other trauma team members. RESULTS Data were collected following 24 trauma team activations: TTCs from 12 activations received multisource feedback and 12 received standard feedback. The self-reported intention for practice change was not significantly different between groups initially (4.0 vs 4.0, p=0.57) and at 3 weeks (4.0 vs 3.0, p=0.25). Multisource feedback was perceived to be helpful and superior to the existing feedback process. Feasibility was identified as a challenge. CONCLUSIONS The self-reported intention for practice change was no different for TTCs who received multisource feedback and those who received standard feedback. Multisource feedback was favourably received by trauma team members, and TTCs perceived multisource feedback as useful for their development.
Collapse
Affiliation(s)
- Leah Allen
- Department of Emergency Medicine, Queen's University, Kingston, Ontario, Canada
| | - Andrew K Hall
- Department of Emergency Medicine, University of Ottawa Faculty of Medicine, Ottawa, Ontario, Canada
| | - Heather Braund
- Office of Professional Development and Educational Scholarship, Queen's University Faculty of Health Sciences, Kingston, Ontario, Canada
| | - Timothy Chaplin
- Department of Emergency Medicine, Queen's University, Kingston, Ontario, Canada
| |
Collapse
|
5
|
Mackenzie MJ, Hagel C, Lin Y, Hall AK, Grant VJ, Doshi S. The Reliability of the Resuscitation Assessment Tool (RAT) in Assessing Emergency Medicine Resident Competence in Pediatric Resuscitation Scenarios: A Prospective Observational Pilot Study. Cureus 2023; 15:e35869. [PMID: 37033538 PMCID: PMC10079254 DOI: 10.7759/cureus.35869] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/07/2023] [Indexed: 03/09/2023] Open
Abstract
Introduction Emergency medicine (EM) postgraduate medical education in Canada has transitioned from traditional time-based training to competency-based medical education (CBME). In order to promote residents through stages of training, simulated assessments are needed to evaluate residents in high-stakes but low-frequency medical emergencies. There remains a gap in the literature pertaining to the use of evaluative tools in simulation, such as the Resuscitation Assessment Tool (RAT) in the new CBME curriculum design. Methods We completed a pilot study of resident physicians in one Canadian EM training program to evaluate the effectiveness and reliability of a simulation-based RAT for pediatric resuscitation. We recorded 10 EM trainees completing simulated scenarios and had nine EM physicians use the RAT tool to evaluate their performances. Generalizability theory was used to evaluate the reliability of the RAT tool. Results The mean RAT score for the management of pediatric myocarditis, cardiac arrest, and septic shock (appendicitis) across raters was 3.70, 3.73, and 4.50, respectively. The overall generalizability coefficient for testing simulated pediatric performance competency was 0.77 for internal consistency and 0.75 for absolute agreement. The performance of senior participants was superior to that of junior participants in the management of pediatric myocarditis (p = 0.01) but not statistically significant in the management of pediatric septic shock (p=0.77) or cardiac arrest (p =0.61). Conclusion Overall, our findings suggest that with an appropriately chosen simulated scenario, the RAT tool can be used effectively for the simulation of high-stakes and low-frequency scenarios for practice to enhance the new CBME curriculum in emergency medicine training programs.
Collapse
|
6
|
Sheppard G, Williams KL, Metcalfe B, Clark M, Bromley M, Pageau P, Woo M, Yi Y, Devasahayam AJ, Dubrowski A. Using Kane's framework to build an assessment tool for undergraduate medical student's clinical competency with point of care ultrasound. BMC MEDICAL EDUCATION 2023; 23:43. [PMID: 36658642 PMCID: PMC9854184 DOI: 10.1186/s12909-023-04030-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 01/13/2023] [Indexed: 06/17/2023]
Abstract
INTRODUCTION Point-of-care ultrasonography (POCUS) is a portable imaging technology used in clinical settings. There is a need for valid tools to assess clinical competency in POCUS in medical students. The primary aim of this study was to use Kane's framework to evaluate an interpretation-use argument (IUA) for an undergraduate POCUS assessment tool. METHODS Participants from Memorial University of Newfoundland, the University of Calgary, and the University of Ottawa were recruited between 2014 and 2018. A total of 86 participants and seven expert raters were recruited. The participants performed abdominal, sub-xiphoid cardiac, and aorta POCUS scans on a volunteer patient after watching an instruction video. The participant-generated POCUS images were assessed by the raters using a checklist and a global rating scale. Kane's framework was used to determine validity evidence for the scoring inference. Fleiss' kappa was used to measure agreement between seven raters on five questions that reflected clinical competence. The descriptive comments collected from the raters were systematically coded and analyzed. RESULTS The overall agreement between the seven raters on five questions on clinical competency ranged from fair to moderate (κ = 0.32 to 0.55). The themes from the qualitative data were poor image generation and interpretation (22%), items not applicable (20%), poor audio and video quality (20%), poor probe handling (10%), and participant did not verbalize findings (14%). CONCLUSION The POCUS assessment tool requires further modification and testing prior before it can be used for reliable undergraduate POCUS assessment.
Collapse
Affiliation(s)
- Gillian Sheppard
- Discipline of Emergency Medicine, Faculty of Medicine, Memorial University of Newfoundland, St. John's, NL, Canada.
| | - Kerry-Lynn Williams
- Discipline of Emergency Medicine, Faculty of Medicine, Memorial University of Newfoundland, St. John's, NL, Canada
| | - Brian Metcalfe
- Discipline of Emergency Medicine, Faculty of Medicine, Memorial University of Newfoundland, St. John's, NL, Canada
| | - Marcia Clark
- Division of Orthopedic Surgery, University of Calgary, Calgary, Canada
| | - Mark Bromley
- Division of Emergency Medicine, University of Calgary, Calgary, Canada
| | - Paul Pageau
- Department of Emergency Medicine, University of Ottawa and Ottawa Hospital Research Institute, Ottawa, Canada
| | - Michael Woo
- Department of Emergency Medicine, University of Ottawa and Ottawa Hospital Research Institute, Ottawa, Canada
| | - Yanqing Yi
- Division of Community Health and Humanities, Faculty of Medicine, Memorial University of Newfoundland, St. John's, NL, Canada
| | | | - Adam Dubrowski
- Faculty of Health Sciences, Ontario Technology University, Oshawa, Canada
| |
Collapse
|
7
|
Woods R, Singh S, Thoma B, Patocka C, Cheung W, Monteiro S, Chan TM. Validity evidence for the Quality of Assessment for Learning score: a quality metric for supervisor comments in Competency Based Medical Education. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:19-35. [PMID: 36440075 PMCID: PMC9684040 DOI: 10.36834/cmej.74860] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
BACKGROUND Competency based medical education (CBME) relies on supervisor narrative comments contained within entrustable professional activities (EPA) for programmatic assessment, but the quality of these supervisor comments is unassessed. There is validity evidence supporting the QuAL (Quality of Assessment for Learning) score for rating the usefulness of short narrative comments in direct observation. OBJECTIVE We sought to establish validity evidence for the QuAL score to rate the quality of supervisor narrative comments contained within an EPA by surveying the key end-users of EPA narrative comments: residents, academic advisors, and competence committee members. METHODS In 2020, the authors randomly selected 52 de-identified narrative comments from two emergency medicine EPA databases using purposeful sampling. Six collaborators (two residents, two academic advisors, and two competence committee members) were recruited from each of four EM Residency Programs (Saskatchewan, McMaster, Ottawa, and Calgary) to rate these comments with a utility score and the QuAL score. Correlation between utility and QuAL score were calculated using Pearson's correlation coefficient. Sources of variance and reliability were calculated using a generalizability study. RESULTS All collaborators (n = 24) completed the full study. The QuAL score had a high positive correlation with the utility score amongst the residents (r = 0.80) and academic advisors (r = 0.75) and a moderately high correlation amongst competence committee members (r = 0.68). The generalizability study found that the major source of variance was the comment indicating the tool performs well across raters. CONCLUSION The QuAL score may serve as an outcome measure for program evaluation of supervisors, and as a resource for faculty development.
Collapse
Affiliation(s)
- Rob Woods
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Sim Singh
- College of Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Brent Thoma
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Catherine Patocka
- Department of Emergency Medicine, University of Calgary, Alberta, Canada
| | - Warren Cheung
- Department of Emergency Medicine, University of Ottawa, Ontario, Canada
| | - Sandra Monteiro
- Department of Health Research Methods Evidence and Impact, McMaster University, Ontario, Canada
| | - Teresa M Chan
- Division of Emergency Medicine and Education & Innovation, Department of Medicine, McMaster University, Ontario, Canada
| | | |
Collapse
|
8
|
Miller S, Caretta-Weyer H, Chan T. Beyond competence: rethinking continuing professional development in the age of competence-based medical education. CAN J EMERG MED 2022; 24:563-565. [PMID: 36071320 DOI: 10.1007/s43678-022-00372-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2022] [Accepted: 07/29/2022] [Indexed: 11/28/2022]
Affiliation(s)
- Stephen Miller
- Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada
| | - Holly Caretta-Weyer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California, USA
| | - Teresa Chan
- Division of Emergency Medicine, McMaster University, Hamilton, Ontario, Canada. .,Division of Education and Innovation, Department of Medicine, McMaster University, Hamilton, Ontario, Canada. .,Department of Health Research Methodologies, Evidence, and Impact (HEI), McMaster University, Hamilton, Ontario, Canada. .,McMaster Education Research, Innovation, and Theory (MERIT) Unit, McMaster University, Hamilton, Ontario, Canada. .,Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada. .,Adjunct Associate Professor (Honorary), The Chinese University of Hong Kong - Shenzhen campus, Shenzhen, China.
| |
Collapse
|
9
|
Yilmaz Y, Carey R, Chan TM, Bandi V, Wang S, Woods RA, Mondal D, Thoma B. Developing a dashboard for program evaluation in competency-based training programs: a design-based research project. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:14-27. [PMID: 36310899 PMCID: PMC9588183 DOI: 10.36834/cmej.73554] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
BACKGROUND Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation. METHODS Using a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations. RESULTS Between July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard. CONCLUSIONS We identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.
Collapse
Affiliation(s)
- Yusuf Yilmaz
- Continuing Professional Development Office, and McMaster program for Education Research, Innovation, and Theory (MERIT), McMaster University, Ontario, Canada
- Department of Medical Education, Ege University, Turkey
| | - Robert Carey
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Teresa M Chan
- Continuing Professional Development Office, and McMaster program for Education Research, Innovation, and Theory (MERIT), McMaster University, Ontario, Canada
- Division of Emergency Medicine, Department of Medicine at McMaster University
| | - Venkat Bandi
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Shisong Wang
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Robert A Woods
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Debajyoti Mondal
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Brent Thoma
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
- Royal College of Physicians and Surgeons of Canada, Ontario, Canada
| |
Collapse
|
10
|
Kinder F, Mehmood S, Hodgson H, Giannoudis P, Howard A. Barriers to Trauma Care in South and Central America: a systematic review. EUROPEAN JOURNAL OF ORTHOPAEDIC SURGERY & TRAUMATOLOGY : ORTHOPEDIE TRAUMATOLOGIE 2022; 32:1163-1177. [PMID: 34392445 PMCID: PMC9279262 DOI: 10.1007/s00590-021-03080-3] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/11/2021] [Accepted: 07/26/2021] [Indexed: 11/30/2022]
Abstract
INTRODUCTION Trauma is widespread in Central and South America and is a significant cause of morbidity and mortality. Providing high quality emergency trauma care is of great importance. Understanding the barriers to care is challenging; this systematic review aims to establish current the current challenges and barriers in providing high-quality trauma care within the 21 countries in the region. METHODS OVID Medline, Embase, EBM reviews and Global Health databases were systematically searched in October 2020. Records were screened by two independent researchers. Data were extracted according to a predetermined proforma. Studies of any type, published in the preceding decade were included, excluding grey literature and non-English records. Trauma was defined as blunt or penetrating injury from an external force. Studies were individually critically appraised and assessed for bias using the RTI item bank. RESULTS 57 records met the inclusion criteria. 20 countries were covered at least once. Nine key barriers were identified: training (37/57), resources and equipment (33/57), protocols (29/57), staffing (17/57), transport and logistics (16/57), finance (15/57), socio-cultural (13/57), capacity (9/57), public education (4/57). CONCLUSION Nine key barriers negatively impact on the provision of high-quality trauma care and highlight potential areas for improving care in Central & South America. Many countries in the region, along with rural areas, are under-represented by the current literature and future research is urgently required to assess barriers to trauma management in these countries. No funding was received. CLINICAL TRIAL REGISTRATION PROSPERO CRD42020220380.
Collapse
Affiliation(s)
| | | | | | - Peter Giannoudis
- LIRRM, Leeds University, Leeds, UK
- Leeds Orthopaedic Trauma Sciences, Leeds University, Leeds, UK
- Leeds General Infirmary, Leeds, UK
| | - Anthony Howard
- LIRRM, Leeds University, Leeds, UK
- Leeds Orthopaedic Trauma Sciences, Leeds University, Leeds, UK
- NDORMS, University of Oxford, Oxford, UK
- Academic T&O Unit, Clarendon Wing, D floor, Great George Street, Leeds, LS1 3EX UK
| |
Collapse
|
11
|
Landreville JM, Wood TJ, Frank JR, Cheung WJ. Does direct observation influence the quality of workplace-based assessment documentation? AEM EDUCATION AND TRAINING 2022; 6:e10781. [PMID: 35903424 PMCID: PMC9305723 DOI: 10.1002/aet2.10781] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Revised: 06/07/2022] [Accepted: 06/08/2022] [Indexed: 05/30/2023]
Abstract
Background A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). Methods The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. Results A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. Conclusions Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.
Collapse
Affiliation(s)
| | - Timothy J. Wood
- Department of Innovation in Medical EducationUniversity of OttawaOttawaOntarioCanada
| | - Jason R. Frank
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| | - Warren J. Cheung
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| |
Collapse
|
12
|
Chan TM, Dowling S, Tastad K, Chin A, Thoma B. Integrating training, practice, and reflection within a new model for Canadian medical licensure: a concept paper prepared for the Medical Council of Canada. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:68-81. [PMID: 36091730 PMCID: PMC9441128 DOI: 10.36834/cmej.73717] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
In 2020 the Medical Council of Canada created a task force to make recommendations on the modernization of its practices for granting licensure to medical trainees. This task force solicited papers on this topic from subject matter experts. As outlined within this Concept Paper, our proposal would shift licensure away from the traditional focus on high-stakes summative exams in a way that integrates training, clinical practice, and reflection. Specifically, we propose a model of graduated licensure that would have three stages including: a trainee license for trainees that have demonstrated adequate medical knowledge to begin training as a closely supervised resident, a transition to practice license for trainees that have compiled a reflective educational portfolio demonstrating the clinical competence required to begin independent practice with limitations and support, and a fully independent license for unsupervised practice for attendings that have demonstrated competence through a reflective portfolio of clinical analytics. This proposal was reviewed by a diverse group of 30 trainees, practitioners, and administrators in medical education. Their feedback was analyzed and summarized to provide an overview of the likely reception that this proposal would receive from the medical education community.
Collapse
Affiliation(s)
- Teresa M Chan
- Division of Emergency Medicine, Department of Medicine, Faculty of Health Sciences, McMaster University
- Division of Education & Innovation, Department of Medicine
- Faculty of Health Sciences, McMaster University; McMaster Education Research, Innovation, and Theory (MERIT) program
- Office of Continuing Professional Development; Faculty of Health Sciences, McMaster University
| | - Shawn Dowling
- Department of Emergency Medicine, Cumming School of Medicine, University of Calgary
| | - Kara Tastad
- Royal College Emergency Medicine Training Program, University of Toronto
| | - Alvin Chin
- Division of Emergency Medicine, Department of Medicine, Faculty of Health Sciences, McMaster University
| | - Brent Thoma
- Department of Emergency Medicine, University of Saskatchewan
| |
Collapse
|
13
|
Cheung WJ, Wagner N, Frank JR, Oswald A, Van Melle E, Skutovich A, Dalseg TR, Cooke LJ, Hall AK. Implementation of competence committees during the transition to CBME in Canada: A national fidelity-focused evaluation. MEDICAL TEACHER 2022; 44:781-789. [PMID: 35199617 DOI: 10.1080/0142159x.2022.2041191] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE This study evaluated the fidelity of competence committee (CC) implementation in Canadian postgraduate specialist training programs during the transition to competency-based medical education (CBME). METHODS A national survey of CC chairs was distributed to all CBME training programs in November 2019. Survey questions were derived from guiding documents published by the Royal College of Physicians and Surgeons of Canada reflecting intended processes and design. RESULTS Response rate was 39% (113/293) with representation from all eligible disciplines. Committee size ranged from 3 to 20 members, 42% of programs included external members, and 20% included a resident representative. Most programs (72%) reported that a primary review and synthesis of resident assessment data occurs prior to the meeting, with some data reviewed collectively during meetings. When determining entrustable professional activity (EPA) achievement, most programs followed the national specialty guidelines closely with some exceptions (53%). Documented concerns about professionalism, EPA narrative comments, and EPA entrustment scores were most highly weighted when determining resident progress decisions. CONCLUSIONS Heterogeneity in CC implementation likely reflects local adaptations, but may also explain some of the variable challenges faced by programs during the transition to CBME. Our results offer educational leaders important fidelity data that can help inform the larger evaluation and transformation of CBME.
Collapse
Affiliation(s)
- Warren J Cheung
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
| | - Natalie Wagner
- Office of Professional Development & Educational Scholarship and Department of Biomedical & Molecular Sciences, Queen's University, Kingston, Canada
| | - Jason R Frank
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
| | - Anna Oswald
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Medicine, University of Alberta, Edmonton, Canada
| | - Elaine Van Melle
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Family Medicine, Queen's University, Kingston, Canada
| | | | - Timothy R Dalseg
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Medicine, Division of Emergency Medicine, University of Toronto, Toronto, Canada
| | - Lara J Cooke
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Andrew K Hall
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
| |
Collapse
|
14
|
Yilmaz Y, Jurado Nunez A, Ariaeinejad A, Lee M, Sherbino J, Chan TM. Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education. JMIR MEDICAL EDUCATION 2022; 8:e30537. [PMID: 35622398 PMCID: PMC9187970 DOI: 10.2196/30537] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/19/2021] [Revised: 12/05/2021] [Accepted: 04/30/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. OBJECTIVE This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings. METHODS NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported. RESULTS The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%). CONCLUSIONS The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.
Collapse
Affiliation(s)
- Yusuf Yilmaz
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Department of Medical Education, Ege University, Izmir, Turkey
- Program for Faculty Development, Office of Continuing Professional Development, McMaster University, Hamilton, ON, Canada
- Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Alma Jurado Nunez
- Department of Medicine and Masters in eHealth Program, McMaster University, Hamilton, ON, Canada
| | - Ali Ariaeinejad
- Department of Medicine and Masters in eHealth Program, McMaster University, Hamilton, ON, Canada
| | - Mark Lee
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Jonathan Sherbino
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Division of Emergency Medicine, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Division of Education and Innovation, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Teresa M Chan
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Program for Faculty Development, Office of Continuing Professional Development, McMaster University, Hamilton, ON, Canada
- Division of Emergency Medicine, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Division of Education and Innovation, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| |
Collapse
|
15
|
Chaplin T, Braund H, Szulewski A, Dalgarno N, Egan R, Thoma B. Multi-source feedback following simulated resuscitation scenarios: a qualitative study. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:18-30. [PMID: 35572030 PMCID: PMC9099176 DOI: 10.36834/cmej.72387] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
BACKGROUND The direct observation and assessment of learners' resuscitation skills by an attending physician is challenging due to the unpredictable and time-sensitive nature of these events. Multisource feedback (MSF) may address this challenge and improve the quality of assessments provided to learners. We aimed to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a simulation-based resuscitation curriculum. METHODS We conducted a qualitative content analysis of narrative MSF of medical residents in their first postgraduate year of training who were participating in a simulation-based resuscitation course at two Canadian institutions. Assessments included an entrustment score and narrative comments from attending physicians, registered nurses, and resident peers in addition to self-assessment. Narrative comments were transcribed and analyzed thematically using a constant comparative method. RESULTS All 87 residents (100%) participating in the 2017-2018 course provided consent. A total of 223 assessments were included in our analysis. Four themes emerged from the narrative data: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. Relative to other assessor groups, feedback from nurses focused on patient-centred care and communication while attending physicians focused on the medical expert theme. Peer feedback was the most positive. Self-assessments included comments within each of the four themes. CONCLUSIONS In the context of a simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale and may offer a more holistic assessment of resuscitation skills within a competency-based medical education (CBME) program of assessment.
Collapse
Affiliation(s)
- Timothy Chaplin
- Department of Emergency Medicine, Queen’s University, Ontario, Canada
| | - Heather Braund
- Office of Professional Development and Educational Scholarship, Faculty of Health Sciences, Queen’s University, Ontario, Canada
| | - Adam Szulewski
- Department of Emergency Medicine, Queen’s University, Ontario, Canada
- Departments of Psychology, Queen’s University, Ontario, Canada
| | - Nancy Dalgarno
- Office of Professional Development and Educational Scholarship, Faculty of Health Sciences, Queen’s University, Ontario, Canada
| | - Rylan Egan
- Health Quality Programs, Faculty of Health Sciences, Queen’s University, Ontario, Canada
| | - Brent Thoma
- Department of Emergency Medicine, College of Medicine , University of Saskatchewan, Saskatchewan, Canada
| |
Collapse
|
16
|
Pandya A, Patocka C, Huffman J. Simulation for assessment of Entrustable Professional Activities in an emergency medicine residency program. CAN J EMERG MED 2022; 24:84-87. [PMID: 34780048 DOI: 10.1007/s43678-021-00209-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2021] [Accepted: 08/18/2021] [Indexed: 11/28/2022]
Abstract
In 2018, Canadian post-graduate Emergency Medicine (EM) programs transitioned to Competence-by-Design. Residents are now assessed using Entrustable Professional Activities (EPAs). We developed and implemented simulation for assessment to mitigate anticipated challenges with residents completing the required number of observations of resuscitation-based EPAs. Our survey of trainees who participated in these sessions suggests that it may be a feasible and acceptable method for EPA assessment.
Collapse
Affiliation(s)
- Anjli Pandya
- Department of Emergency Medicine, Rm C231, Foothills Medical Centre, University of Calgary, 1403 29th Street NW, Calgary, AB, T2N 2T9, Canada.
| | - Catherine Patocka
- Department of Emergency Medicine, Rm C231, Foothills Medical Centre, University of Calgary, 1403 29th Street NW, Calgary, AB, T2N 2T9, Canada
| | - James Huffman
- Department of Emergency Medicine, Rm C231, Foothills Medical Centre, University of Calgary, 1403 29th Street NW, Calgary, AB, T2N 2T9, Canada
| |
Collapse
|
17
|
Ng C, Primiani N, Orchanian-Cheff A. Rapid Cycle Deliberate Practice in Healthcare Simulation: a Scoping Review. MEDICAL SCIENCE EDUCATOR 2021; 31:2105-2120. [PMID: 34950533 PMCID: PMC8651942 DOI: 10.1007/s40670-021-01446-0] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Accepted: 10/13/2021] [Indexed: 06/14/2023]
Abstract
Rapid cycle deliberate practice (RCDP) is a type of simulation-based medical education (SBME) where learners cycle between deliberate practice and directed feedback until skill mastery is achieved before progressing to subsequent learning objectives. This scoping review examines and summarizes the literature on RCDP, compares RCDP to other modes of instruction, and identifies knowledge gaps for future research. Of the 1224 articles identified, 23 studies met inclusion criteria. The studies varied in design, RCDP technique implementation strategies, and outcome measures. RCDP is associated with positive outcomes in immediate learner performance. It is unclear if RCDP is superior to traditional simulation.
Collapse
Affiliation(s)
- Carly Ng
- Division of Emergency Medicine, Department of Family and Community Medicine, University of Toronto, 200 Elizabeth Street, Toronto, ON M5G 2C4 Canada
| | - Nadia Primiani
- Division of Emergency Medicine, Department of Family and Community Medicine, University of Toronto, 200 Elizabeth Street, Toronto, ON M5G 2C4 Canada
| | - Ani Orchanian-Cheff
- Library and Information Services, University Health Network, Toronto, ON Canada
| |
Collapse
|
18
|
Torre D, Rice NE, Ryan A, Bok H, Dawson LJ, Bierer B, Wilkinson TJ, Tait GR, Laughlin T, Veerapen K, Heeneman S, Freeman A, van der Vleuten C. Ottawa 2020 consensus statements for programmatic assessment - 2. Implementation and practice. MEDICAL TEACHER 2021; 43:1149-1160. [PMID: 34330202 DOI: 10.1080/0142159x.2021.1956681] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
INTRODUCTION Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts. METHODS After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis. RESULTS AND DISCUSSION A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation.
Collapse
Affiliation(s)
- Dario Torre
- Department of Medicine, Uniformed Services University of Health Sciences, Bethesda, MD, USA
| | - Neil E Rice
- College of Medicine and Health, University of Exeter Medical School, Exeter, UK
| | - Anna Ryan
- Department of Medical Education, Melbourne Medical School, University of Melbourne, Melbourne, Australia
| | - Harold Bok
- Department of Population Health Sciences, Faculty of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands
| | - Luke J Dawson
- School of Dentistry, University of Liverpool, Liverpool, UK
| | - Beth Bierer
- Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, OH, USA
| | - Tim J Wilkinson
- Education unit, University of Otago, Christchurch, New Zealand
| | - Glendon R Tait
- MD Program, Dept. of Psychiatry, and The Wilson Centre, University of Toronto, Toronto, Canada
| | - Tom Laughlin
- Department of Family Medicine, Dalhousie University, Halifax, Canada
| | - Kiran Veerapen
- Faculty of Medicine, University of British Columbia, Vancouver, Canada
| | - Sylvia Heeneman
- Department of Educational Development and Research, School of Health Profession Education, Maastricht University, Maastricht, The Netherlands
| | - Adrian Freeman
- College of Medicine and Health, University of Exeter Medical School, Exeter, UK
| | - Cees van der Vleuten
- Department of Educational Development and Research, School of Health Profession Education, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
19
|
Batt AM, Williams B, Brydges M, Leyenaar M, Tavares W. New ways of seeing: supplementing existing competency framework development guidelines with systems thinking. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:1355-1371. [PMID: 34003391 DOI: 10.1007/s10459-021-10054-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/17/2020] [Accepted: 05/05/2021] [Indexed: 06/12/2023]
Abstract
Competency frameworks provide a link between professional practice, education, training, and assessment. They support and inform downstream processes such as curriculum design, assessment, accreditation and professional accountability. However, existing guidelines are limited in accounting for the complexities of professional practice potentially undermining utility of such guidelines and validity of outcomes. This necessitates additional ways of "seeing" situated and context-specific practice. We highlight what a conceptual framework informed by systems thinking can offer when developing competency frameworks. Mirroring shifts towards systems thinking in program evaluation and quality improvement, we suggest that similar approaches that identify and make use of the role and influence of system features and contexts can provide ways of augmenting existing guidelines when developing competency frameworks. We framed a systems thinking approach in two ways. First using an adaptation of Ecological Systems Theory which offers a realist perspective of the person and environment, and the evolving interaction between the two. Second, by employing complexity thinking, which obligates attention to the relationships and influences of features within the system, we can explore the multiple complex, unique, and context-embedded problems that exist within and have stake in real-world practice settings. The ability to represent clinical practice when developing competency frameworks can be improved when features that may be relevant, including their potential interactions, are identified and understood. A conceptual framework informed by systems thinking makes visible features of a practice in context that may otherwise be overlooked when developing competency frameworks using existing guidelines.
Collapse
Affiliation(s)
- Alan M Batt
- Department of Paramedicine, Monash University, Frankston, VIC, Australia.
- McNally Project for Paramedicine Research, Toronto, ON, Canada.
- Fanshawe College, London, ON, Canada.
| | - Brett Williams
- Department of Paramedicine, Monash University, Frankston, VIC, Australia
| | - Madison Brydges
- McNally Project for Paramedicine Research, Toronto, ON, Canada
- Department of Health, Ageing and Society, McMaster University, Hamilton, ON, Canada
| | - Matthew Leyenaar
- McNally Project for Paramedicine Research, Toronto, ON, Canada
- Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON, Canada
| | - Walter Tavares
- McNally Project for Paramedicine Research, Toronto, ON, Canada
- The Wilson Centre, Department of Medicine, University of Toronto/University Health Network, Toronto, ON, Canada
- Post‑MD Education (Post‑Graduate Medical Education/Continued Professional Development), University of Toronto, Toronto, ON, Canada
| |
Collapse
|
20
|
Robinson TJG, Wagner N, Szulewski A, Dudek N, Cheung WJ, Hall AK. Exploring the use of rating scales with entrustment anchors in workplace-based assessment. MEDICAL EDUCATION 2021; 55:1047-1055. [PMID: 34060651 DOI: 10.1111/medu.14573] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/01/2020] [Revised: 04/07/2021] [Accepted: 05/26/2021] [Indexed: 06/12/2023]
Abstract
PURPOSE Competency-based medical education (CBME) has prompted widespread implementation of workplace-based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale. METHOD A convenience sample of 50 semi-structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line-by-line coding to identify themes. RESULTS Interviews captured interactions between 70% (26/37) of full-time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were 'in the room' or not. CONCLUSIONS By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of 'in-the-moment' assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.
Collapse
Affiliation(s)
| | - Natalie Wagner
- Department of Biomedical & Molecular Sciences, Queen's University, Kingston, ON, Canada
- Office of Professional Development & Educational Scholarship, Queen's University, Kingston, ON, Canada
| | - Adam Szulewski
- Department of Emergency Medicine, Queen's University, Kingston, ON, Canada
- Department of Psychology, Queen's University, Kingston, ON, Canada
| | - Nancy Dudek
- Department of Medicine and The Ottawa Hospital, University of Ottawa, Ottawa, ON, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| | - Warren J Cheung
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Andrew K Hall
- Department of Emergency Medicine, Queen's University, Kingston, ON, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| |
Collapse
|
21
|
Yilmaz Y, Carey R, Chan TM, Bandi V, Wang S, Woods RA, Mondal D, Thoma B. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. CANADIAN MEDICAL EDUCATION JOURNAL 2021; 12:48-64. [PMID: 34567305 PMCID: PMC8463237 DOI: 10.36834/cmej.72067] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
BACKGROUND Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires frequent assessments of entrustable professional activities (EPAs). Faculty struggle to provide helpful feedback and assign appropriate entrustment scores. CBME faculty development initiatives rarely incorporate teaching metrics. Dashboards could be used to visualize faculty assessment data to support faculty development. METHODS Using a design-based research process, we identified faculty development needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. Data was collected within the emergency medicine residency program at the University of Saskatchewan through interviews with program leaders, faculty development experts, and faculty participating in development sessions. Two investigators thematically analyzed interview transcripts to identify faculty needs that were audited by a third investigator. The needs were described using representative quotes and the dashboard elements designed to address them. RESULTS Between July 1, 2019 and December 11, 2020 we conducted 15 interviews with nine participants (two program leaders, three faculty development experts, and four faculty members). Three needs emerged as themes from the analysis: analysis of assessments, contextualization of assessments, and accessible reporting. We addressed these needs by designing an accessible dashboard to present contextualized quantitative and narrative assessment data for each faculty member. CONCLUSIONS We identified faculty development needs related to EPA assessments and designed dashboard elements to meet them. The resulting dashboard was used for faculty development sessions. This work will inform the development of CBME assessment dashboards for faculty.
Collapse
Affiliation(s)
- Yusuf Yilmaz
- Continuing Professional Development Office and McMaster Education Research, Innovation, and Theory (MERIT) Program, McMaster University, Ontario, Canada
- Department of Medical Education, Ege University, Izmir, Turkey
| | - Robert Carey
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Teresa M Chan
- Continuing Professional Development Office and McMaster Education Research, Innovation, and Theory (MERIT) Program, McMaster University, Ontario, Canada
- Emergency Medicine, Department of Medicine, McMaster University, Ontario, Canada
| | - Venkat Bandi
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Shisong Wang
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Robert A Woods
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Debajyoti Mondal
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Brent Thoma
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| |
Collapse
|
22
|
Abstract
Even before starting your evening shift you know it's going to be busy. Ambulances are lined up in front of the hospital, and the charge nurse already seems stressed out. The senior Emergency Medicine (EM) resident is standing in the physician office, ready to start her shift as well. You have worked with her a few times during this rotation. She is competent, you trust in her management plans for all her individual patients. Together you both review the patient tracker: a variety of patient presentations ready to be seen, plus an additional 20 patients in the waiting room. Negotiating the learning objective for the shift, the resident indicates that she would like to work on more efficiently managing patient flow and the administration of the emergency department (ED). But…isn't that a skill you just learn from experience? You wonder what evidence-informed strategies might exist for training her for this next step.
Collapse
|
23
|
Sample S, Al Rimawi H, Bérczi B, Chorley A, Pardhan A, Chan TM. Seeing potential opportunities for teaching (SPOT): Evaluating a bundle of interventions to augment entrustable professional activity acquisition. AEM EDUCATION AND TRAINING 2021; 5:e10631. [PMID: 34471797 PMCID: PMC8381386 DOI: 10.1002/aet2.10631] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/14/2021] [Revised: 05/10/2021] [Accepted: 06/02/2021] [Indexed: 06/13/2023]
Abstract
INTRODUCTION Within the Canadian competency-based medical education system, entrustable professional activities (EPAs) are used to assess residents on performed clinical duties. This study aimed to determine whether implementing a bundle of two interventions (a case-based discussion intervention and a rotation-based nudging system) could increase the number of EPA assessments that could occur for our trainees. METHODS The authors designed an intervention bundle with two components: 1) a case-based workshop where trainees discussed which EPAs could be assessed with multiple cases and 2) a nudging system wherein each trainee was reminded of EPAs that would be useful to them on each rotation in their first year. We conducted a retrospective program evaluation to compare the intervention cohort (2019) to two historical cohorts using similar EPAs (2017, 2018). RESULTS Data from 22 trainees (seven in 2017, eight in 2018, and seven in 2019) were analyzed. There was a marked increase in the total number of EPA assessments acquired in the 2019 cohort (average per resident = 285.7, 95% confidence interval [CI] = 256.1 to 312.3, range = 195-350) compared to the two other years (2018 [average = 132.4, 95% CI = 107.5 to 157.02, range = 107-167] and 2017 [70.1, 95% CI 45.3 to 91.0, range = 49-95]), yielding an effect size of Cohen's d = 4.02 for our intervention bundle. CONCLUSIONS Within the limitations of a small sample size, there was a strong effect of introducing two interventions (a case-based orientation and a nudging system) upon EPA assessments with PGY-1 residents. These strategies may be useful to others seeking to improve EPA assessment numbers in other specialties and clinical environments.
Collapse
Affiliation(s)
- Spencer Sample
- Emergency Medicine Postgraduate Training ProgramMcMaster Royal College of Physicians and Surgeons of CanadaHamiltonOntarioCanada
| | - Hussein Al Rimawi
- Emergency Medicine Postgraduate Training ProgramMcMaster Royal College of Physicians and Surgeons of CanadaHamiltonOntarioCanada
| | - Beatrix Bérczi
- Emergency Medicine Postgraduate Training ProgramMcMaster Royal College of Physicians and Surgeons of CanadaHamiltonOntarioCanada
| | - Alexander Chorley
- Emergency Medicine Postgraduate Training ProgramMcMaster Royal College of Physicians and Surgeons of CanadaHamiltonOntarioCanada
- Division of Emergency Medicine, Department of Medicine, Faculty of Health SciencesMcMaster UniversityHamiltonOntarioCanada
- McMaster Education Research, Innovation, and Theory (MERIT)HamiltonOntarioCanada
| | - Alim Pardhan
- Emergency Medicine Postgraduate Training ProgramMcMaster Royal College of Physicians and Surgeons of CanadaHamiltonOntarioCanada
- Division of Emergency Medicine, Department of Medicine, Faculty of Health SciencesMcMaster UniversityHamiltonOntarioCanada
| | - Teresa M. Chan
- Emergency Medicine Postgraduate Training ProgramMcMaster Royal College of Physicians and Surgeons of CanadaHamiltonOntarioCanada
- Division of Emergency Medicine, Department of Medicine, Faculty of Health SciencesMcMaster UniversityHamiltonOntarioCanada
- McMaster Education Research, Innovation, and Theory (MERIT)HamiltonOntarioCanada
- Division of Education and Innovation, Department of Medicine, Faculty of Health SciencesMcMaster UniversityHamiltonOntarioCanada
| |
Collapse
|
24
|
Hale SJ, Wakeling S, Bhalerao A, Balakumaran J, Huang S, Mondoux S, Blain JB, Chan TM. Feeling the flow with a serious game workshop: GridlockED as Medical Education 2 study (GAME2 study). AEM EDUCATION AND TRAINING 2021; 5:e10576. [PMID: 34124522 PMCID: PMC8171742 DOI: 10.1002/aet2.10576] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/26/2020] [Revised: 01/02/2021] [Accepted: 01/11/2021] [Indexed: 06/12/2023]
Abstract
OBJECTIVE GridlockED gameplay workshops were delivered in Canada. This project investigated workshop attendees' experiences, seeking to identify learning points to inform improvement of the workshop. METHODS GridlockED sessions were held through 2018 and 2019. Workshops targeted medical trainees. After a standardized video, learners played for approximately 90 minutes. Learners completed a postgameplay survey with 7-point Likert scale questions about their experience. RESULTS Seventy-two participants responded to our survey (41 medical students, 13 physician assistant students, 12 emergency medicine residents, and six faculty members). Trainees rated GridlockED as both enjoyable and a meaningful educational experience, with a mean (±SD) rating of 6.53 (±0.96) of 7 for enjoyment and 6.17 (±1.13) for education. Attendees identified teamwork and communication (49%) as the most helpful learning domain, with patient flow (43%) being second and basics of how the ED worked (31%) being third. The respondents self-identified top areas of learning as resource management (38.9%), improved understanding of various provider roles in the ED (33%), and improved communication skills (33%). CONCLUSION Medical learners identified GridlockED to be an educational and enjoyable learning experience. Attendees reported that playing this serious game assisted with learning about health systems and communication.
Collapse
Affiliation(s)
- Stephen J. Hale
- Royal College of Physicians and Surgeons of Canada Emergency Medicine Residency Training ProgramUniversity of ManitobaWinnipegManitobaCanada
| | - Sonja Wakeling
- Royal College of Physicians and Surgeons of Canada Emergency Medicine Residency Training ProgramMcMaster UniversityHamiltonOntarioCanada
| | - Anuja Bhalerao
- Canadian College of Family Physicians Residency Training ProgramQueen's UniversityKingstonOntarioCanada
| | - Janatani Balakumaran
- Royal College of Physicians and Surgeons of Canada Emergency Medicine Residency Training ProgramMcMaster UniversityHamiltonOntarioCanada
| | - Simon Huang
- Royal College of Physicians and Surgeons of Canada Emergency Medicine Residency Training ProgramDalhousie UniversityHalifaxNova ScotiaCanada
| | - Shawn Mondoux
- Division of Emergency MedicineDepartment of MedicineMcMaster UniversityHamiltonOntarioCanada
- McMaster Program for Education Research, Innovation, and Theory (MERITHamiltonOntarioCanada
| | - J. Bruce Blain
- School of ManagementGeorge Brown CollegeTorontoOntarioCanada
- Ontario Institute for Studies in Education/University of TorontoTorontoOntarioCanada
| | - Teresa M. Chan
- Program for Faculty DevelopmentFaculty of Health SciencesMcMaster UniversityHamiltonOntarioCanada
- Department of MedicineDivision of Emergency Medicine and Division of Education & InnovationMcMaster UniversityHamiltonOntarioCanada
- McMaster Program for Education Research, Innovation, and Theory (MERITHamiltonOntarioCanada
| |
Collapse
|
25
|
Caretta-Weyer HA, Chan T, Bigham BL, Kinnear B, Huwendiek S, Schumacher DJ. If we could turn back time: Imagining time-variable, competency-based medical education in the context of COVID-19. MEDICAL TEACHER 2021; 43:774-779. [PMID: 34027813 DOI: 10.1080/0142159x.2021.1925641] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
The COVID-19 pandemic has exposed a paradox in historical models of medical education: organizations responsible for applying consistent standards for progression have needed to adapt to training environments marked by inconsistency and change. Although some institutions have maintained their traditional requirements, others have accelerated their programs to rush nearly graduated trainees to the front lines. One interpretation of the unplanned shortening of the duration of training programs during a crisis is that standards have been lowered. But it is also possible that these trainees were examined according to the same standards as usual and were judged to have already met them. This paper discusses the impacts of the COVID-19 pandemic on the current workforce, provides an analysis of how competency-based medical education (CBME) in the context of the pandemic might have mitigated wide-scale disruption, and identifies structural barriers to achieving an ideal state. The paper further calls upon universities, health centres, governments, certifying bodies, regulatory authorities, and health care professionals to work collectively on a truly time-variable model of CBME. The pandemic has made clear that time variability in medical education already exists and should be adopted widely and formally. If our systems today had used a framework of outcome competencies, sequenced progression, tailored learning, focused instruction, and programmatic assessment, we may have been even more nimble in changing our systems to care for our patients with COVID-19.
Collapse
Affiliation(s)
| | - Teresa Chan
- Department of Medicine, McMaster University, Hamilton, Canada
- McMaster Program for Education Research, Innovation, and Theory (MERIT), Hamilton, Canada
| | - Blair L Bigham
- Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, Canada
| | - Benjamin Kinnear
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Sören Huwendiek
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Daniel J Schumacher
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH, USA
- Division of Emergency Medicine, Cincinnati Children's Hospital, Cincinnati, OH, USA
| |
Collapse
|
26
|
Magee K, Sherbino J. Letter to the Editor. CAN J EMERG MED 2021; 23:565-566. [PMID: 33704693 DOI: 10.1007/s43678-021-00101-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Accepted: 02/09/2021] [Indexed: 12/01/2022]
Affiliation(s)
- Kirk Magee
- Emergency Medicine Specialty Committee, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada.
- Emergency Department, QEII Health Science Centre, 1796 Summer Street, Suite 355 Halifax Infirmary, Halifax, NS, B3H 3A7, Canada.
| | - Jonathan Sherbino
- Emergency Medicine Specialty Committee, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| |
Collapse
|
27
|
Bell C, Hall AK, Wagner N, Rang L, Newbigging J, McKaigney C. The Ultrasound Competency Assessment Tool (UCAT): Development and Evaluation of a Novel Competency-based Assessment Tool for Point-of-care Ultrasound. AEM EDUCATION AND TRAINING 2021; 5:e10520. [PMID: 34041429 PMCID: PMC8138101 DOI: 10.1002/aet2.10520] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Revised: 07/15/2020] [Accepted: 08/09/2020] [Indexed: 05/09/2023]
Abstract
OBJECTIVES Point-of-care ultrasound (POCUS) has become an integral diagnostic and interventional tool. Barriers to POCUS training persist, and it continues to remain heterogeneous across training programs. Structured POCUS assessment tools exist, but remain limited in their feasibility, acceptability, reliability, and validity; none of these tools are entrustment-based. The objective of this study was to derive a simple, entrustment-based POCUS competency assessment tool and pilot it in an assessment setting. METHODS This study was composed of two phases. First, a three-step modified Delphi design surveyed 60 members of the Canadian Association of Emergency Physicians Emergency Ultrasound Committee (EUC) to derive the anchors for the tool. Subsequently, the derived ultrasound competency assessment tool (UCAT) was used to assess trainee (N = 37) performance on a simulated FAST examination. The intraclass correlation (ICC) for inter-rater reliability and Cronbach's alpha for internal consistency were calculated. A statistical analysis was performed to compare the UCAT to other competency surrogates. RESULTS The three-round Delphi had 22, 26, and 26 responses from the EUC members. Consensus was reached, and anchors for the domains of preparation, image acquisition, image optimization, and clinical integration achieved approval rates between 92 and 96%. The UCAT pilot revealed excellent inter-rater reliability (with ICC values of 0.69-0.89; p < 0.01) and high internal consistency (α = 0.91). While UCAT scores were not impacted by level of training, they were significantly impacted by the number of previous POCUS studies completed. CONCLUSIONS We developed and successfully piloted the UCAT, an entrustment-based bedside POCUS competency assessment tool suitable for rapid deployment. The findings from this study indicate early validity evidence for the use of the UCAT as an assessment of trainee POCUS competence on FAST. The UCAT should be trialed in different populations performing several POCUS study types.
Collapse
Affiliation(s)
- Colin Bell
- Department of Emergency MedicineQueen's UniversityKingstonOntarioCanada
| | - Andrew K. Hall
- Department of Emergency MedicineQueen's UniversityKingstonOntarioCanada
| | - Natalie Wagner
- Department of Biomedical & Molecular SciencesQueen's UniversityKingstonOntarioCanada
- Office of Professional Development & Educational ScholarshipQueen's UniversityKingstonOntarioCanada
| | - Louise Rang
- Department of Emergency MedicineQueen's UniversityKingstonOntarioCanada
| | - Joseph Newbigging
- Department of Emergency MedicineQueen's UniversityKingstonOntarioCanada
| | - Conor McKaigney
- Department of Emergency MedicineUniversity of CalgaryCalgaryAlbertaCanada
| |
Collapse
|
28
|
Chan TM, Sebok‐Syer SS, Cheung WJ, Pusic M, Stehman C, Gottlieb M. Workplace-based Assessment Data in Emergency Medicine: A Scoping Review of the Literature. AEM EDUCATION AND TRAINING 2021; 5:e10544. [PMID: 34099992 PMCID: PMC8166307 DOI: 10.1002/aet2.10544] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/04/2020] [Revised: 10/02/2020] [Accepted: 10/05/2020] [Indexed: 06/01/2023]
Abstract
OBJECTIVE In the era of competency-based medical education (CBME), the collection of more and more trainee data is being mandated by accrediting bodies such as the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada. However, few efforts have been made to synthesize the literature around the current issues surrounding workplace-based assessment (WBA) data. This scoping review seeks to synthesize the landscape of literature on the topic of data collection and utilization for trainees' WBAs in emergency medicine (EM). METHODS The authors conducted a scoping review in the style of Arksey and O'Malley, seeking to synthesize and map literature on collecting, aggregating, and reporting WBA data. The authors extracted, mapped, and synthesized literature that describes, supports, and substantiates effective data collection and utilization in the context of the CBME movement within EM. RESULTS Our literature search retrieved 189 potentially relevant references (after removing duplicates) that were screened to 29 abstracts and papers relevant to collecting, aggregating, and reporting WBAs. Our analysis shows that there is an increasing temporal trend toward contributions in these topics, with the majority of the papers (16/29) being published in the past 3 years alone. CONCLUSION There is increasing interest in the areas around data collection and utilization in the age of CBME. The field, however, is only beginning to emerge, leaving more work that can and should be done in this area.
Collapse
Affiliation(s)
- Teresa M. Chan
- From theDepartment of MedicineDivision of Emergency Medicine and the Division of Education & InnovationMcMaster UniversityHamiltonOntarioCanada
- theProgram for Faculty DevelopmentFaculty of Health SciencesMcMaster UniversityHamiltonOntarioCanada
- and the McMaster Program for Education Research, Innovation, and TheoryMcMaster UniversityHamiltonOntarioCanada
| | | | - Warren J. Cheung
- theDepartment of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| | - Martin Pusic
- theDepartment of PediatricsHarvard Medical SchoolBostonMAUSA
| | | | - Michael Gottlieb
- and theDepartment of Emergency MedicineRush University Medical CenterChicagoILUSA
| |
Collapse
|
29
|
Crickmer M, Lam T, Tavares W, Meshkat N. Do PGY-1 residents in Emergency Medicine have enough experiences in resuscitations and other clinical procedures to meet the requirements of a Competence by Design curriculum? CANADIAN MEDICAL EDUCATION JOURNAL 2021; 12:100-104. [PMID: 34249195 PMCID: PMC8263047 DOI: 10.36834/cmej.70921] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
BACKGROUND With the transition to a Competence by Design (CBD) curriculum, Fellow of the Royal College of Physicians in Emergency Medicine (FRCP-EM) training has created guidelines on experiences residents should have before progressing. We sought to quantify adult medical resuscitations and clinical procedures completed by PGY1 FRCP-EM residents to compare them to CBD requirements with the aim to identify areas of limited exposure requiring curriculum revisions prior to nation-wide CBD implementation. METHODS Twenty-two PGY1 residents from four FRCP-EM programs recorded their activities from July 2017 to June 2018 in an online log that tracked resuscitations and procedures along with role assumed, supervision, and level of comfort. RESULTS In total 515 resuscitations were logged with the median number per resident 15 (range 0 to 98). The most frequent resuscitation was altered mental status and the least was unstable dysrhythmia. 557 total procedures were logged with the median number 75 (range 8 to 273). The most frequent procedure done was simple laceration repair and the least frequent was intraosseous access. CONCLUSIONS Unstable dysrhythmias and cardiorespiratory arrest along with intraosseous access and arthrocentesis are low event clinical exposures. In the era of CBD, the misalignment of entrustrable professional activity (EPA) targets and curriculum delivery should be monitored/reviewed to ensure expectations are realistic and that sufficient exposures are available.
Collapse
Affiliation(s)
- Michael Crickmer
- Department of Emergency Medicine, University of Toronto, Ontario, Canada
| | - Tobi Lam
- Wilson Center, University of Toronto, Ontario, Canada
| | | | - Nazanin Meshkat
- Department of Emergency Medicine, University of Toronto, Ontario, Canada
| |
Collapse
|
30
|
Bandiera G, Hall AK. Capturing the forest and the trees: workplace-based assessment tools in emergency medicine. CAN J EMERG MED 2021; 23:265-266. [PMID: 33959929 DOI: 10.1007/s43678-021-00125-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2021] [Accepted: 03/23/2021] [Indexed: 10/21/2022]
Affiliation(s)
- Glen Bandiera
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.,Unity Health Toronto (St. Michael's), Toronto, ON, Canada
| | - Andrew K Hall
- Department of Emergency Medicine, Queen's University, Kingston, ON, Canada. .,Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada.
| |
Collapse
|
31
|
Landreville JM, Frank JR, Cheung WJ. Does direct observation happen early in a new competency-based residency program? AEM EDUCATION AND TRAINING 2021; 5:e10591. [PMID: 33842816 PMCID: PMC8019151 DOI: 10.1002/aet2.10591] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/11/2020] [Revised: 02/16/2021] [Accepted: 02/23/2021] [Indexed: 06/01/2023]
Abstract
BACKGROUND A key component of competency-based medical education is workplace-based assessment, which includes observation (direct or indirect) of residents. Direct observation has been emphasized as an ideal form of assessment yet challenges have been identified that may limit its adoption. At present, it remains unclear how often direct and indirect observation are being used within the clinical setting. The objective of this study was to describe patterns of observation in an emergency medicine competency-based program 2 years postimplementation. METHODS Emergency medicine residents (n = 19) recorded the type of observation they received (direct or indirect) following workplace-based entrustable professional activity (EPA) assessments from December 15, 2019, to April 30, 2020. Assessment forms were reviewed and analyzed to describe patters of observation. RESULTS Assessments were collected on all 19 eligible residents (100% participation). A total of 1,070 EPA assessments were completed during the study period, of which 798 (74.6%) had the type of observation recorded. Of these recorded observations, 546 (68.4%) were directly observed and 252 (31.6%) were indirectly observed. The length of written comments contained within assessments following direct and indirect observation did not differ significantly. There was no significant association between resident gender and observation type or resident stage of training and observation type. Certain EPA assessments showed a clear preference toward either direct or indirect observation. CONCLUSIONS To the best of our knowledge, this study is the first to report patterns of observation in a competency-based residency program. The results suggest that direct observation can be quickly adopted as the primary means of workplace-based assessment. Indirect observation comprised a sizeable minority of observations and may be an underrecognized contributor to workplace-based assessment. The preference toward either direct or indirect observation for certain EPA assessments suggests that the entrustable professional activity itself may influence the type of observation.
Collapse
Affiliation(s)
| | - Jason R. Frank
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| | - Warren J. Cheung
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
| |
Collapse
|
32
|
Dowhos K, Sherbino J, Chan TM, Nagji A. Infographics, podcasts, and blogs: a multi-channel, asynchronous, digital faculty experience to improve clinical teaching (MAX FacDev). CAN J EMERG MED 2021; 23:390-393. [PMID: 33788176 PMCID: PMC8011059 DOI: 10.1007/s43678-020-00069-5] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2020] [Accepted: 12/15/2020] [Indexed: 11/29/2022]
Abstract
Attendance at faculty development events are often limited to those with time and interest, but all clinical teachers should have access to continuing faculty development. A multi-channel, asynchronous, digital faculty experience strategy (MAX FacDev) was used to engage busy emergency medicine (EM) teachers associated with a distributed medical education network involving ten geographically distinct teaching sites. An evidence-informed education bundle on key principles for clinical teaching was developed. The education bundle included five topics, serialized via: an infographic series posted in distributed medical education EM departments, a podcast series and a blog. The target audience included 102 faculty members and 46 residents. Within 8 months of launching MAX FacDev, there were 1508 podcast listens and 7686 pageviews. An education bundle can efficiently deliver on-demand faculty development. Amplifying key messages via multiple channels increases the reach of faculty development and reinforces the messages.
Collapse
Affiliation(s)
- Krista Dowhos
- Division of Emergency Medicine, Department of Family Medicine, McMaster University, Hamilton, ON, Canada
| | - Jonathan Sherbino
- Division of Emergency Medicine, Department of Medicine, McMaster Education Research Innovation and Theory (MERIT) Program, McMaster University, Hamilton, ON, Canada
| | - Teresa M Chan
- Division of Emergency Medicine, Department of Medicine, McMaster Education Research Innovation and Theory (MERIT) Program, McMaster University, Hamilton, ON, Canada
| | - Alim Nagji
- Division of Emergency Medicine, Department of Family Medicine, McMaster University, Hamilton, ON, Canada.
| |
Collapse
|
33
|
Comparing the Ottawa Emergency Department Shift Observation Tool (O-EDShOT) to the traditional daily encounter card: measuring the quality of documented assessments. CAN J EMERG MED 2021; 23:383-389. [PMID: 33512695 DOI: 10.1007/s43678-020-00070-y] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2020] [Accepted: 12/14/2020] [Indexed: 10/22/2022]
Abstract
OBJECTIVES The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a workplace-based assessment designed to assess a trainee's performance across an entire shift. It was developed in response to validity concerns with traditional end-of-shift workplace-based assessments, such as the daily encounter card. The O-EDShOT previously demonstrated strong psychometric characteristics; however, it remains unknown whether the O-EDShOT facilitates measurable improvements in the quality of documented assessments compared to daily encounter cards. METHODS Three randomly selected daily encounter cards and three O-EDShOTs completed by 24 faculty were scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR), a previously published 9-item quantitative measure of the quality of a completed workplace-based assessment. Automated-CCERR (A-CCERR) scores, which do not require raters, were also calculated. Paired sample t tests were conducted to compare the quality of assessments between O-EDShOTs and DECs as measured by the CCERR and A-CCERR. RESULTS CCERR scores were significantly higher for O-EDShOTs (mean(SD) = 25.6(2.6)) compared to daily encounter cards (21.5(3.9); t(23) = 5.2, p < 0.001, d = 1.1). A-CCERR scores were also significantly higher for O-EDShOTs (mean(SD) = 18.5(1.6)) than for daily encounter cards (15.5(1.2); t(24) = 8.4, p < 0.001). CCERR items 1, 4 and 9 were rated significantly higher for O-EDShOTs compared to daily encounter cards. CONCLUSIONS The O-EDShOT yields higher quality documented assessments when compared to the traditional end-of-shift daily encounter card. Our results provide additional validity evidence for the O-EDShOT as an assessment tool for capturing trainee on-shift performance that can be used as a stimulus for actionable feedback and as a source for high-quality workplace-based assessment data to inform decisions about emergency medicine trainee progress and promotion.
Collapse
|
34
|
Carey R, Wilson G, Bandi V, Mondal D, Martin LJ, Woods R, Chan T, Thoma B. Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project. CANADIAN MEDICAL EDUCATION JOURNAL 2020; 11:e31-e45. [PMID: 33349752 PMCID: PMC7749685 DOI: 10.36834/cmej.69682] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
BACKGROUND Canadian specialty programs are implementing Competence By Design, a competency-based medical education (CBME) program which requires frequent assessments of entrustable professional activities. To be used for learning, the large amount of assessment data needs to be interpreted by residents, but little work has been done to determine how visualizing and interacting with this data can be supported. Within the University of Saskatchewan emergency medicine residency program, we sought to determine how our residents' CBME assessment data should be presented to support their learning and to develop a dashboard that meets our residents' needs. METHODS We utilized a design-based research process to identify and address resident needs surrounding the presentation of their assessment data. Data was collected within the emergency medicine residency program at the University of Saskatchewan via four resident focus groups held over 10 months. Focus group discussions were analyzed using a grounded theory approach to identify resident needs. This guided the development of a dashboard which contained elements (data, analytics, and visualizations) that support their interpretation of the data. The identified needs are described using quotes from the focus groups as well as visualizations of the dashboard elements. RESULTS Resident needs were classified under three themes: (1) Provide guidance through the assessment program, (2) Present workplace-based assessment data, and (3) Present other assessment data. Seventeen dashboard elements were designed to address these needs. CONCLUSIONS Our design-based research process identified resident needs and developed dashboard elements to meet them. This work will inform the creation and evolution of CBME assessment dashboards designed to support resident learning.
Collapse
Affiliation(s)
- Robert Carey
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Grayson Wilson
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Venkat Bandi
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Debajyoti Mondal
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Lynsey J. Martin
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Rob Woods
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Teresa Chan
- Division of Emergency Medicine, Department of Medicine, McMaster University, Ontario, Canada
- McMaster program for Education Research, Innovation, and Theory (MERIT), McMaster University, Ontario, Canada
| | - Brent Thoma
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| |
Collapse
|
35
|
Breaking down the silos in simulation-based education: Exploring, refining, and standardizing. CAN J EMERG MED 2020; 22:733-734. [DOI: 10.1017/cem.2020.471] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
36
|
Abstract
Resident education in emergency medicine (EM) relies upon a variety of teaching platforms and mediums, including real-life clinical scenarios, simulation, academic day (lectures, small group sessions), journal clubs, and teaching learners. However, the coronavirus disease 2019 (COVID-19) pandemic has disrupted teaching and learning, forcing programs to adapt to ensure residents can progress in their training.1 Suddenly, academic days cannot be held in person, emergency department (ED) volumes are dynamically changing, and the role of residents in ED procedures has been questioned. Furthermore, medical student rotations through the ED have been cancelled, decreasing resident exposure to undergraduate teaching. These changes to resident education threaten resident wellness and will have downstream effects on training and personal professional development. In response, programs must develop strategies to ensure that residents continue receiving high-quality training in a safe learning environment. In this review, we will cover recommended strategies put forth by two large EM programs in Ontario (Table 1).
Collapse
|
37
|
Thoma B, Hall AK, Clark K, Meshkat N, Cheung WJ, Desaulniers P, Ffrench C, Meiwald A, Meyers C, Patocka C, Beatty L, Chan TM. Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study. J Grad Med Educ 2020; 12:425-434. [PMID: 32879682 PMCID: PMC7450748 DOI: 10.4300/jgme-d-19-00803.1] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/12/2019] [Revised: 02/11/2020] [Accepted: 05/20/2020] [Indexed: 01/08/2023] Open
Abstract
BACKGROUND In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement. OBJECTIVE We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels. METHODS Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. RESULTS Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested. CONCLUSIONS There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.
Collapse
|
38
|
Concerns with Current Approach to Competency Based Medical Education of EM Specialists. CAN J EMERG MED 2020; 22:E7. [DOI: 10.1017/cem.2020.356] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
39
|
Relationship between ratings of performance in the simulated and workplace environments among emergency medicine residents. CAN J EMERG MED 2020; 22:811-818. [DOI: 10.1017/cem.2020.388] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
ABSTRACTObjectivesThe Emergency Medicine (EM) Specialty Committee of the Royal College of Physicians and Surgeons of Canada (RCPSC) specifies that resuscitation entrustable professional activities (EPAs) can be assessed in the workplace and simulated environments. However, limited validity evidence for these assessments in either setting exists. We sought to determine if EPA ratings improve over time and whether an association exists between ratings in the workplace v. simulation environment.MethodsAll Foundations EPA1 (F1) assessments were collected for first-year residents (n = 9) in our program during the 2018–2019 academic year. This EPA focuses on initiating and assisting in the resuscitation of critically ill patients. EPA ratings obtained in the workplace and simulation environments were compared using Lin's concordance correlation coefficient (CCC). To determine whether ratings in the two environments differed as residents progressed through training, a within-subjects analysis of variance was conducted with training environment and month as independent variables.ResultsWe collected 104 workplace and 36 simulation assessments. No correlation was observed between mean EPA ratings in the two environments (CCC(8) = -0.01; p = 0.93). Ratings in both settings improved significantly over time (F(2,16) = 18.8; p < 0.001; η2= 0.70), from 2.9 ± 1.2 in months 1–4 to 3.5 ± 0.2 in months 9–12. Workplace ratings (3.4 ± 0.1) were consistently higher than simulation ratings (2.9 ± 0.2) (F(2,16) = 7.2; p = 0.028; η2= 0.47).ConclusionsNo correlation was observed between EPA F1 ratings in the workplace v. simulation environments. Further studies are needed to clarify the conflicting results of our study with others and build an evidence base for the validity of EPA assessments in simulated and workplace environments.
Collapse
|
40
|
Thoma B, Bandi V, Carey R, Mondal D, Woods R, Martin L, Chan T. Developing a dashboard to meet Competence Committee needs: a design-based research project. CANADIAN MEDICAL EDUCATION JOURNAL 2020; 11:e16-e34. [PMID: 32215140 PMCID: PMC7082472 DOI: 10.36834/cmej.68903] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/09/2023]
Abstract
BACKGROUND Competency-based programs are being adopted in medical education around the world. Competence Committees must visualize learner assessment data effectively to support their decision-making. Dashboards play an integral role in decision support systems in other fields. Design-based research allows the simultaneous development and study of educational environments. METHODS We utilized a design-based research process within the emergency medicine residency program at the University of Saskatchewan to identify the data, analytics, and visualizations needed by its Competence Committee, and developed a dashboard incorporating these elements. Narrative data were collected from two focus groups, five interviews, and the observation of two Competence Committee meetings. Data were qualitatively analyzed to develop a thematic framework outlining the needs of the Competence Committee and to inform the development of the dashboard. RESULTS The qualitative analysis identified four Competence Committee needs (Explore Workplace-Based Assessment Data, Explore Other Assessment Data, Understand the Data in Context, and Ensure the Security of the Data). These needs were described with narratives and represented through visualizations of the dashboard elements. CONCLUSIONS This work addresses the practical challenges of supporting data-driven decision making by Competence Committees and will inform the development of dashboards for programs, institutions, and learner management systems.
Collapse
Affiliation(s)
- Brent Thoma
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
- Correspondence: Dr. Brent Thoma, Room 2646, Box 16, 103 Hospital Drive, Saskatoon, SK S7N 0W8; ; phone: 1-306-881-0112; Twitter: @Brent_Thoma
| | - Venkat Bandi
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Robert Carey
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Debajyoti Mondal
- Department of Computer Science, University of Saskatchewan, Saskatchewan, Canada
| | - Rob Woods
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Lynsey Martin
- Department of Emergency Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Teresa Chan
- Division of Emergency Medicine, Department of Medicine, McMaster University, Ontario, Canada
- McMaster program for Education Research, Innovation, and Theory (MERIT), Ontario, Canada
| |
Collapse
|