1
|
Scarff CE, Bearman M, Chiavaroli N, Trumble S. Assessor discomfort and failure to fail in clinical performance assessments. BMC MEDICAL EDUCATION 2023; 23:901. [PMID: 38012637 PMCID: PMC10680261 DOI: 10.1186/s12909-023-04688-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Accepted: 09/14/2023] [Indexed: 11/29/2023]
Abstract
BACKGROUND Assessment of trainee performance in the workplace is critical to ensuring high standards of clinical care. However, some supervisors find the task to be challenging, and may feel unable to deliver their true judgement on a trainee's performance. They may 'keep MUM' (that is, keep mum about undesirable messages) and fail to fail an underperforming trainee. In this study, we explore the effect of discomfort on assessors. METHODS Using a survey method, supervisors of trainees in the Australasian College of Dermatologists were asked to self-report experiences of discomfort in various aspects of trainee workplace assessment and for their engagement in MUM behaviours including failure to fail. RESULTS Sixty-one responses were received from 135 eligible assessors. 12.5% of assessors self-reported they had failed to fail a trainee and 18% admitted they had grade inflated a trainee's score on a clinical performance assessment in the previous 12-month period. Assessors who reported higher levels of discomfort in the clinical performance assessment context were significantly more likely to report previously failing to fail a trainee. The study did not reveal significant associations with assessor demographics and self-reports of discomfort or MUM behaviours. CONCLUSIONS This study reveals the impact of assessor discomfort on the accuracy of assessment information and feedback to trainees, including as a contributing factor to the failure to fail phenomenon. Addressing assessor experience of discomfort offers one opportunity to impact on the complex and multifactorial issue that failure to fail represents.
Collapse
Affiliation(s)
- Catherine E Scarff
- Department of Medical Education, Melbourne Medical School, University of Melbourne, Room N722, Level 7 North Medical Building Grattan Street, Melbourne, VIC, Australia.
| | - Margaret Bearman
- Centre for Research in Assessment and Digital Learning (CRADLE), Deakin University, Melbourne, VIC, Australia
| | - Neville Chiavaroli
- Department of Medical Education, Melbourne Medical School, University of Melbourne, Room N722, Level 7 North Medical Building Grattan Street, Melbourne, VIC, Australia
- Australian Council for Educational Research, Camberwell, Australia
| | - Stephen Trumble
- Department of Medical Education, Melbourne Medical School, University of Melbourne, Room N722, Level 7 North Medical Building Grattan Street, Melbourne, VIC, Australia
| |
Collapse
|
2
|
Matava CT, Alam F, Kealey A, Bahrey LA, McCreath GA, Walsh CM. The influence of resident and faculty gender on assessments in anesthesia competency-based medical education. Can J Anaesth 2023; 70:978-987. [PMID: 37165126 PMCID: PMC10171726 DOI: 10.1007/s12630-023-02454-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2022] [Revised: 11/15/2022] [Accepted: 11/17/2022] [Indexed: 05/12/2023] Open
Abstract
PURPOSE Competency-based medical education (CBME) relies on frequent workplace-based assessments of trainees, providing opportunities for conscious and implicit biases to reflect in these assessments. We aimed to examine the influence of resident and faculty gender on performance ratings of residents within a CBME system. METHODS This retrospective cohort study took place from August 2017 to January 2021 using resident assessment data from two workplace-based assessments: the Anesthesia Clinical Encounter Assessment (ACEA) and Entrustable Professional Activities (EPAs). Self-reported gender data were also extracted. The primary outcome-gender-based differences in entrustment ratings of residents on the ACEA and EPAs-was evaluated using mixed-effects logistic regression, with differences reported through odds ratios and confidence intervals (α = 0.01). Gender-based differences in the receipt of free-text comments on the ACEA and EPAs were also explored. RESULTS In total, 14,376 ACEA and 4,467 EPA assessments were analyzed. There were no significant differences in entrustment ratings on either assessment tool between men and women residents. Regardless of whether assessments were completed by men or women faculty, entrustment rates between men and women residents were not significantly different for any postgraduate year level. Additionally, men and women residents received strengths-related and actions-related comments on both assessments at comparable frequencies, irrespective of faculty gender. CONCLUSION We found no gender-based differences in entrustment ratings for both the ACEA and EPAs, which suggests an absence of resident gender bias within this CBME system. Given considerable heterogeneity in rater leniency, future work would be strengthened by using rater leniency-adjusted scores rather than raw scores.
Collapse
Affiliation(s)
- Clyde T Matava
- Department of Anesthesia and Pain Medicine, The Hospital for Sick Children, 555 University Ave., Toronto, ON, M5G 1X8, Canada.
- Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
- SickKids Research Institute, The Hospital for Sick Children, Toronto, ON, Canada.
| | - Fahad Alam
- Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
- Department of Anesthesia, Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Alayne Kealey
- Department of Anesthesia and Pain Medicine, The Hospital for Sick Children, 555 University Ave., Toronto, ON, M5G 1X8, Canada
- Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Lisa A Bahrey
- Department of Anesthesiology and Pain Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
- Department of Anesthesia, University Health Network-Toronto General Hospital, Toronto, ON, Canada
| | - Graham A McCreath
- SickKids Research Institute, The Hospital for Sick Children, Toronto, ON, Canada
| | - Catharine M Walsh
- SickKids Research Institute, The Hospital for Sick Children, Toronto, ON, Canada
- Division of Gastroenterology, Hepatology and Nutrition, The Hospital for Sick Children, Toronto, ON, Canada
- Department of Paediatrics, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
- The Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
3
|
Bentley H, Darras KE, Forster BB, Sedlic A, Hague CJ. Review of Challenges to the Implementation of Competence by Design in Post-Graduate Medical Education: What Can Diagnostic Radiology Learn from the Experience of Other Specialty Disciplines? Acad Radiol 2022; 29:1887-1896. [PMID: 35094947 DOI: 10.1016/j.acra.2021.11.025] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2021] [Revised: 11/27/2021] [Accepted: 11/30/2021] [Indexed: 01/26/2023]
Abstract
Competence by Design (CBD) is a medical education initiative instituted by the Royal College of Physicians and Surgeons of Canada to improve the training of resident physicians in specialty disciplines. CBD integrates Competency Based Medical Education with traditional specialty discipline post-graduate medical education (PGME) training through the application of an organizational framework of competencies. Various specialty disciplines in Canada have transitioned to CBD since 2017 in a staggered approach. Diagnostic radiology PGME programs in Canada are expected to transition to CBD in 2022 for the incoming resident physician cohort. This article reviews potential challenges to the implementation of CBD in diagnostic radiology PGME programs and proposes evidence-informed targeted strategies and solutions to address these challenges. It is important for diagnostic radiology PGME programs to understand the challenges pertaining to the implementation of CBD so that they may be able to successfully implement this or similar medical education initiatives in their programs. Moreover, as radiology subspecialty PGME programs, such as nuclear medicine, interventional radiology, neuroradiology, and pediatric radiology, likewise transition to CBD and diagnostic radiology PGME programs internationally increasingly implement other Competency Based Medical Education models, the implications of the challenges pertaining to the implementation of CBD will further become of increasing importance.
Collapse
Affiliation(s)
- Helena Bentley
- Department of Radiology, Faculty of Medicine, Gordon & Leslie Diamond Health Care Centre, University of British Columbia, 11th Floor, 2775 Laurel Street, Vancouver, BC V5Z 1M9, Canada.
| | - Kathryn E Darras
- Department of Radiology, Faculty of Medicine, Gordon & Leslie Diamond Health Care Centre, University of British Columbia, 11th Floor, 2775 Laurel Street, Vancouver, BC V5Z 1M9, Canada; Department of Radiology, Vancouver General Hospital, Vancouver, BC, Canada
| | - Bruce B Forster
- Department of Radiology, Faculty of Medicine, Gordon & Leslie Diamond Health Care Centre, University of British Columbia, 11th Floor, 2775 Laurel Street, Vancouver, BC V5Z 1M9, Canada; Department of Radiology, Vancouver General Hospital, Vancouver, BC, Canada
| | - Anto Sedlic
- Department of Radiology, Faculty of Medicine, Gordon & Leslie Diamond Health Care Centre, University of British Columbia, 11th Floor, 2775 Laurel Street, Vancouver, BC V5Z 1M9, Canada; Department of Radiology, Vancouver General Hospital, Vancouver, BC, Canada
| | - Cameron J Hague
- Department of Radiology, Faculty of Medicine, Gordon & Leslie Diamond Health Care Centre, University of British Columbia, 11th Floor, 2775 Laurel Street, Vancouver, BC V5Z 1M9, Canada; Department of Radiology, St Paul's Hospital, Vancouver, BC, Canada
| |
Collapse
|
4
|
Kealey A, Naik VN. Competency-Based Medical Training in Anesthesiology: Has It Delivered on the Promise of Better Education? Anesth Analg 2022; 135:223-229. [PMID: 35839492 DOI: 10.1213/ane.0000000000006091] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Affiliation(s)
- Alayne Kealey
- From the Department of Anesthesia, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada.,Department of Anesthesiology and Pain Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Viren N Naik
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, Ontario, Canada
| |
Collapse
|
5
|
Gingerich A, Sebok-Syer SS, Lingard L, Watling CJ. The shift from disbelieving underperformance to recognising failure: A tipping point model. MEDICAL EDUCATION 2022; 56:395-406. [PMID: 34668213 DOI: 10.1111/medu.14681] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Revised: 10/05/2021] [Accepted: 10/15/2021] [Indexed: 06/13/2023]
Abstract
CONTEXT Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed. METHODS Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process. RESULTS The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm. CONCLUSION Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.
Collapse
Affiliation(s)
- Andrea Gingerich
- Division of Medical Sciences, University of Northern British Columbia, Prince George, British Columbia, Canada
| | | | - Lorelei Lingard
- Schulich School of Medicine & Dentistry, Centre for Education Research & Innovation, Western University, London, Ontario, Canada
| | - Christopher J Watling
- Schulich School of Medicine & Dentistry, Centre for Education Research & Innovation, Western University, London, Ontario, Canada
| |
Collapse
|
6
|
Silver EM, Flores LE, Muir R, Padilla KE, Salinas M, Lorello GR. Letter to the Editor on "Investigating faculty assessment of anesthesia trainees and the failing-to-fail phenomenon: a randomized controlled trial". Can J Anaesth 2021; 68:1300-1301. [PMID: 34031805 DOI: 10.1007/s12630-021-02029-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Revised: 04/26/2021] [Accepted: 04/26/2021] [Indexed: 11/24/2022] Open
Affiliation(s)
- Emily M Silver
- Department of Psychology, University of Chicago, Chicago, IL, USA
| | - Laura E Flores
- College of Allied Health Professions, University of Nebraska Medical Center, Omaha, NE, USA
| | - Roshell Muir
- Division of Infectious Diseases and HIV Medicine, Department of Medicine, Drexel University College of Medicine, Philadelphia, PA, USA
| | - Kathleen E Padilla
- School of Criminology and Criminal Justice, Arizona State University, Phoenix, AZ, USA
| | - Manisha Salinas
- Center for Health Equity and Community Engagement Research, Mayo Clinic, Jacksonville, FL, USA
| | - Gianni R Lorello
- Department of Anesthesiology and Pain Medicine, University of Toronto, Toronto, ON, Canada. .,Department of Anesthesia and Pain Management, University Health Network - Toronto Western Hospital, Toronto, ON, Canada. .,The Wilson Centre, University Health Network, Toronto, ON, Canada. .,Women's College Research Institute, Women's College Hospital, Toronto, ON, Canada.
| |
Collapse
|