1
|
Wade DT. Does a service provide safe, effective rehabilitation? An evaluation method for providers and purchasers. Clin Rehabil 2024:2692155241259644. [PMID: 39053145 DOI: 10.1177/02692155241259644] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/27/2024]
Abstract
BACKGROUND Independent organisations monitor the safety and governance of clinical services but do not assess specialist expertise. Peer review can assess service capability but is resource-intense and infeasible. THE PROBLEM How can you ensure a service provides safe, effective rehabilitation? You ask them to provide data as evidence that they can be trusted to do so. This article suggests a structured approach to providing data on entrustability. AN ANALOGY How is the specialist skill of a doctor in training established? They provide evidence about high-level outcomes (capabilities in practice) related to their speciality. An educational supervisor assesses whether they can be trusted to perform safely and effectively without supervision. The capabilities in practice define their expertise. THE SOLUTION A service can use seven high-level rehabilitation service capabilities, based on the clinical capabilities associated with medical training, with observable indicative descriptors, to collect evidence of their rehabilitation approach. A service must also select four to eight high-level competencies indicating they can rehabilitate their patient caseload safely and effectively. These competencies also need indicative descriptors as evidence of their performance in the service; 11 examples are given. CAPABILITIES. The seven rehabilitation capabilities are: using the biopsychosocial model, having a multi-professional team, making a person-centred rehabilitation plan, working collaboratively across all boundaries, tailoring treatments to the patient's needs, ensuring staff have specific competencies required for their caseload, and acknowledging and managing uncertainty and complexity. CONCLUSION. Service providers could use this structured approach to develop and provide users with evidence of their rehabilitation expertise.
Collapse
Affiliation(s)
- Derick T Wade
- Centre for Movement, Occupation and Rehabilitation Sciences (MOReS), Faculty of Health Sciences, Oxford Brookes University, Headington Campus, Oxford OX3 0BP, UK
| |
Collapse
|
2
|
Nair BR, Moonen - van Loon JMW, van Lierop M, Govaerts M. Leveraging Narrative Feedback in Programmatic Assessment: The Potential of Automated Text Analysis to Support Coaching and Decision-Making in Programmatic Assessment. ADVANCES IN MEDICAL EDUCATION AND PRACTICE 2024; 15:671-683. [PMID: 39050116 PMCID: PMC11268569 DOI: 10.2147/amep.s465259] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/21/2024] [Accepted: 06/27/2024] [Indexed: 07/27/2024]
Abstract
Introduction Current assessment approaches increasingly use narratives to support learning, coaching and high-stakes decision-making. Interpretation of narratives, however, can be challenging and time-consuming, potentially resulting in suboptimal or inadequate use of assessment data. Support for learners, coaches as well as decision-makers in the use and interpretation of these narratives therefore seems essential. Methods We explored the utility of automated text analysis techniques to support interpretation of narrative assessment data, collected across 926 clinical assessments of 80 trainees, in an International Medical Graduates' licensing program in Australia. We employed topic modelling and sentiment analysis techniques to automatically identify predominant feedback themes as well as the sentiment polarity of feedback messages. We furthermore sought to examine the associations between feedback polarity, numerical performance scores, and overall judgments about task performance. Results Topic modelling yielded three distinctive feedback themes: Medical Skills, Knowledge, and Communication & Professionalism. The volume of feedback varied across topics and clinical settings, but assessors used more words when providing feedback to trainees who did not meet competence standards. Although sentiment polarity and performance scores did not seem to correlate at the level of single assessments, findings showed a strong positive correlation between the average performance scores and the average algorithmically assigned sentiment polarity. Discussion This study shows that use of automated text analysis techniques can pave the way for a more efficient, structured, and meaningful learning, coaching, and assessment experience for learners, coaches and decision-makers alike. When used appropriately, these techniques may facilitate more meaningful and in-depth conversations about assessment data, by supporting stakeholders in interpretation of large amounts of feedback. Future research is vital to fully unlock the potential of automated text analysis, to support meaningful integration into educational practices.
Collapse
Affiliation(s)
- Balakrishnan R Nair
- University of Newcastle, Centre for Medical Professional Development, Newcastle, Australia
| | - Joyce M W Moonen - van Loon
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Marion van Lierop
- Department of Family Medicine, Faculty of Health Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Marjan Govaerts
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| |
Collapse
|
3
|
Schumacher DJ, Michelson C, Winn AS, Turner DA, Martini A, Kinnear B. A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees. MEDICAL EDUCATION 2024; 58:812-824. [PMID: 38088227 DOI: 10.1111/medu.15296] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 11/03/2023] [Accepted: 11/21/2023] [Indexed: 06/04/2024]
Abstract
INTRODUCTION The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why? METHODS Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs. RESULTS PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes. DISCUSSION PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Catherine Michelson
- Ann and Robert H. Lurie Children's Hospital of Chicago/Northwestern University, Chicago, Illinois, USA
| | - Ariel S Winn
- Boston Children's Hospital/Harvard Medical School, Boston, Massachusetts, USA
| | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | - Abigail Martini
- Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
| | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
4
|
Chiel LE, Marcus CH, Gomez AR, Williams K, Buraks O, Fishman M, Angerman J, Winn AS. Peer evaluations in an anonymous and open system: Intern reactions and evaluation analysis. MEDICAL TEACHER 2024; 46:919-924. [PMID: 38065679 DOI: 10.1080/0142159x.2023.2287985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Accepted: 11/22/2023] [Indexed: 06/27/2024]
Abstract
PURPOSE Feedback and evaluation from peers is fundamental to trainees' professional development but may be uncomfortable to provide non-anonymously. We aimed to understand resident perception of anonymous and open written evaluation systems and to analyze evaluations in each of these systems. MATERIALS AND METHOD We compared two years of intern peer evaluations at a large United States-based pediatric residency program - the 2018-2019 and 2019-2020 years during which intern peer evaluations were anonymous and open, respectively. We electronically surveyed interns about their perceptions of peer evaluations and analyzed four aspects of the evaluations themselves: (1) orientation, (2) caliber, (3) Likert-scale, and (4) word count. RESULTS 40 (78%) and 38 (75%) interns participated in the survey in the anonymous and open years, respectively. Respondents reported being more likely to avoid writing constructive comments in the open year. There were more high caliber comments in the open year. Likert-scale ratings of peers were lower in the open year. Word count was longer in the open year. CONCLUSIONS While interns expressed more discomfort evaluating peers in an open evaluation system, they wrote longer and more high caliber comments in an open system than in an anonymous system. Residency programs should consider professional development in writing peer evaluation.
Collapse
Affiliation(s)
- Laura E Chiel
- Division of Pulmonary Medicine, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Carolyn H Marcus
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Amanda R Gomez
- Division of Gastroenterology, Hepatology, and Nutrition, Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago and Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Kathryn Williams
- Biostatistics and Research Design Center, Institutional Centers for Clinical and Translational Research, Boston Children's Hospital, Boston, Massachusetts, USA
| | | | - Michael Fishman
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Jessica Angerman
- Boston Combined Residency Program at Boston Children's Hospital and Health Informatics at Northwestern University, Evanston, Illinois, USA
| | - Ariel S Winn
- Division of General Pediatrics, Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
5
|
Li M, Kurahashi AM, Kawaguchi S, Siemens I, Sirianni G, Myers J. When words are your scalpel, what and how information is exchanged may be differently salient to assessors. MEDICAL EDUCATION 2024. [PMID: 38850193 DOI: 10.1111/medu.15458] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Revised: 05/12/2024] [Accepted: 05/24/2024] [Indexed: 06/10/2024]
Abstract
PURPOSE Variable assessments of learner performances can occur when different assessors determine different elements to be differently important or salient. How assessors determine the importance of performance elements has historically been thought to occur idiosyncratically and thus be amenable to assessor training interventions. More recently, a main source of variation found among assessors was two underlying factors that were differently emphasised: medical expertise and interpersonal skills. This gave legitimacy to the theory that different interpretations of the same performance may represent multiple truths. A faculty development activity introducing assessors to entrustable professional activities in which they estimated a learner's level of readiness for entrustment provided an opportunity to qualitatively explore assessor variation in the context of an interaction and in a setting in which interpersonal skills are highly valued. METHODS Using a constructivist grounded theory approach, we explored variation in assessment processes among a group of palliative medicine assessors who completed a simulated direct observation and assessment of the same learner interaction. RESULTS Despite identifying similar learner strengths and areas for improvement, the estimated level of readiness for entrustment varied substantially among assessors. Those who estimated the learner as not yet ready for entrustment seemed to prioritise what information was exchanged and viewed missed information as performance gaps. Those who estimated the learner as ready for entrustment seemed to prioritise how information was exchanged and viewed the same missed information as personal style differences or appropriate clinical judgement. When presented with a summary, assessors expressed surprise and concern about the variation. CONCLUSION A main source of variation among our assessors was the differential salience of performance elements that align with medical expertise and interpersonal skills. These data support the theory that when assessing an interaction, differential salience for these two factors may be an important and perhaps inevitable source of assessor variation.
Collapse
Affiliation(s)
- Melissa Li
- Division of Palliative Care, Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | | | - Sarah Kawaguchi
- Division of Palliative Care, Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Isaac Siemens
- Division of Palliative Care, Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Giovanna Sirianni
- Division of Palliative Care, Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Jeff Myers
- Division of Palliative Care, Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| |
Collapse
|
6
|
Hanmore T, Moon CC, Curtis R, Hopman W, Baxter S. Is time really of the essence? Timeliness of narrative feedback in ophthalmology CBME assessments. MEDICAL TEACHER 2024; 46:705-710. [PMID: 37910021 DOI: 10.1080/0142159x.2023.2274286] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/03/2023]
Abstract
PURPOSE Competency-based medical education relies on a strong program of assessment, and quality comments play a vital role in ensuring its success. The goal of this study is to determine the effect of the timeliness of assessment completion on the quality of the feedback. MATERIALS AND METHODS Using the Quality of Assessment for Learning (QuAL) score 2478 assessments were evaluated. The assessments included those completed between July 2017 and December 2020 for 18 ophthalmology residents. Spearman correlation, Mann-Whitney U and Kruskal-Wallis tests were used to assess variations in QuAL scores based on the timeliness of assessment completion. RESULTS The timeliness of assessment completion ranged from 0 to 299 d with the mean time for completion being 3 d. As the delay increased, the QuAL score decreased. Feedback provided 4, 5, and 14 d post-encounter demonstrated statistically significant differences in the QuAL score. Additionally, there was a significant difference in the timeliness of feedback when there is no written comment. CONCLUSIONS This study demonstrates that the timeliness of assessment completion might have an effect on the quality of written feedback. Written feedback should be completed within 14 d of the encounter to optimize quantity and quality.
Collapse
Affiliation(s)
- Tessa Hanmore
- Department of Ophthalmology, Queen's University, Kingston, Canada
- Department of Physical Medicine and Rehabilitation, Queen's University, Kingston, Canada
- Department of Psychiatry, Queen's University, Kingston, Canada
| | | | - Rachel Curtis
- Department of Ophthalmology, Queen's University, Kingston, Canada
| | - Wilma Hopman
- Department of Public Health Sciences, Queen's University, Kingston, Canada
| | - Stephanie Baxter
- Department of Ophthalmology, Queen's University, Kingston, Canada
| |
Collapse
|
7
|
Ginsburg S, Stroud L, Brydges R, Melvin L, Hatala R. Dual purposes by design: exploring alignment between residents' and academic advisors' documents in a longitudinal program. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2024:10.1007/s10459-024-10318-2. [PMID: 38438699 DOI: 10.1007/s10459-024-10318-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Accepted: 02/04/2024] [Indexed: 03/06/2024]
Abstract
Longitudinal academic advising (AA) and coaching programs are increasingly implemented in competency based medical education (CBME) to help residents reflect and act on the voluminous assessment data they receive. Documents created by residents for purposes of reflection are often used for a second, summative purpose-to help competence committees make decisions-which may be problematic. Using inductive, thematic analysis we analyzed written comments generated by 21 resident-AA dyads in one large internal medicine program who met over a 2 year period to determine what residents write when asked to reflect, how this aligns with what the AAs report, and what changes occur over time (total 109 resident self-reflections and 105 AA reports). Residents commented more on their developing autonomy, progress and improvement than AAs, who commented far more on performance measures. Over time, residents' writing shifted away from intrinsic roles, patient care and improvement towards what AAs focused on, including getting EPAs (entrustable professional activities), studying and exams. For EPAs, the emphasis was on getting sufficient numbers rather than reflecting on what residents were learning. Our findings challenge the practice of dual-purposing documents, by questioning the blurring of formative and summative intent, the structure of forms and their multiple conflicting purposes, and assumptions about the advising relationship over time. Our study suggests a need to re-evaluate how reflective documents are used in CBME programs. Further research should explore whether and how documentation can best be used to support resident growth and development.
Collapse
Affiliation(s)
- Shiphra Ginsburg
- Department of Medicine, Mount Sinai Hospital, Toronto, ON, Canada.
- Wilson Centre for Research in Education, University Health Network, Toronto, ON, Canada.
| | - Lynfa Stroud
- Department of Medicine, Sunnybrook Health Sciences Centre, Toronto, ON, Canada
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Ryan Brydges
- Wilson Centre for Research in Education, University Health Network, Toronto, ON, Canada
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
- Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, ON, Canada
| | - Lindsay Melvin
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
- Department of Medicine, University Health Network, Toronto, ON, Canada
| | - Rose Hatala
- Department of Medicine, University of British Columbia, Vancouver, BC, Canada
- Centre for Health Education Scholarship, University of British Columbia, Vancouver, Canada
| |
Collapse
|
8
|
Choo EK, Woods R, Walker ME, O’Brien JM, Chan TM. The Quality of Assessment for Learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study. CANADIAN MEDICAL EDUCATION JOURNAL 2023; 14:78-85. [PMID: 38226296 PMCID: PMC10787859 DOI: 10.36834/cmej.75876] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 01/17/2024]
Abstract
Background Competency based residency programs depend on high quality feedback from the assessment of entrustable professional activities (EPA). The Quality of Assessment for Learning (QuAL) score is a tool developed to rate the quality of narrative comments in workplace-based assessments; it has validity evidence for scoring the quality of narrative feedback provided to emergency medicine residents, but it is unknown whether the QuAL score is reliable in the assessment of narrative feedback in other postgraduate programs. Methods Fifty sets of EPA narratives from a single academic year at our competency based medical education post-graduate anesthesia program were selected by stratified sampling within defined parameters [e.g. resident gender and stage of training, assessor gender, Competency By Design training level, and word count (≥17 or <17 words)]. Two competency committee members and two medical students rated the quality of narrative feedback using a utility score and QuAL score. We used Kendall's tau-b co-efficient to compare the perceived utility of the written feedback to the quality assessed with the QuAL score. The authors used generalizability and decision studies to estimate the reliability and generalizability coefficients. Results Both the faculty's utility scores and QuAL scores (r = 0.646, p < 0.001) and the trainees' utility scores and QuAL scores (r = 0.667, p < 0.001) were moderately correlated. Results from the generalizability studies showed that utility scores were reliable with two raters for both faculty (Epsilon=0.87, Phi=0.86) and trainees (Epsilon=0.88, Phi=0.88). Conclusions The QuAL score is correlated with faculty- and trainee-rated utility of anesthesia EPA feedback. Both faculty and trainees can reliability apply the QuAL score to anesthesia EPA narrative feedback. This tool has the potential to be used for faculty development and program evaluation in Competency Based Medical Education. Other programs could consider replicating our study in their specialty.
Collapse
Affiliation(s)
- Eugene K Choo
- Department of Anesthesiology, College of Medicine, University of Saskatchewan, Saskatchewan, Canada;
| | - Rob Woods
- Department of Emergency Medicine, College of Medicine, University of Saskatchewan, Saskatchewan, Canada
| | - Mary Ellen Walker
- Department of Anesthesiology, College of Medicine, University of Saskatchewan, Saskatchewan, Canada;
| | - Jennifer M O’Brien
- Department of Anesthesiology, College of Medicine, University of Saskatchewan, Saskatchewan, Canada;
| | - Teresa M Chan
- Department of Medicine (Division of Emergency Medicine; Division of Education & Innovation), Michael G. DeGroote School of Medicine, Faculty of Health Sciences, McMaster University and Office of Continuing Professional Development & McMaster Education Research, Innovation, and Theory (MERIT) Program, Faculty of Health Sciences, McMaster University, Ontario, Canada
| |
Collapse
|
9
|
Mineyko A, Morrison M. Documenting Performance and Feedback in Medical Education: An Essential Skill. MEDICAL SCIENCE EDUCATOR 2023; 33:1399-1403. [PMID: 38188380 PMCID: PMC10766914 DOI: 10.1007/s40670-023-01888-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 09/12/2023] [Indexed: 01/09/2024]
Abstract
Documentation of performance provides feedback to medical trainees on their progress and is required by Programs as a record for monitoring whether trainees are achieving expectations against accepted standards. Despite the importance of performance documentation and the focus on improving feedback delivery in medical education, there has been little written in the literature on documentation best practices. Documentation is an essential skill that cannot be learned solely by observation, as most is confidential. However, teaching documentation of learner performance is rarely discussed. This article describes the authors' experience in teaching the skill of effective documentation of trainee performance for the purpose of providing feedback, monitoring progress, and recording evaluations.
Collapse
Affiliation(s)
- Aleksandra Mineyko
- Department of Pediatrics, University of Calgary, Calgary, Canada
- Section of Neurology, Department of Pediatrics, Alberta Children’s Hospital, 28 Oki Dr. NW, Calgary, AB T3B 6A8 Canada
| | | |
Collapse
|
10
|
Carney PA, Sebok-Syer SS, Pusic MV, Gillespie CC, Westervelt M, Goldhamer MEJ. Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education. MEDICAL EDUCATION ONLINE 2023; 28:2178913. [PMID: 36821373 PMCID: PMC9970252 DOI: 10.1080/10872981.2023.2178913] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Revised: 02/02/2023] [Accepted: 02/07/2023] [Indexed: 06/18/2023]
Abstract
Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.
Collapse
Affiliation(s)
- Patricia A. Carney
- Professor of Family Medicine, Oregon Health & Science University, Portland, OR, USA
| | | | | | | | - Marjorie Westervelt
- Director of Assessment, Evaluation and Scholarship, University of California, Davis, CA, USA
| | - Mary Ellen J. Goldhamer
- Medicine, Harvard Medical School, Massachusetts General Hospital, and Mass General Brigham, Boston, MA, USA
| |
Collapse
|
11
|
Marty AP, Linsenmeyer M, George B, Young JQ, Breckwoldt J, Ten Cate O. Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154. MEDICAL TEACHER 2023; 45:1203-1213. [PMID: 36706225 DOI: 10.1080/0142159x.2023.2168527] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.
Collapse
Affiliation(s)
| | - Machelle Linsenmeyer
- West Virginia School of Osteopathic Medicine, Lewisburg, WV, United States of America
| | - Brian George
- Surgery and Learning Health Sciences, University of Michigan, Ann Arbor, Michigan, United States of America
| | - John Q Young
- Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell &, Zucker Hillside Hospital, NY, United States of America
| | - Jan Breckwoldt
- Institute of Anesthesia at the University Hospital Zurich, Switzerland
| | - Olle Ten Cate
- Utrecht Center for Research and Development of Health Professions Education at UMC Utrecht, The Netherlands
| |
Collapse
|
12
|
Szulewski A, Braund H, Dagnone DJ, McEwen L, Dalgarno N, Schultz KW, Hall AK. The Assessment Burden in Competency-Based Medical Education: How Programs Are Adapting. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:1261-1267. [PMID: 37343164 DOI: 10.1097/acm.0000000000005305] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/23/2023]
Abstract
Residents and faculty have described a burden of assessment related to the implementation of competency-based medical education (CBME), which may undermine its benefits. Although this concerning signal has been identified, little has been done to identify adaptations to address this problem. Grounded in an analysis of an early Canadian pan-institutional CBME adopter's experience, this article describes postgraduate programs' adaptations related to the challenges of assessment in CBME. From June 2019-September 2022, 8 residency programs underwent a standardized Rapid Evaluation guided by the Core Components Framework (CCF). Sixty interviews and 18 focus groups were held with invested partners. Transcripts were analyzed abductively using CCF, and ideal implementation was compared with enacted implementation. These findings were then shared back with program leaders, adaptations were subsequently developed, and technical reports were generated for each program. Researchers reviewed the technical reports to identify themes related to the burden of assessment with a subsequent focus on identifying adaptations across programs. Three themes were identified: (1) disparate mental models of assessment processes in CBME, (2) challenges in workplace-based assessment processes, and (3) challenges in performance review and decision making. Theme 1 included entrustment interpretation and lack of shared mindset for performance standards. Adaptations included revising entrustment scales, faculty development, and formalizing resident membership. Theme 2 involved direct observation, timeliness of assessment completion, and feedback quality. Adaptations included alternative assessment strategies beyond entrustable professional activity forms and proactive assessment planning. Theme 3 related to resident data monitoring and competence committee decision making. Adaptations included adding resident representatives to the competence committee and assessment platform enhancements. These adaptations represent responses to the concerning signal of significant burden of assessment within CBME being experienced broadly. The authors hope other programs may learn from their institution's experience and navigate the CBME-related assessment burden their invested partners may be facing.
Collapse
Affiliation(s)
- Adam Szulewski
- A. Szulewski is associate professor, Departments of Emergency Medicine and Psychology, and educational scholarship lead, Postgraduate Medical Education, Queen's University, Kingston, Ontario, Canada; ORCID: https://orcid.org/0000-0002-3076-6221
| | - Heather Braund
- H. Braund is associate director of scholarship and simulation education, Office of Professional Development and Educational Scholarship, and assistant (adjunct) professor, Department of Biomedical and Molecular Sciences and School of Medicine, Queen's University, Kingston, Ontario, Canada; ORCID: https://orcid.org/0000-0002-9749-7193
| | - Damon J Dagnone
- D.J. Dagnone is associate professor, Department of Emergency Medicine, Queen's University, Kingston, Ontario, Canada; ORCID: https://orcid.org/0000-0001-6963-7948
| | - Laura McEwen
- L. McEwen is director of assessment and evaluation of postgraduate medical education and assistant professor, Department of Pediatrics, Postgraduate Medical Education, Queen's University, Kingston, Ontario, Canada; ORCID: https://orcid.org/0000-0003-2457-5311
| | - Nancy Dalgarno
- N. Dalgarno is director of education scholarship, Office of Professional Development and Educational Scholarship, and assistant professor (adjunct), Department of Biomedical and Molecular Sciences and Master of Health Professions Education, Queen's University, Kingston, Ontario, Canada; ORCID: https://orcid.org/0000-0001-7932-9949
| | - Karen W Schultz
- K.W. Schultz is professor, Department of Family Medicine, and associate dean of postgraduate medical education, Queen's University, Kingston, Ontario, Canada; ORCID: https://orcid.org/0000-0003-0208-3981
| | - Andrew K Hall
- A.K. Hall is associate professor and vice chair of education, Department of Emergency Medicine, University of Ottawa, and clinician educator, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada; ORCID: https://orcid.org/0000-0003-1227-5397
| |
Collapse
|
13
|
Yeates P, Maluf A, Cope N, McCray G, McBain S, Beardow D, Fuller R, McKinley RB. Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE). BMC MEDICAL EDUCATION 2023; 23:803. [PMID: 37885005 PMCID: PMC10605484 DOI: 10.1186/s12909-023-04774-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Accepted: 10/14/2023] [Indexed: 10/28/2023]
Abstract
PURPOSE Ensuring equivalence of examiners' judgements within distributed objective structured clinical exams (OSCEs) is key to both fairness and validity but is hampered by lack of cross-over in the performances which different groups of examiners observe. This study develops a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA) using it to compare examiners scoring from different OSCE sites for the first time. MATERIALS/ METHODS Within a summative 16 station OSCE, volunteer students were videoed on each station and all examiners invited to score station-specific comparator videos in addition to usual student scoring. Linkage provided through the video-scores enabled use of Many Facet Rasch Modelling (MFRM) to compare 1/ examiner-cohort and 2/ site effects on students' scores. RESULTS Examiner-cohorts varied by 6.9% in the overall score allocated to students of the same ability. Whilst only a tiny difference was apparent between sites, examiner-cohort variability was greater in one site than the other. Adjusting student scores produced a median change in rank position of 6 places (0.48 deciles), however 26.9% of students changed their rank position by at least 1 decile. By contrast, only 1 student's pass/fail classification was altered by score adjustment. CONCLUSIONS Whilst comparatively limited examiner participation rates may limit interpretation of score adjustment in this instance, this study demonstrates the feasibility of using VESCA for quality assurance purposes in large scale distributed OSCEs.
Collapse
Affiliation(s)
- Peter Yeates
- School of Medicine, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK.
- Fairfield General Hospital, Northern Care Alliance NHS Foundation Trust, Bury, Greater Manchester, UK.
| | - Adriano Maluf
- School of Medicine, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK
| | - Natalie Cope
- School of Medicine, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK
| | - Gareth McCray
- School of Medicine, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK
| | - Stuart McBain
- School of Medicine, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK
| | - Dominic Beardow
- School of Medicine, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK
| | - Richard Fuller
- Christie Education, Christie Hospitals NHS Foundation Trust, Manchester , UK
| | - Robert Bob McKinley
- School of Medicine, Keele University, David Weatherall Building, Keele, Staffordshire, ST5 5BG, UK
| |
Collapse
|
14
|
Curtis C, Kassam A, Lord J, Cooke LJ. Competence committees decision-making; an interplay of data, group orientation, and intangible impressions. BMC MEDICAL EDUCATION 2023; 23:748. [PMID: 37817152 PMCID: PMC10565978 DOI: 10.1186/s12909-023-04693-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2022] [Accepted: 09/16/2023] [Indexed: 10/12/2023]
Abstract
BACKGROUND The implementation of competency-based medical education and utilization of competence committees (CC) represents a paradigm shift in residency education. This qualitative study aimed to explore the methods used by two operational CC and their members to make decisions about progression and competence of their residents. METHODS An instrumental case study methodology was used to study the CC of two postgraduate training programs. Transcripts from observed CC meetings, member interviews, and guiding documents were analyzed using a constructivist grounded theory approach to reveal themes explaining the decision-making process. RESULTS Our study found that the CC followed a process that began within a social decision schema model and evolved to a discussion that invoked social influence theory, shared mental models, and social judgment scheme to clarify the points of contention. We identified that the CC decision-making was at risk of bias, primarily influenced by the group composition, the group orientation and individual members' mindset, as well as their personal experiences with the trainees. CONCLUSIONS Increased awareness of the sources of bias in CC functioning and familiarity with the CC role in competency-based medical education would enable committees to provide valuable feedback to all trainees regardless of their trajectory.
Collapse
Affiliation(s)
- Colleen Curtis
- Department of Pediatrics, University of Calgary, 28 Oki Dr. NW, Calgary, AB, T3B 6A8, Canada.
| | - Aliya Kassam
- Department of Community Health Sciences, University of Calgary, Calgary, Canada
| | - Jason Lord
- Departments of Emergency Medicine and Critical Care Medicine, University of Calgary, Calgary, Canada
| | - Lara J Cooke
- Department of Clinical Neurosciences, University of Calgary, Calgary, Canada
| |
Collapse
|
15
|
Lip A, Watling CJ, Ginsburg S. What does "Timely" Mean to Residents? Challenging Feedback Assumptions in Postgraduate Education. PERSPECTIVES ON MEDICAL EDUCATION 2023; 12:218-227. [PMID: 37334109 PMCID: PMC10275343 DOI: 10.5334/pme.1052] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Accepted: 05/29/2023] [Indexed: 06/20/2023]
Abstract
Introduction Current orthodoxy states that feedback should be timely and face-to-face, yet the optimal timing and mode of delivery for feedback is unclear. We explored what "optimal timing" means from residents' points of view as feedback providers and receivers, to ultimately inform strategies to optimize feedback in training. Methods As near-peers who have dual roles in both providing and receiving feedback, 16 subspecialty (PGY4 and 5) internal medicine residents were interviewed about their perceptions of the optimal timing and format of feedback. Using constructivist grounded theory, interviews were conducted and analyzed iteratively. Results Drawing on their experiences as providers and recipients, residents described simultaneously considering and weighing multiple factors when deciding on when and how to provide feedback. These included their own readiness to engage in providing meaningful feedback, the perceived receptiveness of the learner and the apparent urgency of feedback delivery (e.g., if patient safety was at stake). Face-to-face verbal feedback was valued for encouraging dialogue but could be uncomfortable and limited by time constraints. Written feedback could be more honest and concise, and the possibility of asynchronous delivery had potential to overcome issues with timing and discomfort. Discussion Participants' perceptions of the optimal timing of feedback challenge current assumptions about the benefits of "immediate" versus "delayed". The concept of "optimal timing" for feedback was found to be complex and context-dependent, defying a formulaic approach. There may be a role for asynchronous and/or written feedback, which has potential to address unique issues identified issues in near-peer relationships.
Collapse
Affiliation(s)
- Alyssa Lip
- Department of Medicine, Temerty Faculty of Medicine, University of Toronto, CA
| | - Christopher J. Watling
- Centre for Education Research and Innovation, Schulich School of Medicine & Dentistry, Western University, London, Ontario, CA
| | - Shiphra Ginsburg
- Department of Medicine, Sinai Health System and Faculty of Medicine, University of Toronto, scientist, Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, CA
- Canada Research Chair in Health Professions Education, CA
| |
Collapse
|
16
|
Kleijer EFW, Schuurmans MJ, Ten Cate O, Pool IA. Preceptors' considerations when entrusting professional activities to postgraduate nursing students: A qualitative study. NURSE EDUCATION TODAY 2023; 125:105799. [PMID: 36989637 DOI: 10.1016/j.nedt.2023.105799] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/04/2022] [Revised: 02/28/2023] [Accepted: 03/09/2023] [Indexed: 06/19/2023]
Abstract
BACKGROUND Nurse-preceptors regularly struggle to evaluate students' readiness to take care of patients unsupervised, even with sophisticated workplace-based assessment tools. Preceptors' gut feelings are not always captured well, but are critical for judgement of readiness for learner entrustment with care tasks. Studies in medical education report features that clinicians consider important when trusting students with clinical responsibilities that might also apply in nursing. OBJECTIVES To unravel preceptors' considerations when entrusting professional activities to postgraduate nursing students. The findings may contribute to the improvement of workplace-based assessments and the training of preceptors. METHODS Thematic analysis of semi-structured interviews with sixteen nurse-preceptors from three postgraduate nursing specialisations in Dutch hospitals. RESULTS Three themes emerged: CONCLUSIONS: For preceptors of postgraduate nursing students, entrustment requires more than merely insight into objectively measurable competencies. Entrusting is accompanied by subjectivity related to what preceptors expected of students. These expectations are in line with suggested factors in the literature-capability, integrity, reliability, agency, and humility-considered before entrusting students with clinical responsibilities identified in medical training. Entrusting is also accompanied by what preceptors realise about their own role in entrustment decisions. Combining different information sources made assessment more transparent and the implicit more explicit.
Collapse
Affiliation(s)
- E F Wilma Kleijer
- Academy, University Medical Centre Utrecht, Utrecht, The Netherlands.
| | - Marieke J Schuurmans
- Educational Centre, University Medical Centre Utrecht, Utrecht, The Netherlands.
| | - Olle Ten Cate
- Utrecht Centre for Research and Development of Health Professions Education, University Medical Centre Utrecht, Utrecht, The Netherlands.
| | | |
Collapse
|
17
|
Ginsburg S, Stroud L. Necessary but Insufficient and Possibly Counterproductive: The Complex Problem of Teaching Evaluations. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:300-303. [PMID: 36538693 DOI: 10.1097/acm.0000000000005006] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
The evaluation of clinical teachers' performance has long been a subject of research and debate, yet teaching evaluations (TEs) by students remain problematic. Despite their intuitive appeal, there is little evidence that TEs are associated with students' learning in the classroom or clinical setting. TEs are also subject to many forms of bias and are confounded by construct-irrelevant factors, such as the teacher's physical attractiveness or personality. Yet they are used almost exclusively as evaluations of and feedback to teachers. In this commentary, the authors review the literature on what TEs are meant to do, what they actually do in the real world, and their overall impact. The authors also consider productive ways forward. While TEs are certainly necessary to provide the crucial student voice, they are insufficient as the sole way to assess teachers. Further, they are often counterproductive. TEs carry so much weight for faculty that they can act as a disincentive for teachers to challenge learners and provide them with the critical feedback they often need, lest students give them poor ratings. To address these challenges, changes are needed, including embedding TEs in a programmatic assessment framework. For example, TEs might be used for formative feedback only, while other sources of data, such as peer assessments, learning outcomes, 360-degree feedback, and teacher reflections, could be collated into a portfolio to provide a more meaningful evaluation for teachers. Robust, transparent systems should be in place that dictate how TE data are used and to ensure they are not misused. Clinical teachers who do not "fail to fail" learners but instead take the time and effort to identify and support learners in difficulty should be recognized and rewarded. Learners need this support to succeed and the obligation to protect patients demands it.
Collapse
Affiliation(s)
- Shiphra Ginsburg
- S. Ginsburg is professor of medicine, Department of Medicine, Sinai Health System and Faculty of Medicine, University of Toronto, and scientist, Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, Canada, and Canada Research Chair in Health Professions Education; ORCID: http://orcid.org/0000-0002-4595-6650
| | - Lynfa Stroud
- L. Stroud is associate professor, Department of Medicine and Sunnybrook Health Sciences Centre, and education researcher, Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
18
|
Mooney CJ, Pascoe JM, Blatt AE, Lang VJ, Kelly MS, Braun MK, Burch JE, Stone RT. Predictors of faculty narrative evaluation quality in medical school clerkships. MEDICAL EDUCATION 2022; 56:1223-1231. [PMID: 35950329 DOI: 10.1111/medu.14911] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 08/01/2022] [Accepted: 08/08/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Narrative approaches to assessment provide meaningful and valid representations of trainee performance. Yet, narratives are frequently perceived as vague, nonspecific and low quality. To date, there is little research examining factors associated with narrative evaluation quality, particularly in undergraduate medical education. The purpose of this study was to examine associations of faculty- and student-level characteristics with the quality of faculty member's narrative evaluations of clerkship students. METHODS The authors reviewed faculty narrative evaluations of 50 students' clinical performance in their inpatient medicine and neurology clerkships, resulting in 165 and 87 unique evaluations in the respective clerkships. The authors evaluated narrative quality using the Narrative Evaluation Quality Instrument (NEQI). The authors used linear mixed effects modelling to predict total NEQI score. Explanatory covariates included the following: time to evaluation completion, number of weeks spent with student, faculty total weeks on service per year, total faculty years in clinical education, student gender, faculty gender, and an interaction term between student and faculty gender. RESULTS Significantly higher narrative evaluation quality was associated with a shorter time to evaluation completion, with NEQI scores decreasing by approximately 0.3 points every 10 days following students' rotations (p = .004). Additionally, women faculty had statistically higher quality narrative evaluations with NEQI scores 1.92 points greater than men faculty (p = .012). All other covariates were not significant. CONCLUSIONS The quality of faculty members' narrative evaluations of medical students was associated with time to evaluation completion and faculty gender but not faculty experience in clinical education, faculty weeks on service, or the amount of time spent with students. Findings advance understanding on ways to improve the quality of narrative evaluations which are imperative given assessment models that will increase the volume and reliance on narratives.
Collapse
Affiliation(s)
- Christopher J Mooney
- School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
| | - Jennifer M Pascoe
- School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
| | - Amy E Blatt
- School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
| | - Valerie J Lang
- School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
| | | | - Melanie K Braun
- School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
| | - Jaclyn E Burch
- School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
| | | |
Collapse
|
19
|
Hatala R, Ginsburg S, Gauthier S, Melvin L, Taylor D, Gingerich A. Supervising the senior medical resident: Entrusting the role, supporting the tasks. MEDICAL EDUCATION 2022; 56:1194-1202. [PMID: 35869566 DOI: 10.1111/medu.14883] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/15/2022] [Revised: 07/09/2022] [Accepted: 07/19/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Postgraduate competency-based medical education has been implemented with programmatic assessment that relies on entrustment-based ratings. Yet, in less procedurally oriented specialties such as internal medicine, the relationship between entrustment and supervision remains unclear. We undertook the current study to address how internal medicine supervisors conceptualise entrusting senior medical residents while supervising them on the acute care wards. METHODS Guided by constructivist grounded theory, we interviewed 19 physicians who regularly supervised senior internal medicine residents on inpatient wards at three Canadian universities. We developed a theoretical model through iterative cycles of data collection and analysis using a constant comparative process. RESULTS On the internal medicine ward, the senior resident role is viewed as a fundamentally managerial and rudimentary version of the supervisor's role. Supervisors come to trust their residents in the senior role through an early 'hands-on' period of assessment followed by a gradual withdrawal of support to promote independence. When considering entrustment, supervisors focused on entrusting a particular scope of the senior resident role as opposed to entrustment of individual tasks. Irrespective of the scope of the role that was entrusted, supervisors at times stepped in and stepped back to support specific tasks. CONCLUSION Supervisors' stepping in and stepping back to support individual tasks on the acute care ward has an inconsistent relationship to their entrustment of the resident with a particular scope of the senior resident role. In this context, entrustment-based assessment would need to capture more of the holistic perspective of the supervisor's entrustment of the senior resident role. Understanding the dance of supervision, from relatively static overall support of the resident in their role, to fluidly stepping in and out for specific patient care tasks, allows us insight into the affordances of the supervisory relationship and how it may be leveraged for assessment.
Collapse
Affiliation(s)
- Rose Hatala
- Department of Medicine, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - Shiphra Ginsburg
- Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
- Wilson Centre for Education, University of Toronto, Toronto, Ontario, Canada
| | - Stephen Gauthier
- Department of Medicine, Faculty of Medicine, Queen's University, Kingston, Ontario, Canada
| | - Lindsay Melvin
- Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - David Taylor
- Department of Medicine, Faculty of Medicine, Queen's University, Kingston, Ontario, Canada
| | - Andrea Gingerich
- Northern Medical Program, University of Northern British Columbia, Prince George, British Columbia, Canada
| |
Collapse
|
20
|
Branfield Day L, Rassos J, Billick M, Ginsburg S. 'Next steps are…': An exploration of coaching and feedback language in EPA assessment comments. MEDICAL TEACHER 2022; 44:1368-1375. [PMID: 35944554 DOI: 10.1080/0142159x.2022.2098098] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE Entrustable Professional Activities (EPA) assessments are intended to facilitate meaningful, low-stakes coaching and feedback, partly through the provision of written comments. We sought to explore EPA assessment comments provided to internal medicine (IM) residents for evidence of feedback and coaching language as well as politeness. METHODS We collected all written comments from EPA assessments of communication from a first-year IM resident cohort at the University of Toronto. Sensitized by politeness theory, we analyzed data using principles of constructivist grounded theory. RESULTS Nearly all EPA assessments (94%) contained written feedback based on focused clinical encounters. The majority of comments demonstrated coaching language, including phrases like 'don't forget to,' and 'next steps are,' followed by specific suggestions for improvement. A variety of words, including 'autonomy' and 'independence' denoted entrustment decisions. Linguistic politeness strategies such as hedging were pervasive, seemingly to minimize harm to the supervisor-trainee relationship. CONCLUSION Evidence of written coaching feedback suggests that EPA assessment comments are being used as intended as a means of formative feedback to promote learning. Yet, the frequent use of polite language suggests that EPAs may be higher-stakes than expected, highlighting a need for changes to the assessment culture and improved feedback literacy.
Collapse
Affiliation(s)
- Leora Branfield Day
- Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - James Rassos
- Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Maxime Billick
- Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Shiphra Ginsburg
- Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
- Wilson Centre for Research in Education, Toronto, Canada
| |
Collapse
|
21
|
Pearce J. What do student experiences of programmatic assessment tell us about scoring programmatic assessment data? MEDICAL EDUCATION 2022; 56:872-875. [PMID: 35698736 DOI: 10.1111/medu.14852] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Accepted: 06/06/2022] [Indexed: 06/15/2023]
Affiliation(s)
- Jacob Pearce
- Australian Council for Educational Research - Tertiary Education (Assessment), Camberwell, Victoria, Australia
| |
Collapse
|
22
|
Schumacher DJ, Michelson C, Winn AS, Turner DA, Elshoff E, Kinnear B. Making prospective entrustment decisions: Knowing limits, seeking help and defaulting. MEDICAL EDUCATION 2022; 56:892-900. [PMID: 35263474 DOI: 10.1111/medu.14797] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Revised: 02/28/2022] [Accepted: 03/02/2022] [Indexed: 06/14/2023]
Abstract
INTRODUCTION Many studies focused on entrustment have not considered prospective entrustment decisions, where clinical competency and entrustment committees determine how much supervision trainees will require in the future for tasks occurring in not completely known contexts. The authors sought to explore factors that influence and determine prospective entrustment decisions made by members of such committees in graduate medical education (GME) and undergraduate medical education (UME). METHODS The authors conducted a constructivist grounded theory study with 23 faculty participants from GME and UME clinical competency and entrustment committees in the United States between October 2020 and March 2021. Interviews sought to explore factors and considerations participants weigh in making prospective entrustment decisions about trainees. Data collection and analysis occurred in an iterative fashion, ensuring constant comparison. Theoretical sampling was used to confirm, disconfirm and elaborate on the evolving results. RESULTS Trainees' ability to know limits and seek help is the foundation of participants' prospective entrustment decision making. Most participants, however, describe a presumption of trainee readiness to progress and describe commonly making default prospective entrustment decisions unless 'red flags' in performance are present. Although participants desire sufficient and trusted data about trainee performance to inform decisions, they often lack it. Finally, the perceived permanence and consequences of prospective entrustment decisions influence how participants weigh other factors contributing to prospective entrustment decisions. CONCLUSION Trainees' ability to know limits and seek help appears to be the foundation of prospective entrustment decision making. Training programmes should strive to collect and employ robust data supporting and questioning the presence of these attributes.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Catherine Michelson
- Boston Medical Center, Boston University School of Medicine, Boston, Massachusetts, USA
| | - Ariel S Winn
- Boston Children's Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | | | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
23
|
Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments. J Gen Intern Med 2022; 37:2200-2207. [PMID: 35710663 PMCID: PMC9296736 DOI: 10.1007/s11606-022-07509-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Accepted: 03/24/2022] [Indexed: 10/18/2022]
Abstract
BACKGROUND Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding. OBJECTIVE In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks. DESIGN Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel. PARTICIPANTS A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor. MAIN MEASURES Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors. KEY RESULTS IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty. CONCLUSIONS To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.
Collapse
|
24
|
Jeyalingam T, Walsh CM, Tavares W, Mylopoulos M, Hodwitz K, Liu LWC, Heitman SJ, Brydges R. Variable or Fixed? Exploring Entrustment Decision Making in Workplace- and Simulation-Based Assessments. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1057-1064. [PMID: 35263307 DOI: 10.1097/acm.0000000000004661] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE Many models of competency-based medical education (CBME) emphasize assessing entrustable professional activities (EPAs). Despite the centrality of EPAs, researchers have not compared rater entrustment decisions for the same EPA across workplace- and simulation-based assessments. This study aimed to explore rater entrustment decision making across these 2 assessment settings. METHOD An interview-based study using a constructivist grounded theory approach was conducted. Gastroenterology faculty at the University of Toronto and the University of Calgary completed EPA assessments of trainees' endoscopic polypectomy performance in both workplace and simulation settings between November 2019 and January 2021. After each assessment, raters were interviewed to explore how and why they made entrustment decisions within and across settings. Transcribed interview data were coded iteratively using constant comparison to generate themes. RESULTS Analysis of 20 interviews with 10 raters found that participants (1) held multiple meanings of entrustment and expressed variability in how they justified their entrustment decisions and scoring, (2) held personal caveats for making entrustment decisions "comfortably" (i.e., authenticity, task-related variability, opportunity to assess trainee responses to adverse events, and the opportunity to observe multiple performances over time), (3) experienced cognitive tensions between formative and summative purposes when assessing EPAs, and (4) experienced relative freedom when using simulation to formatively assess EPAs but constraint when using only simulation-based assessments for entrustment decision making. CONCLUSIONS Participants spoke about and defined entrustment variably, which appeared to produce variability in how they judged entrustment across participants and within and across assessment settings. These rater idiosyncrasies suggest that programs implementing CBME must consider how such variability affects the aggregation of EPA assessments, especially those collected in different settings. Program leaders might also consider how to fulfill raters' criteria for comfortably making entrustment decisions by ensuring clear definitions and purposes when designing and integrating workplace- and simulation-based assessments.
Collapse
Affiliation(s)
- Thurarshen Jeyalingam
- T. Jeyalingam is an advanced fellow in luminal therapeutic endoscopy, University of Calgary, Calgary, Alberta, Canada; ORCID: http://orcid.org/0000-0002-7254-9639
| | - Catharine M Walsh
- C.M. Walsh is a staff gastroenterologist, Division of Gastroenterology, Hepatology and Nutrition, educational researcher, SickKids Learning Institute, scientist, Child Health Evaluative Sciences, SickKids Research Institute, Hospital for Sick Children, scientist, Wilson Centre, and associate professor of paediatrics, University of Toronto, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0003-3928-703X
| | - Walter Tavares
- W. Tavares is assistant professor and scientist, Wilson Centre and Temerty Faculty of Medicine, University Health Network and University of Toronto, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0001-8267-9448
| | - Maria Mylopoulos
- M. Mylopoulos is associate professor, Department of Paediatrics, and scientist and associate director, Wilson Centre, University of Toronto, Temerty Faculty of Medicine, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0003-0012-5375
| | - Kathryn Hodwitz
- K. Hodwitz is a clinical research specialist, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0003-3099-1709
| | - Louis W C Liu
- L.W.C. Liu is associate professor, Department of Medicine, University of Toronto, and head, Division of Gastroenterology and Hepatology, University Health Network and Sinai Health, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0001-6899-7941
| | - Steven J Heitman
- S.J. Heitman is associate professor, Departments of Medicine and Community Health Sciences, Cumming School of Medicine, holds the N.B. Hershfield Chair in Therapeutic Endoscopy, University of Calgary, is medical director, Forzani & MacPhail Colon Cancer Screening Centre, and scientific director, Digestive Health Strategic Clinical Network, Alberta Health Services, Calgary, Alberta, Canada; ORCID: http://orcid.org/0000-0002-4952-779X
| | - Ryan Brydges
- R. Brydges is a scientist and holds the Professorship in Technology-Enabled Education, St. Michael's Hospital, Unity Health Toronto, and is associate professor, Department of Medicine, University of Toronto, Toronto, Ontario, Canada; ORCID: https://orcid.org/0000-0001-5203-7049
| |
Collapse
|
25
|
Barman L, McGrath C, Josephsson S, Silén C, Bolander Laksov K. Safeguarding fairness in assessments-How teachers develop joint practices. MEDICAL EDUCATION 2022; 56:651-659. [PMID: 35263464 PMCID: PMC9310582 DOI: 10.1111/medu.14789] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Revised: 02/03/2022] [Accepted: 02/26/2022] [Indexed: 06/14/2023]
Abstract
INTRODUCTION In light of reforms demanding increased transparency of student performance assessments, this study offers an in-depth perspective of how teachers develop their assessment practice. Much is known about factors that influence assessments, and different solutions claim to improve the validity and reliability of assessments of students' clinical competency. However, little is known about how teachers go about improving their assessment practices. This study aims to contribute empirical findings about how teachers' assessment practice may change when shared criteria for assessing students' clinical competency are developed and implemented. METHODS Using a narrative-in-action research approach grounded in narrative theory about human sense-making, one group including nine health professions teachers was studied over a period of 1 year. Drawing upon data from observations, interviews, formal documents and written reflections from these teachers, we performed a narrative analysis to reveal how these teachers made sense of experiences associated with the development and implementation of joint grading criteria for assessing students' clinical performances. RESULTS The findings present a narrative showing how a shared assessment practice took years to develop and was based on the teachers changed approach to scrutiny. The teachers became highly motivated to use grading criteria to ensure fairness in assessments, but more importantly, to fulfil their moral obligation towards patients. The narrative also demonstrates how these teachers reasoned about dilemmas that arose when they applied standardised assessment criteria. DISCUSSION The narrative analysis shows clearly how teachers' development and application of assessment standards are embedded in local practices. Our findings highlight the importance of teachers' joint discussions on how to interpret criteria applied in formative and summative assessments of students' performances. In particular, teachers' different approaches to assessing 'pieces of skills' versus making holistic judgements on students' performances, regardless of whether the grading criteria are clear and well-articulated on paper, should be acknowledged. Understanding the journey that these teachers made gives new perspectives as to how faculty can be supported when assessments of professionalism and clinical competency are developed.
Collapse
Affiliation(s)
- Linda Barman
- Department of Learning in Engineering SciencesKTH Royal Institute of TechnologyStockholmSweden
| | - Cormac McGrath
- Department of EducationStockholm UniversityStockholmSweden
| | - Staffan Josephsson
- Department of Neurobiology, Care Sciences and SocietyKarolinska InstitutetStockholmSweden
| | - Charlotte Silén
- Department of Learning, Informatics, Management and EthicsKarolinska InstitutetStockholmSweden
| | - Klara Bolander Laksov
- Department of EducationStockholm UniversityStockholmSweden
- Department of Learning, Informatics, Management and EthicsKarolinska InstitutetStockholmSweden
| |
Collapse
|
26
|
Ross S, Hamza D, Zulla R, Stasiuk S, Nichols D. Development of and Preliminary Validity Evidence for the EFeCT Feedback Scoring Tool. J Grad Med Educ 2022; 14:71-79. [PMID: 35222824 PMCID: PMC8848874 DOI: 10.4300/jgme-d-21-00602.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/08/2021] [Revised: 08/31/2021] [Accepted: 11/02/2021] [Indexed: 02/03/2023] Open
Abstract
BACKGROUND Narrative feedback, like verbal feedback, is essential to learning. Regardless of form, all feedback should be of high quality. This is becoming even more important as programs incorporate narrative feedback into the constellation of evidence used for summative decision-making. Continuously improving the quality of narrative feedback requires tools for evaluating it, and time to score. A tool is needed that does not require clinical educator expertise so scoring can be delegated to others. OBJECTIVE To develop an evidence-based tool to evaluate the quality of documented feedback that could be reliably used by clinical educators and non-experts. METHODS Following a literature review to identify elements of high-quality feedback, an expert consensus panel developed the scoring tool. Messick's unified concept of construct validity guided the collection of validity evidence throughout development and piloting (2013-2020). RESULTS The Evaluation of Feedback Captured Tool (EFeCT) contains 5 categories considered to be essential elements of high-quality feedback. Preliminary validity evidence supports content, substantive, and consequential validity facets. Generalizability evidence supports that EFeCT scores assigned to feedback samples show consistent interrater reliability scores between raters across 5 sessions, regardless of level of medical education or clinical expertise (Session 1: n=3, ICC=0.94; Session 2: n=6, ICC=0.90; Session 3: n=5, ICC=0.91; Session 4: n=6, ICC=0.89; Session 5: n=6, ICC=0.92). CONCLUSIONS There is preliminary validity evidence for the EFeCT as a useful tool for scoring the quality of documented feedback captured on assessment forms. Generalizability evidence indicated comparable EFeCT scores by raters regardless of level of expertise.
Collapse
Affiliation(s)
- Shelley Ross
- Shelley Ross, PhD, is Professor, Department of Family Medicine, University of Alberta, Edmonton, AB, Canada
| | - Deena Hamza
- Deena Hamza, PhD, is Competency-Based Medical Education Evaluation Lead for Postgraduate Medical Education, University of Alberta, Edmonton, AB, Canada
| | - Rosslynn Zulla
- Rosslynn Zulla, PhD, is a Specialist/Advisor, Faculty of Social Work, University of Calgary, AB, Canada
| | - Samantha Stasiuk
- Samantha Stasiuk, MD, MHPE, is Clinical Assistant Professor, Department of Family Practice, University of British Columbia, BC, Canada
| | - Darren Nichols
- Darren Nichols, MD, is Associate Professor, Department of Family Medicine, University of Alberta, Edmonton, AB, Canada
| |
Collapse
|
27
|
Thoma B, Ellaway RH, Chan TM. From Utopia Through Dystopia: Charting a Course for Learning Analytics in Competency-Based Medical Education. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S89-S95. [PMID: 34183609 DOI: 10.1097/acm.0000000000004092] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The transition to the assessment of entrustable professional activities as part of competency-based medical education (CBME) has substantially increased the number of assessments completed on each trainee. Many CBME programs are having difficulty synthesizing the increased amount of assessment data. Learning analytics are a way of addressing this by systematically drawing inferences from large datasets to support trainee learning, faculty development, and program evaluation. Early work in this field has tended to emphasize the significant potential of analytics in medical education. However, concerns have been raised regarding data security, data ownership, validity, and other issues that could transform these dreams into nightmares. In this paper, the authors explore these contrasting perspectives by alternately describing utopian and dystopian futures for learning analytics within CBME. Seeing learning analytics as an important way to maximize the value of CBME assessment data for organizational development, they argue that their implementation should continue within the guidance of an ethical framework.
Collapse
Affiliation(s)
- Brent Thoma
- B. Thoma is associate professor, Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada, and clinician educator, Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada; ORCID: https://orcid.org/0000-0003-1124-5786
| | - Rachel H Ellaway
- R.H. Ellaway is professor, Department of Community Health Sciences, and director, Office of Health and Medical Education Scholarship, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada; ORCID: https://orcid.org/0000-0002-3759-6624
| | - Teresa M Chan
- T.M. Chan is associate professor, Division of Emergency Medicine, Department of Medicine, assistant dean, Program for Faculty Development, Faculty of Health Sciences, and adjunct scientist, McMaster Education Research, Innovation, and Theory (MERIT) program, McMaster University, Hamilton, Ontario, Canada; ORCID: https://orcid.org/0000-0001-6104-462X
| |
Collapse
|
28
|
Schumacher DJ, Turner DA. Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S1-S5. [PMID: 34183594 DOI: 10.1097/acm.0000000000004097] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
- Daniel J Schumacher
- D.J. Schumacher is associate professor of pediatrics, Cincinnati Children's Hospital Medical Center and University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - David A Turner
- D.A. Turner is vice president for competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina
| |
Collapse
|