1
|
Costich M, Friedman S, Robinson V, Catallozzi M. Implementation and faculty perception of outpatient medical student workplace-based assessments. CLINICAL TEACHER 2024; 21:e13751. [PMID: 38433555 DOI: 10.1111/tct.13751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 02/06/2024] [Indexed: 03/05/2024]
Abstract
BACKGROUND There is growing interest in use of entrustable professional activity (EPA)-grounded workplace-based assessments (WBAs) to assess medical students through direct observation in the clinical setting. However, there has been very little reflection on how these tools are received by the faculty using them to deliver feedback. Faculty acceptance of WBAs is fundamentally important to sustained utilisation in the clinical setting, and understanding faculty perceptions of the WBA as an adjunct for giving targeted feedback is necessary to guide future faculty development in this area. APPROACH Use of a formative EPA-grounded WBA was implemented in the ambulatory setting during the paediatrics clerkship following performance-driven training and frame-of-reference training with faculty. Surveys and semi-structured interviews with faculty members explored how faculty perceived the tool and its impact on feedback delivery. EVALUATION Faculty reported providing more specific, task-oriented feedback following implementation of the WBA, as well as greater timeliness of feedback and greater satisfaction with opportunities to provide feedback, although these later two findings did not reach significance. Themes from the interviews reflected the benefits of WBAs, persistent barriers to the provision of feedback and suggestions for improvement of the WBA. IMPLICATIONS EPA-grounded WBAs are feasible to implement in the outpatient primary care setting and improve feedback delivery around core EPAs. The WBAs positively impacted the way faculty conceptualise feedback and provide learners with more actionable, behaviour-based feedback. Findings will inform modifications to the WBA and future faculty development and training to allow for sustainable WBA utilisation in the core clerkship.
Collapse
Affiliation(s)
- Marguerite Costich
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Suzanne Friedman
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Victoria Robinson
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Marina Catallozzi
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Population and Family Health, Mailman School of Public Health, Columbia University Irving Medical Center, New York, New York, USA
| |
Collapse
|
2
|
Andreou V, Peters S, Eggermont J, Schoenmakers B. A needs assessment for enhancing workplace-based assessment: a grounded theory study. BMC MEDICAL EDUCATION 2024; 24:659. [PMID: 38872142 DOI: 10.1186/s12909-024-05636-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Accepted: 06/06/2024] [Indexed: 06/15/2024]
Abstract
OBJECTIVES Workplace-based assessment (WBA) has been vigorously criticized for not fulfilling its educational purpose by medical educators. A comprehensive exploration of stakeholders' needs regarding WBA is essential to optimize its implementation in clinical practice. METHOD Three homogeneous focus groups were conducted with three groups of stakeholders: General Practitioner (GP) trainees, GP trainers, and GP tutors. Due to COVID-19 measures, we opted for an online asynchronous form to enable participation. An constructivist grounded theory approach was used to employ this study and allow the identification of stakeholders' needs for using WBA. RESULTS Three core needs for WBA were identified in the analysis. Within GP Training, stakeholders found WBA essential, primarily, for establishing learning goals, secondarily, for assessment purposes, and, lastly, for providing or receiving feedback. CONCLUSION All stakeholders perceive WBA as valuable when it fosters learning. The identified needs were notably influenced by agency, trust, availability, and mutual understanding. These were facilitating factors influencing needs for WBA. Embracing these insights can significantly illuminate the landscape of workplace learning culture for clinical educators and guide a successful implementation of WBA.
Collapse
Affiliation(s)
- Vasiliki Andreou
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium.
| | - Sanne Peters
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
- School of Health Sciences, Faculty of Medicine, Dentistry and Health Sciences, The University of Melbourne, Melbourne, Australia
| | - Jan Eggermont
- Department of Cellular and Molecular Medicine, KU Leuven, Leuven, Belgium
| | - Birgitte Schoenmakers
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
| |
Collapse
|
3
|
de Laat JM, van der Horst-Schrivers AN, Appelman-Dijkstra NM, Bisschop PH, Dreijerink KM, Drent ML, van de Klauw MM, de Ranitz WL, Stades AM, Stikkelbroeck NM, Timmers HJ, ten Cate O. Assessment of Entrustable Professional Activities Among Dutch Endocrine Supervisors. JOURNAL OF CME 2024; 13:2360137. [PMID: 38831939 PMCID: PMC11146265 DOI: 10.1080/28338073.2024.2360137] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/04/2024] [Accepted: 05/19/2024] [Indexed: 06/05/2024]
Abstract
Entrustable Professional Activities (EPAs) are an important tool to support individualisation of medical training in a competency-based setting and are increasingly implemented in the clinical speciality training for endocrinologist. This study aims to assess interrater agreement and factors that potentially impact EPA scores. Five known factors that affect entrustment decisions in health profesions training (capability, integrity, reliability, humility, agency) were used in this study. A case-vignette study using standardised written cases. Case vignettes (n = 6) on the topics thyroid disease, pituitary disease, adrenal disease, calcium and bone disorders, diabetes mellitus, and gonadal disorders were written by two endocrinologists and a medical education expert and assessed by endocrinologists experienced in the supervision of residents in training. Primary outcome is the inter-rater agreement of entrustment decisions for endocrine EPAs among raters. Secondary outcomes included the dichotomous interrater agreement (entrusted vs. non-entrusted), and an exploration of factors that impact decision-making. The study protocol was registered and approved by the Ethical Review Board of the Netherlands Association for Medical Education (NVMO-ERB # 2020.2.5). Nine endocrinologists from six different academic regions participated. Overall, the Fleiss Kappa measure of agreement for the EPA level was 0.11 (95% CI: 0.03-0.22) and for the entrustment decision 0.24 (95% CI 0.11-0.37). Of the five features that impacted the entrustment decision, capability was ranked as the most important by a majority of raters (56%-67%) in every case. There is a considerable discrepancy between the EPA levels assigned by different raters. These findings emphasise the need to base entrustment decisions on multiple observations, made by a team of supervisors and enriched with factors other than direct medical competence.
Collapse
Affiliation(s)
- Joanne M. de Laat
- Department of Internal Medicine, Division of Endocrinology, Radboud University Medical Center, Nijmegen, The Netherlands
- Utrecht Center for Research and Development of Health Professions Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | | | | | - Peter H. Bisschop
- Department of Endocrinology and Metabolism, Amsterdam UMC, Location Academic Medical Center, Amsterdam, The Netherlands
| | - Koen M.A. Dreijerink
- Department of Internal Medicine, Amsterdam UMC, Location VU University Medical Center, Amsterdam, The Netherlands
| | - Madeleine L. Drent
- Department of Internal Medicine, Amsterdam UMC, Location VU University Medical Center, Amsterdam, The Netherlands
| | - Melanie M. van de Klauw
- Department of Endocrinology, University Medical Center Groningen, Groningen, The Netherlands
| | - Wendela L. de Ranitz
- Department of Endocrinology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Aline M.E. Stades
- Department of Endocrinology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Nike M.M.L. Stikkelbroeck
- Department of Internal Medicine, Division of Endocrinology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Henri J.L.M. Timmers
- Department of Internal Medicine, Division of Endocrinology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Olle ten Cate
- Utrecht Center for Research and Development of Health Professions Education, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
4
|
Sims DA, Lucio-Ramirez CA, Cilliers FJ. Factors influencing clinician-educators' assessment practice in varied Southern contexts: a health behaviour theory perspective. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2024:10.1007/s10459-024-10341-3. [PMID: 38811446 DOI: 10.1007/s10459-024-10341-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 05/12/2024] [Indexed: 05/31/2024]
Abstract
In many contexts, responsibility for exit-level assessment design and implementation in undergraduate medical programmes lies with individuals who convene clinical clerkships. Their assessment practice has significant consequences for students' learning and the patients and communities that graduates will serve. Interventions to enhance assessment must involve these assessors, yet little is known about factors influencing their assessment practice. The purpose of this study was to explore factors that influence assessment practice of clerkship convenors in three varied low-and-middle income contexts in the global South. Taking assessment practice as a behaviour, Health Behaviour Theory (HBT) was deployed as a theoretical framework to explore, describe and explain assessor behaviour. Thirty-one clinician-educators responsible for designing and implementing high-stakes clerkship assessment were interviewed in South Africa and Mexico. Interacting personal and contextual factors influencing clinician-educator assessment intention and action were identified. These included attitude, influenced by impact and response appraisal, and perceived self-efficacy; along with interpersonal, physical and organisational, and distal contextual factors. Personal competencies and conducive environments supported intention to action transition. While previous research has typically explored factors in isolation, the HBT framing enabled a systematic and coherent account of assessor behaviour. These findings add a particular contextual perspective to understanding assessment practice, yet also resonate with and extend existing work that predominantly emanates from high-income contexts in the global North. These findings provide a foundation for the planning of assessment change initiatives, such as targeted, multi-factorial faculty development.
Collapse
Affiliation(s)
- Danica Anne Sims
- University of Oxford, Oxford, UK.
- University of Johannesburg, Johannesburg, South Africa.
| | | | | |
Collapse
|
5
|
Ohlin S, King S, Takashima M, Ossenberg C, Henderson A. Learning in the workplace: Development of a simple language statement assessment tool that supports second-level nurse practice. Nurse Educ Pract 2024; 77:103983. [PMID: 38701684 DOI: 10.1016/j.nepr.2024.103983] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Revised: 04/15/2024] [Accepted: 04/18/2024] [Indexed: 05/05/2024]
Abstract
AIM To focus learning through clarity of the enrolled nurse (EN) role (a second tier nurse position) through development of a user-friendly workplace performance assessment tool commensurate with EN standards for practice. BACKGROUND Internationally, the nursing workforce comprises regulated and unregulated staff. In Australia, similar to other western countries, there are two tiers of regulated workforce, namely Registered Nurses (RNs) and Enrolled Nurses (ENs). Differences in RN and EN standards based on the education preparation are not always clearly differentiated in workplace practice. Roles are often seen as interchangeable: Improved clarity of both regulated and unregulated roles, when numbers of healthcare workers are burgeoning, assists performance assessment that guides further learning and safe practice. DESIGN Two phase sequential, non-experimental design. METHODS Phase one used focus groups (n=48), expert reference panel (n=8) and end-users (n=16) to develop simple language statements. Phase two involved field testing of the statements. FINDINGS A 30-item, criterion-based workplace performance tool was developed. Principal component analysis of completed tools indicated work could be organised around three key areas of practice, namely, higher order thinking and problem solving, routine daily activities of care and personal and social attributes. DISCUSSION Participants reported the statement items assisted in determining suitable activities and accompanying cues in discussing learning needs. Analysis assisted with discriminating broader elements of EN workplace performance. CONCLUSIONS Workplace learning is important for nurses to continue to build their capacity to deliver optimum care. Assessment tools that describe professional capability in plain language statements and provide examples of supportive behavioural cues help guide on-going learning through improving the validity and thereby consistency of assessment processes. Furthermore, comprehensible and meaningful statements and cues can readily be adopted by students and educators to target learning and feedback thereby enhancing clarity of the EN role, to distinguish from other nursing roles.
Collapse
Affiliation(s)
- Simone Ohlin
- Central Queensland University, Queensland, Australia
| | - Sue King
- Central Queensland University, Queensland, Australia
| | - Mari Takashima
- School of Nursing, Midwifery and Social Work, University of Queensland, Australia
| | - Christine Ossenberg
- Central Queensland University, Queensland, Australia; Princess Alexandra Hospital, Woolloongabba, Queensland, Australia
| | - Amanda Henderson
- Central Queensland University, Queensland, Australia; Princess Alexandra Hospital, Woolloongabba, Queensland, Australia.
| |
Collapse
|
6
|
Braund H, Dalgarno N, O’Dell R, Taylor DR. Making assessment a team sport: a qualitative study of facilitated group feedback in internal medicine residency. CANADIAN MEDICAL EDUCATION JOURNAL 2024; 15:14-26. [PMID: 38827914 PMCID: PMC11139793 DOI: 10.36834/cmej.75250] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/05/2024]
Abstract
Purpose Competency-based medical education relies on feedback from workplace-based assessment (WBA) to direct learning. Unfortunately, WBAs often lack rich narrative feedback and show bias towards Medical Expert aspects of care. Building on research examining interactive assessment approaches, the Queen's University Internal Medicine residency program introduced a facilitated, team-based assessment initiative ("Feedback Fridays") in July 2017, aimed at improving holistic assessment of resident performance on the inpatient medicine teaching units. In this study, we aim to explore how Feedback Fridays contributed to formative assessment of Internal Medicine residents within our current model of competency-based training. Method A total of 53 residents participated in facilitated, biweekly group assessment sessions during the 2017 and 2018 academic year. Each session was a 30-minute facilitated assessment discussion done with one inpatient team, which included medical students, residents, and their supervising attending. Feedback from the discussion was collected, summarized, and documented in narrative form in electronic WBA forms by the program's assessment officer for the residents. For research purposes, verbatim transcripts of feedback sessions were analyzed thematically. Results The researchers identified four major themes for feedback: communication, intra- and inter-personal awareness, leadership and teamwork, and learning opportunities. Although feedback related to a broad range of activities, it showed strong emphasis on competencies within the intrinsic CanMEDS roles. Additionally, a clear formative focus in the feedback was another important finding. Conclusions The introduction of facilitated team-based assessment in the Queen's Internal Medicine program filled an important gap in WBA by providing learners with detailed feedback across all CanMEDS roles and by providing constructive recommendations for identified areas for improvement.
Collapse
Affiliation(s)
- Heather Braund
- Office of Professional Development and Educational Scholarship, Ontario, Canada
- Faculty of Education, Queen’s University, Ontario, Canada
| | - Nancy Dalgarno
- Office of Professional Development and Educational Scholarship, Ontario, Canada
- Department of Biomedical and Molecular Sciences, Faculty of Health Sciences, Queen’s University, Ontario, Canada
| | - Rachel O’Dell
- Department of Internal Medicine, Faculty of Health Sciences, Queen’s University, Ontario, Canada
| | - David R Taylor
- Academy for Teachers and Educators, Department of Medicine, Queen’s University, Ontario, Canada
| |
Collapse
|
7
|
Ebneter AS, Kaya E, Mair P, Affollter B, Eychmueller S. Basic Training in Palliative Medicine for Internal Medicine Residents: Pilot Testing of a Canadian Model in Switzerland. Palliat Med Rep 2024; 5:171-176. [PMID: 38665225 PMCID: PMC11043622 DOI: 10.1089/pmr.2024.0004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/28/2024] [Indexed: 04/28/2024] Open
Abstract
Background In Switzerland, palliative care (PC) clinical training is well established at undergraduate and specialist postgraduate levels. However, postgraduate nonspecialist training curricula are less documented. Local Problem A structured curriculum for nonspecialist rotation within internal medicine (IM) in specialized PC wards is lacking. Objective To pilot two versions of a PC nonspecialist curriculum for IM residents in Swiss PC units. Methods In the pilot phase, two curricula-short immersion (3-10 weeks, based on the University of Toronto's Internal-Medicine PC Rotation) and standard nonspecialist (11-18 weeks, based on the Canadian Society of Palliative Care Physician Competencies)-were assessed using a mixed-method online survey. One university and two nonuniversity sites participated. The analysis was descriptive. Results Five residents and eight supervisors of five training rotations (July-October 2023) responded. Overall, curriculum quality and feasibility (content and time) received positive ratings across all groups, with high satisfaction concerning organization, educational design, learning support, climate, experience, and facilities. Nonuniversity sites were generally rated more positively than university sites. Qualitative feedback paralleled these findings, highlighting the curriculum's relevance and fit with learners' needs and suggesting potential simplifications and more personalized planning. Conclusions Establishing short and standard duration curricula for a PC program is viable and well received by nonspecialist trainees. Future implementation should concentrate on personalized learning objectives and streamlining the content and structure of the competencies. Cooperation within various training settings (university and regional hospitals) as well as on an international level (e.g., Canada-Switzerland) may further improve the quality of the proposed training formats.
Collapse
Affiliation(s)
- Andreas Samuel Ebneter
- University Center for Palliative Care, University Hospital, Inselspital Bern, Switzerland
| | - Ebru Kaya
- Division of Palliative Medicine, University of Toronto and University Health Network, Toronto, Ontario, Canada
| | - Petra Mair
- Palliative Care Unit, Internal Medicine Clinic, Spital Thun, Thun, Switzerland
| | - Barbara Affollter
- Palliative Care Unit, Internal Medicine Clinic, Spital Emmental, Burgdorf, Switzerland
| | - Steffen Eychmueller
- University Center for Palliative Care, University Hospital, Inselspital Bern, Switzerland
| |
Collapse
|
8
|
Frank JR, Karpinski J, Sherbino J, Snell LS, Atkinson A, Oswald A, Hall AK, Cooke L, Dojeiji S, Richardson D, Cheung WJ, Cavalcanti RB, Dalseg TR, Thoma B, Flynn L, Gofton W, Dudek N, Bhanji F, Wong BMF, Razack S, Anderson R, Dubois D, Boucher A, Gomes MM, Taber S, Gorman LJ, Fulford J, Naik V, Harris KA, St. Croix R, van Melle E. Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education. PERSPECTIVES ON MEDICAL EDUCATION 2024; 13:201-223. [PMID: 38525203 PMCID: PMC10959143 DOI: 10.5334/pme.1096] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Accepted: 02/16/2024] [Indexed: 03/26/2024]
Abstract
Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.
Collapse
Affiliation(s)
- Jason R. Frank
- Centre for Innovation in Medical Education and Professor, Department of Emergency Medicine, Faculty of Medicine, University of Ottawa, ON, Canada
| | - Jolanta Karpinski
- Department of Medicine, University of Ottawa, Ottawa, ON, Canada
- Competency Based Medical Education, University of Ottawa, Ottawa, ON, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| | | | - Linda S. Snell
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Medicine and Health Sciences Education, McGill University, Montreal, QC, Canada
| | - Adelle Atkinson
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Paediatrics, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Anna Oswald
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Medicine, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada
- Competency Based Medical Education, University of Alberta, Edmonton, AB, Canada
| | - Andrew K. Hall
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Emergency Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Lara Cooke
- Division of Neurology, Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Susan Dojeiji
- Physical Medicine and Rehabilitation, University of Ottawa, Ottawa, ON, Canada
| | - Denyse Richardson
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Physical Medicine and Rehabilitation, Queen’s University, Kingston, ON, Canada
| | - Warren J. Cheung
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Medicine, University of Toronto, Toronto, ON, Canada
| | - Rodrigo B. Cavalcanti
- Department of Medicine, University of Toronto, Toronto, ON, Canada
- HoPingKong Centre, University Health Network, Toronto, ON, Canada
| | - Timothy R. Dalseg
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Division of Emergency Medicine, University of Toronto, Toronto, ON, Canada
| | - Brent Thoma
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Emergency Medicine, University of Saskatchewan, Saskatoon, SK, Canada
| | - Leslie Flynn
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Departments of Psychiatry and Family Medicine, and Co-Director Master of Health Sciences Education, Queen’s University, Kingston, ON, Canada
| | - Wade Gofton
- Department of Surgery (Division of Orthopedic Surgery), The Ottawa Hospital and University of Ottawa, Ottawa, ON, Canada
| | - Nancy Dudek
- Department of Medicine (Division of Physical Medicine & Rehabilitation) and The Ottawa Hospital, University of Ottawa, Ottawa, Ontario, Canada
| | - Farhan Bhanji
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Faculty of Medicine and Health Sciences, McGill University, Montreal, QC, Canada
| | - Brian M.-F. Wong
- Centre for Quality Improvement and Patient Safety, University of Toronto, Toronto, Canada
| | - Saleem Razack
- Centre for Health Education Scholarship, University of British Columbia and BC Children’s Hospital, Vancouver, BC, Canada
| | - Robert Anderson
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Northern Ontario School of Medicine University, Sudbury, ON, Canada
| | - Daniel Dubois
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Andrée Boucher
- Department of Medicine (Division of Endocrinology), Universitéde Montréal, Montréal, QC, Canada
| | - Marcio M. Gomes
- Department of Pathology and Laboratory Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Sarah Taber
- Office of Standards and Assessment, Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| | - Lisa J. Gorman
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
| | - Jane Fulford
- Canadian Internet Registration Authority, Canada
| | - Viren Naik
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, ON, Canada
| | - Kenneth A. Harris
- Royal College of Physicians and Surgeons of Canada, Canada
- Emeritus, Western University, Canada
| | - Rhonda St. Croix
- Learning and Connecting at the Royal College of Physicians and Surgeons of Canada, Canada
| | - Elaine van Melle
- Royal College of Physicians and Surgeons of Canada, Ottawa, ON, Canada
- Department of Family Medicine, Queen’s University, Kingston, ON, Canada
| |
Collapse
|
9
|
Baboolal SO, Singaram VS. Specialist training: workplace-based assessments impact on teaching, learning and feedback to support competency-based postgraduate programs. BMC MEDICAL EDUCATION 2023; 23:941. [PMID: 38082397 PMCID: PMC10712152 DOI: 10.1186/s12909-023-04922-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/13/2023] [Accepted: 11/30/2023] [Indexed: 12/18/2023]
Abstract
BACKGROUND Workplace-based assessments (WBAs) are part of a competency-based curriculum where training progression is dependent on the achievement of defined competencies in a real-world clinical environment. There is a significant literature gap on the impact of WBAs implemented in resource constrained countries and their contextual challenges. This study aimed to examine the use, impact, and educational context of WBAs in South African medical specialist training programs drawing on perspectives from both trainees and trainers to identify educational challenges and propose effective solutions. METHODS A mixed methods national electronic survey was conducted with specialist medical trainees and supervising trainers from all eight specialist training institutions in South Africa involving 16 specialities. The survey responses were voluntary and anonymous. The survey was closed after seven months when data saturation was achieved. Descriptive statistical analysis was performed using SPSS Version 27 (SPSS Inc, 2012, Chicago, IL) for the quantitative analysis. The thematic coding framework for the qualitative analysis was facilitated by NVivo Version 12 software. RESULTS There were 108 ethnically diverse supervising trainers and 248 specialist trainees' survey respondents. Across the 16 medical specialities, 45% of the respondents were using WBAs. Despite contextual resource and staff challenges, this study found that WBAs had a positive impact to Kirkpatrick level 2 in providing actionable feedback to improve competency. WBA users had a significantly higher rating for trainee supervision (p < 0.01), general quality of feedback on trainee competence (< 0.01) and the specialist training program (p = 0.03) compared to WBA non-users. They also had a higher rating for the assessment of the trainee as a professional (p < 0.01); scholar (p < 0.01); communicator (p < 0.01); collaborator (p = 0.001) and leader/manager (p < 0.001) based on the AfriMEDS competency framework. Racism, sexism and favouritism were challenges that negatively affected the training programs. CONCLUSION Overall, this study reports that the use of WBAs had a substantially favourable impact on teaching, learning, feedback and supports a competency-based approach to specialist training programs. Addressing the contextual concerns that negatively impact training; training the trainees and trainers about their relationship, roles and responsibilities; and focusing on a trainee-centred, inclusive and empowering teaching approach will help further enhance its effectiveness.
Collapse
Affiliation(s)
- Sandika O Baboolal
- School of Clinical Medicine, College of Health Sciences, University of KwaZulu Natal, Durban, South Africa.
- Ophthalmology Department, James Paget University Hospital, Great Yarmouth, UK.
| | - Veena S Singaram
- School of Clinical Medicine, College of Health Sciences, University of KwaZulu Natal, Durban, South Africa
| |
Collapse
|
10
|
Tavares W, Kinnear B, Schumacher DJ, Forte M. "Rater training" re-imagined for work-based assessment in medical education. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:1697-1709. [PMID: 37140661 DOI: 10.1007/s10459-023-10237-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Accepted: 04/30/2023] [Indexed: 05/05/2023]
Abstract
In this perspective, the authors critically examine "rater training" as it has been conceptualized and used in medical education. By "rater training," they mean the educational events intended to improve rater performance and contributions during assessment events. Historically, rater training programs have focused on modifying faculty behaviours to achieve psychometric ideals (e.g., reliability, inter-rater reliability, accuracy). The authors argue these ideals may now be poorly aligned with contemporary research informing work-based assessment, introducing a compatibility threat, with no clear direction on how to proceed. To address this issue, the authors provide a brief historical review of "rater training" and provide an analysis of the literature examining the effectiveness of rater training programs. They focus mainly on what has served to define effectiveness or improvements. They then draw on philosophical and conceptual shifts in assessment to demonstrate why the function, effectiveness aims, and structure of rater training requires reimagining. These include shifting competencies for assessors, viewing assessment as a complex cognitive task enacted in a social context, evolving views on biases, and reprioritizing which validity evidence should be most sought in medical education. The authors aim to advance the discussion on rater training by challenging implicit incompatibility issues and stimulating ways to overcome them. They propose that "rater training" (a moniker they suggest be reserved for strong psychometric aims) be augmented with "assessor readiness" programs that link to contemporary assessment science and enact the principle of compatibility between that science and ways of engaging with advances in real-world faculty-learner contexts.
Collapse
Affiliation(s)
- Walter Tavares
- Department of Health and Society, Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
| | - Benjamin Kinnear
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Daniel J Schumacher
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Milena Forte
- Department of Family and Community Medicine, Temerty Faculty of Medicine, Mount Sinai Hospital, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
11
|
Marty AP, Linsenmeyer M, George B, Young JQ, Breckwoldt J, Ten Cate O. Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154. MEDICAL TEACHER 2023; 45:1203-1213. [PMID: 36706225 DOI: 10.1080/0142159x.2023.2168527] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.
Collapse
Affiliation(s)
| | - Machelle Linsenmeyer
- West Virginia School of Osteopathic Medicine, Lewisburg, WV, United States of America
| | - Brian George
- Surgery and Learning Health Sciences, University of Michigan, Ann Arbor, Michigan, United States of America
| | - John Q Young
- Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell &, Zucker Hillside Hospital, NY, United States of America
| | - Jan Breckwoldt
- Institute of Anesthesia at the University Hospital Zurich, Switzerland
| | - Olle Ten Cate
- Utrecht Center for Research and Development of Health Professions Education at UMC Utrecht, The Netherlands
| |
Collapse
|
12
|
Nel D, McNamee L, Wright M, Alseidi AA, Cairncross L, Jonas E, Burch V. Competency Assessment of General Surgery Trainees: A Perspective From the Global South, in a CBME-Naive Context. JOURNAL OF SURGICAL EDUCATION 2023; 80:1462-1471. [PMID: 37453897 DOI: 10.1016/j.jsurg.2023.06.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Revised: 05/30/2023] [Accepted: 06/18/2023] [Indexed: 07/18/2023]
Abstract
OBJECTIVE Before proceeding with local implementation of competency-based medical education-related assessment practices designed and evaluated in the Global North, we sought to challenge the assumption that this would be perceived as both necessary and acceptable in our context where training and assessment is based on a traditional, knowledge-focused approach. The aim of this study was to determine the perspectives of general surgery trainees and consultants towards the assessment of competence, how this has been achieved previously, and how it should be performed in the future at the University of Cape Town (UCT), South Africa. DESIGN Semi-structured interviews were conducted with consultants and trainees. Interviews were transcribed and then analyzed using a Reflexive Thematic Analysis approach. SETTING AND PARTICIPANTS Ten consultants (5 senior and 5 junior) and 10 trainees (5 South African and 5 international) from the Division of General Surgery at UCT in August 2022. RESULTS Five unique themes were developed: (1) Assessment of competence is essential, (2) competence includes multiple domains of practice, (3) a surgeon must be able to operate, (4) previously used methods were inadequate to assess competence, and (5) frequent assessment with feedback is desired. The themes were considered in the context of Situated Learning Theory, particularly Communities of Practice and their role in the training for, and authentic assessment of, competence in general surgery trainees. CONCLUSIONS Participants described a need to develop and implement a new competency assessment program for general surgery training in this context, which is aligned with described competency-based medical education principles. Thoughtful integration of the formative and summative use of direct observation in the workplace, with a clear emphasis on procedural ability and the provision of high-quality feedback, may enhance the successful implementation of a strategy for competency-based assessment in general surgery training programs.
Collapse
Affiliation(s)
- D Nel
- Department of Surgery, Groote Schuur Hospital and University of Cape Town, Cape Town, South Africa.
| | - L McNamee
- Center for Higher Education Development, University of Cape Town, Cape Town, South Africa
| | - M Wright
- Department of Radiodiagnosis, Tygerberg Hospital, University of Stellenbosch, Cape Town, South Africa
| | - A A Alseidi
- Department of Surgery, University of California, San Francisco, California
| | - L Cairncross
- Department of Surgery, Groote Schuur Hospital and University of Cape Town, Cape Town, South Africa
| | - E Jonas
- Department of Surgery, Groote Schuur Hospital and University of Cape Town, Cape Town, South Africa
| | - V Burch
- Department of Medicine, Groote Schuur Hospital and University of Cape Town, and the Colleges of Medicine of South Africa, South Africa
| |
Collapse
|
13
|
Dulay M, Bowen JL, Weppner WG, Eastburn A, Poppe AP, Spanos P, Wojtaszek D, Printz D, Kaminetzky CP. Interprofessional population health advocacy: Developing and implementing a panel management curriculum in five Veterans Administration primary care practices. J Interprof Care 2023; 37:S75-S85. [PMID: 29746221 DOI: 10.1080/13561820.2018.1469476] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2017] [Revised: 12/04/2017] [Accepted: 04/23/2018] [Indexed: 10/16/2022]
Abstract
Health care systems expect primary care clinicians to manage panels of patients and improve population health, yet few have been trained to do so. An interprofessional panel management (PM) curriculum is one possible strategy to address this training gap and supply future primary care practices with clinicians and teams prepared to work together to improve the health of individual patients and populations. This paper describes a Veterans Administration (VA) sponsored multi-site interprofessional PM curriculum development effort. Five VA Centers of Excellence in Primary Care Education collaborated to identify a common set of interprofessionally relevant desired learning outcomes (DLOs) for the PM and to develop assessment instruments for monitoring trainees' PM learning. Authors cataloged teaching and learning activities across sites. Results from pilot testing were systematically discussed leading to iterative revisions of curricular elements. Authors completed a retrospective self-assessment of curriculum implementation for the academic year 2015-16 using a 5-point scale: contemplation (score = 0), pilot (1), action (2), maintenance (3), and embedded (4). Implementation scores were analyzed using descriptive statistics. DLOs were organized into five categories (individual patients, populations, guidelines/measures, teamwork, and improvement) along with a developmental continuum and mapped to program competencies. Instruction and implementation varied across sites based on resources and priorities. Between 2015 and 2016, 159 trainees (internal medicine residents, nurse practitioner students and residents, pharmacy residents, and psychology post-doctoral fellows) participated in the PM curriculum. Curriculum implementation scores for guidelines/measures and improvement DLOs were similar for all trainees; scores for individual patients, populations, and teamwork DLOs were more advanced for nurse practitioner and physician trainees. In conclusion, collaboratively identified DLOs for PM guided development of assessment instruments and instructional approaches for panel management activities in interprofessional teams. This PM curriculum and associated tools provide resources for educators in other settings.
Collapse
Affiliation(s)
- Maya Dulay
- Center of Excellence in Primary Care Education, San Francisco, CA, USA
- Department of Medicine, University of California, San Francisco, CA, USA
| | - Judith L Bowen
- Schools of Medicine and Nursing, Oregon Health & Science University, Portland, WA, USA
- Office of Academic Affiliations, Department of Veterans Affairs (VA), Washington DC, USA
| | - William G Weppner
- Center of Excellence in Primary Care Education, Boise VA Medical Center, Boise, ID, USA
- Department of Medicine, University of Washington, Seattle, WA, USA
| | - Abigail Eastburn
- Center of Excellence in Primary Care Education, San Francisco, CA, USA
- Department of Medicine, University of California, San Francisco, CA, USA
| | - Anne P Poppe
- Center of Excellence in Primary Care Education, Seattle, WA, USA
- School of Nursing, University of Washington, Seattle, WA, USA
| | - Pete Spanos
- Center of Excellence in Primary Care Education, Louis Stokes Cleveland VAMC, Cleveland, OH, USA
| | - Danielle Wojtaszek
- West Haven Center of Excellence in Primary Care Education, VA Connecticut Health Care System, West Haven, CT, USA
| | - Destiny Printz
- Center of Excellence in Primary Care Education, VA Connecticut Healthcare System, West Haven, CT, USA
- Department of Psychiatry, Yale School of Medicine, West Haven, CT, USA
| | - Catherine P Kaminetzky
- VA Puget Sound Health Care System, Seattle, WA, USA
- School of Medicine, University of Washington, Seattle, WA
| |
Collapse
|
14
|
Baboolal SO, Singaram VS. The Use, Effectiveness, and Impact of Workplace-Based Assessments on Teaching, Supervision and Feedback Across Surgical Specialties. JOURNAL OF SURGICAL EDUCATION 2023; 80:1158-1171. [PMID: 37407351 DOI: 10.1016/j.jsurg.2023.05.012] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Revised: 04/17/2023] [Accepted: 05/15/2023] [Indexed: 07/07/2023]
Abstract
OBJECTIVE To investigate the use and effectiveness of Workplace-based assessments (WBAs) and their impact on training, feedback, and perioperative teaching in surgical training programs. DESIGN A mixed methods cross-sectional, national electronic survey was conducted with surgical trainees and consultant trainers. SETTINGS The trainees and supervising faculty were from all 8 major surgical training universities across 11 surgical disciplines in South Africa. PARTICIPANTS A total of 108 surgical trainees and 41 supervising consultant trainers from 11 surgical disciplines across 8 surgical training universities responded to the survey. RESULTS The most significant educational gap identified by both the surgical trainees and trainers across all surgical disciplines was inadequate perioperative feedback. A third of the respondents were currently using workplace-based assessments. The WBA users (both trainees and trainers) had a higher rating for the general quality of surgical feedback than WBA nonusers (p = 0.02). WBA users also had a higher rating for the general quality of feedback given to trainees on their skills and competence (p = 0.04) and a higher rating for trainee supervision (p = 0.01) and the specialist training program overall (p = 0.01). The WBA users also had a higher rating for the assessment of competencies such as the trainee as an effective communicator (p < 0.01) and collaborator (p = 0.04). CONCLUSION This study found that the use of WBAs enhances the quality and effectiveness of feedback in surgical training programs. We also found that the use of WBAs enhance perioperative teaching and learning and improves the assessment of relational competencies. This was also associated with high ratings for the quality of trainee supervision. Faculty and trainee development, strengthening the trainee-trainer relationship, and integrating iterative stakeholder feedback could help realize the full potential of WBAs to augment surgical training across disciplines.
Collapse
Affiliation(s)
- Sandika O Baboolal
- School of Clinical Medicine, College of Health Sciences, University of KwaZulu Natal, Durban, South Africa.
| | - Veena S Singaram
- School of Clinical Medicine, College of Health Sciences, University of KwaZulu Natal, Durban, South Africa
| |
Collapse
|
15
|
Mughal Z, Patel S, Gupta KK, Metcalfe C, Beech T, Jennings C. Evaluating the perceptions of workplace-based assessments in surgical training: a systematic review. Ann R Coll Surg Engl 2023; 105:507-512. [PMID: 36374289 PMCID: PMC10313445 DOI: 10.1308/rcsann.2022.0113] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/01/2022] [Indexed: 11/24/2023] Open
Abstract
INTRODUCTION Workplace-based assessments (WBAs) are intended to maximise learning opportunities in surgical training. There is speculation as to whether mandatory assessments in this form contribute to a tick-box culture. The objective of this review was to investigate surgical trainees' attitudes towards WBAs. METHODS This systematic review of qualitative studies was performed in accordance with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement. The literature was searched on the Medline®, Embase™, PubMed and Web of Science™ databases on 22 March 2022. RESULTS Sixteen studies were included in the review, mostly carried out on users of the Intercollegiate Surgical Curriculum Programme portfolio in the UK. Trainees felt that WBAs were educationally useful, providing opportunity for feedback, but this was overshadowed by a pressure to reach a set annual quota for WBAs and achieve high scores. Other themes included inaccurate recording of WBAs, the role of WBAs as formative or summative assessments, engagement and accessibility of trainers, and lack of time to complete WBAs. CONCLUSIONS Negative perceptions about WBAs were widespread among surgical trainees despite a recognition of their capacity to facilitate learning. This review supports the recent removal of the annual quota for WBAs in UK surgical training programmes.
Collapse
Affiliation(s)
- Z Mughal
- University Hospitals Birmingham NHS Foundation Trust, UK
| | - S Patel
- Shrewsbury and Telford Hospital NHS Foundation Trust, UK
| | - K Kumar Gupta
- University Hospitals Birmingham NHS Foundation Trust, UK
| | - C Metcalfe
- University Hospitals Birmingham NHS Foundation Trust, UK
| | - T Beech
- University Hospitals Birmingham NHS Foundation Trust, UK
| | - C Jennings
- University Hospitals Birmingham NHS Foundation Trust, UK
| |
Collapse
|
16
|
Gauthier S, Braund H, Dalgarno N, Taylor D. Assessment-Seeking Strategies: Navigating the Decision to Initiate Workplace-Based Assessment. TEACHING AND LEARNING IN MEDICINE 2023:1-10. [PMID: 37384570 DOI: 10.1080/10401334.2023.2229803] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/29/2022] [Revised: 04/13/2023] [Accepted: 06/01/2023] [Indexed: 07/01/2023]
Abstract
Phenomenon: Competency-based medical education (CBME) relies on workplace-based assessment (WBA) to generate formative feedback (assessment for learning-AfL) and make inferences about competence (assessment of learning-AoL). When approaches to CBME rely on residents to initiate WBA, learners experience tension between seeking WBA for learning and for establishing competence. How learners resolve this tension may lead to unintended consequences for both AfL and AoL. We sought to explore the factors that impact both decisions to seek and not to seek WBA and use the findings to build a model of assessment-seeking strategy used by residents. In building this model we consider how the link between WBA and promotion or progression within a program impacts an individual's assessment-seeking strategy. Approach: We conducted 20 semi-structured interviews with internal medicine residents at Queen's University about the factors that influence their decision to seek or avoid WBA. Using grounded theory methodology, we applied a constant comparative analysis to collect data iteratively and identify themes. A conceptual model was developed to describe the interaction of factors impacting the decision to seek and initiate WBA. Findings: Participants identified two main motivations when deciding to seek assessments: the need to fulfill program requirements and the desire to receive feedback for learning. Analysis suggested that these motivations are often at odds with each other. Participants also described several moderating factors that impact the decision to initiate assessments, irrespective of the primary underlying motivation. These included resident performance, assessor factors, training program expectations, and clinical context. A conceptual framework was developed to describe the factors that lead to strategic assessment-seeking behaviors. Insights: Faced with the dual purpose of WBA in CBME, resident behavior in initiating assessment is guided by specific assessment-seeking strategies. Strategies reflect individual underlying motivations, influenced by four moderating factors. These findings have broad implications for programmatic assessment in a CBME context including validity considerations for assessment data used in summative decision-making including readiness for unsupervised practice.
Collapse
Affiliation(s)
- Stephen Gauthier
- Department of Medicine, Queen's University, Kingston, Ontario, Canada
| | - Heather Braund
- Office of Professional Development and Educational Scholarship, Faculty of Health Sciences, Queen's University, Kingston, Ontario, Canada
| | - Nancy Dalgarno
- Office of Professional Development and Educational Scholarship, Faculty of Health Sciences, Queen's University, Kingston, Ontario, Canada
| | - David Taylor
- Department of Medicine, Queen's University, Kingston, Ontario, Canada
| |
Collapse
|
17
|
Blanchette P, Poitras ME, St-Onge C. Assessing trainee's performance using reported observations: Perceptions of nurse meta-assessors. NURSE EDUCATION TODAY 2023; 126:105836. [PMID: 37167832 DOI: 10.1016/j.nedt.2023.105836] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 04/12/2023] [Accepted: 04/30/2023] [Indexed: 05/13/2023]
Abstract
BACKGROUND Educational and health care organizations who prepare meta-assessors to fulfill their role in the assessment of trainees' performance based on reported observations have little literature to rely on. While the assessment of trainees' performance based on reported observations has been operationalized, we have yet to understand the elements that can affect its quality fully. Closing this gap in the literature will provide valuable insight that could inform the implementation and quality monitoring of the assessment of trainees' performance based on reported observations. OBJECTIVES The purpose of this study was to explore the elements to consider in the assessment of trainees' performance based on reported observations from the perspectives of meta-assessors. METHODS Design, Settings, Participants, data collection and analysis. The authors adopted Sandelowski's qualitative descriptive approach to interview nurse meta-assessors from two nursing programs. A semi-structured interview guide was used to document the elements to consider in the assessment of nursing trainees' performance based on reported observations, and a survey was used to collect sociodemographic data. The authors conducted a thematic analysis of the interview transcripts. RESULTS Thirteen meta-assessors participated in the study. Three core themes were identified: (1) meta-assessors' appropriation of their perceived assessment roles and activities, (2) team climate of information sharing, and (3) challenges associated with the assessment of trainees' performance based on reported observations. Each theme is comprised of several sub themes. CONCLUSIONS To optimize the quality of the assessment of the trainee's performance based on reported observations and ratings, HPE programs might consider how to clarify better the meta-assessor's roles and activities, as well as how interventions could be created to promote a climate of information sharing and to address the challenges identified. This work will guide educational and health care organizations for better preparation and support for meta-assessors and preceptors.
Collapse
Affiliation(s)
| | - Marie-Eve Poitras
- Department of Family Medicine and Emergency Medicine, University of Sherbrooke, Sherbrooke, Quebec, Canada.
| | - Christina St-Onge
- Department of Medicine, University of Sherbrooke, Sherbrooke, Quebec, Canada
| |
Collapse
|
18
|
Nel D, Jonas E, Burch V, Nel A, Cairncross L, Alseidi A, George B, Ten Cate O. Entrustable professional activities in postgraduate general surgery training: a scoping review protocol. BMJ Open 2023; 13:e067322. [PMID: 36921948 PMCID: PMC10030674 DOI: 10.1136/bmjopen-2022-067322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/17/2023] Open
Abstract
INTRODUCTION The competency-based medical education (CBME) movement continues to gain momentum in postgraduate physician training, resulting in increasing interest among surgical training programmes on how to implement it effectively. Entrustable professional activities (EPAs) were introduced to connect competencies (characteristics/abilities of learners) and the professional activities to be entrusted to them on qualification/graduation. Although reviews related to the field of general surgery have been published on specific aspects of CBME, for example, workplace-based assessment, there is a paucity of published guidance available for surgeon-educators to plan and implement CBME using an EPA framework. This scoping review aims to provide an overview of the current use of EPAs in general surgery. METHODS AND ANALYSIS This scoping review will be based on the original five-stage approach described by Arksey and O'Malley with the addition of protocol-specific items from the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols extension for scoping reviews. The primary research question focuses on describing how EPAs have been used in general surgery and where potential gaps remain that warrant further study. The review will include peer-reviewed journal publications and an extensive review of grey literature sources. The following databases will be searched for published literature from 1 January 2005 to 31 December 2022: PubMed; CINAHL, Africa-Wide Information, PsycInfo and ERIC (via EBSCOhost); Scopus; and SciELO (via Web of Science). Studies that describe the use of EPAs for curriculum design, teaching and/or assessment of competence in postgraduate general surgery training will be eligible for inclusion. ETHICS AND DISSEMINATION The institutional ethics board of the University of Cape Town has granted a waiver of formal approval requirement. The dissemination strategy includes publication of results in peer-reviewed journals, presentation at international conferences and presentation to relevant stakeholders as deemed appropriate.
Collapse
Affiliation(s)
- Daniel Nel
- Department of Surgery, University of Cape Town, Cape Town, South Africa
| | - Eduard Jonas
- Department of Surgery, University of Cape Town, Cape Town, South Africa
| | - Vanessa Burch
- Department of Medicine, Groote Schuur Hospital and University of Cape Town, and the Colleges of Medicine of South Africa, Cape Town, South Africa
| | - Amy Nel
- Department of Public Health, University of Cape Town, Cape Town, South Africa
| | - Lydia Cairncross
- Department of Surgery, University of Cape Town, Cape Town, South Africa
| | - Adnan Alseidi
- Department of Surgery, University of California San Francisco, San Francisco, California, USA
| | - Brian George
- Department of Surgery, University of Michigan, Ann Arbor, Michigan, USA
| | - Olle Ten Cate
- Center for Research and Development of Education, UMC Utrecht, Utrecht, Netherlands
| |
Collapse
|
19
|
Ossenberg C, Mitchell M, Burmeister E, Henderson A. Measuring changes in nursing students' workplace performance following feedback encounters: A quasi-experimental study. NURSE EDUCATION TODAY 2023; 121:105683. [PMID: 36512888 DOI: 10.1016/j.nedt.2022.105683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 11/21/2022] [Accepted: 12/02/2022] [Indexed: 06/17/2023]
Abstract
BACKGROUND Work-integrated learning and performance is intimately linked with assessment and feedback; however, empirical evidence concerning the impact of feedback on student performance in workplace settings is limited. A novel practice-based feedback intervention-the REMARK (nuRse fEedback iMplementAtion frameworRK) program-was developed and implemented. Sustainable feedback processes that involve both learners and learning partners can ultimately maximise learning outcomes, optimise self-regulation, and impact performance. OBJECTIVES To determine the impact of implementing an educational intervention based on best practice principles of feedback on undergraduate nursing student performance during clinical placements in acute healthcare settings. METHODS A non-equivalent, quasi-experimental design was used. A pre- and post-assessment strategy for both control and intervention groups was employed. The REMARK program was based on known attributes of effective feedback critiqued in the literature that foster interactions between a learner and learning partner and, hence, support dialogic feedback. RESULTS Clinical assessment data of 214 final-year nursing students collected at three time points were analysed. Using multivariate modelling, the results indicated that students participating in the REMARK programme (intervention group) had statistically higher performance scores than students in the control group when time and placement setting were controlled. CONCLUSIONS This research provides evidence that engagement with an intervention based on attributes of effective feedback improves nursing students' workplace performance during clinical placement. To achieve the greatest advantage, incorporating strategies that encourage students to adopt an active role in feedback conversations that relate to the student's learning goals are recommended.
Collapse
Affiliation(s)
- Christine Ossenberg
- Nursing Practice Development Unit, Princess Alexandra Hospital, Ipswich Road, Woolloongabba, Qld 4102, Australia.
| | - Marion Mitchell
- Griffith University, School of Nursing and Midwifery, Nathan, Qld 4111, Australia; Menzies Health Institute Queensland, Griffith University, Queensland, Australia
| | - Elizabeth Burmeister
- Nursing Practice Development Unit, Princess Alexandra Hospital, Ipswich Road, Woolloongabba, Qld 4102, Australia
| | - Amanda Henderson
- Nursing Practice Development Unit, Princess Alexandra Hospital, Ipswich Road, Woolloongabba, Qld 4102, Australia; Central Queensland University, 160 Ann Street, Brisbane City, Queensland 4000, Australia
| |
Collapse
|
20
|
Kogan JR, Dine CJ, Conforti LN, Holmboe ES. Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:237-247. [PMID: 35857396 DOI: 10.1097/acm.0000000000004819] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. METHOD This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance. RESULTS Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. CONCLUSIONS Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.
Collapse
Affiliation(s)
- Jennifer R Kogan
- J.R. Kogan is associate dean, Student Success and Professional Development, and professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0001-8426-9506
| | - C Jessica Dine
- C.J. Dine is associate dean, Evaluation and Assessment, and associate professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0001-5894-0861
| | - Lisa N Conforti
- L.N. Conforti is research associate for milestones evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-7317-6221
| | - Eric S Holmboe
- E.S. Holmboe is chief, research, milestones development and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois; ORCID: https://orcid.org/0000-0003-0108-6021
| |
Collapse
|
21
|
Phinney LB, Fluet A, O'Brien BC, Seligman L, Hauer KE. Beyond Checking Boxes: Exploring Tensions With Use of a Workplace-Based Assessment Tool for Formative Assessment in Clerkships. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1511-1520. [PMID: 35703235 DOI: 10.1097/acm.0000000000004774] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PURPOSE To understand the role of a workplace-based assessment (WBA) tool in facilitating feedback for medical students, this study explored changes and tensions in a clerkship feedback activity system through the lens of cultural historical activity theory (CHAT) over 2 years of tool implementation. METHOD This qualitative study uses CHAT to explore WBA use in core clerkships by identifying feedback activity system elements (e.g., community, tools, rules, objects) and tensions among these elements. University of California, San Francisco core clerkship students were invited to participate in semistructured interviews eliciting experience with a WBA tool intended to enhance direct observation and feedback in year 1 (2019) and year 2 (2020) of implementation. In year 1, the WBA tool required supervisor completion in the school's evaluation system on a computer. In year 2, both students and supervisors had WBA completion abilities and could access the form via a smartphone separate from the school's evaluation system. RESULTS Thirty-five students participated in interviews. The authors identified tensions that shifted with time and tool iterations. Year 1 students described tensions related to cumbersome tool design, fear of burdening supervisors, confusion over WBA purpose, WBA as checking boxes, and WBA usefulness depending on clerkship context and culture. Students perceived dissatisfaction with the year 1 tool version among peers and supervisors. The year 2 mobile-based tool and student completion capabilities helped to reduce many of the tensions noted in year 1. Students expressed wider WBA acceptance among peers and supervisors in year 2 and reported understanding WBA to be for low-stakes feedback, thereby supporting formative assessment for learning. CONCLUSIONS Using CHAT to explore changes in a feedback activity system with WBA tool iterations revealed elements important to WBA implementation, including designing technology for tool efficiency and affording students autonomy to document feedback with WBAs.
Collapse
Affiliation(s)
- Lauren B Phinney
- L.B. Phinney is a first-year internal medicine resident, Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California
| | - Angelina Fluet
- A. Fluet is a fourth-year medical student, University of California, San Francisco School of Medicine, San Francisco, California
| | - Bridget C O'Brien
- B.C. O'Brien is professor of medicine and education scientist, Department of Medicine and Center for Faculty Educators, University of California, San Francisco School of Medicine, San Francisco, California
| | - Lee Seligman
- L. Seligman is a second-year internal medicine resident, Department of Medicine, New York-Presbyterian Hospital, Columbia University Irving Medical Center, New York, New York
| | - Karen E Hauer
- K.E. Hauer is associate dean for competency assessment and professional standards and professor, Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California
| |
Collapse
|
22
|
Lam AC, Tang B, Lalwani A, Verma AA, Wong BM, Razak F, Ginsburg S. Methodology paper for the General Medicine Inpatient Initiative Medical Education Database (GEMINI MedED): a retrospective cohort study of internal medicine resident case-mix, clinical care and patient outcomes. BMJ Open 2022; 12:e062264. [PMID: 36153026 PMCID: PMC9511606 DOI: 10.1136/bmjopen-2022-062264] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/03/2022] Open
Abstract
INTRODUCTION Unwarranted variation in patient care among physicians is associated with negative patient outcomes and increased healthcare costs. Care variation likely also exists for resident physicians. Despite the global movement towards outcomes-based and competency-based medical education, current assessment strategies in residency do not routinely incorporate clinical outcomes. The widespread use of electronic health records (EHRs) may enable the implementation of in-training assessments that incorporate clinical care and patient outcomes. METHODS AND ANALYSIS The General Medicine Inpatient Initiative Medical Education Database (GEMINI MedED) is a retrospective cohort study of senior residents (postgraduate year 2/3) enrolled in the University of Toronto Internal Medicine (IM) programme between 1 April 2010 and 31 December 2020. This study focuses on senior IM residents and patients they admit overnight to four academic hospitals. Senior IM residents are responsible for overseeing all overnight admissions; thus, care processes and outcomes for these clinical encounters can be at least partially attributed to the care they provide. Call schedules from each hospital, which list the date, location and senior resident on-call, will be used to link senior residents to EHR data of patients admitted during their on-call shifts. Patient data will be derived from the GEMINI database, which contains administrative (eg, demographic and disposition) and clinical data (eg, laboratory and radiological investigation results) for patients admitted to IM at the four academic hospitals. Overall, this study will examine three domains of resident practice: (1) case-mix variation across residents, hospitals and academic year, (2) resident-sensitive quality measures (EHR-derived metrics that are partially attributable to resident care) and (3) variations in patient outcomes across residents and factors that contribute to such variation. ETHICS AND DISSEMINATION GEMINI MedED was approved by the University of Toronto Ethics Board (RIS#39339). Results from this study will be presented in academic conferences and peer-reviewed journals.
Collapse
Affiliation(s)
- Andrew Cl Lam
- Department of Medicine, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
| | - Brandon Tang
- Department of Medicine, Division of General Internal Medicine, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
| | - Anushka Lalwani
- Li Ka Shing Knowledge Institute, Unity Health Toronto, Toronto, Ontario, Canada
| | - Amol A Verma
- Department of Medicine, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
- Li Ka Shing Knowledge Institute, Unity Health Toronto, Toronto, Ontario, Canada
- Division of General Internal Medicine, Unity Health Toronto, Toronto, Ontario, Canada
| | - Brian M Wong
- Department of Medicine, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
- Division of General Internal Medicine, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
| | - Fahad Razak
- Department of Medicine, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
- Li Ka Shing Knowledge Institute, Unity Health Toronto, Toronto, Ontario, Canada
- Division of General Internal Medicine, Unity Health Toronto, Toronto, Ontario, Canada
| | - Shiphra Ginsburg
- Department of Medicine, Division of Respirology, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
- Division of Respirology, Sinai Health System, Toronto, Ontario, Canada
| |
Collapse
|
23
|
Wilkinson TJ. Four ways to get a grip on making robust decisions from workplace-based assessments. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:43-46. [PMID: 35875436 PMCID: PMC9297242 DOI: 10.36834/cmej.73361] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Synthesising the results of workplace-based assessments to inform robust decisions is seen as both important and difficult. Concerns about failing to fail the trainee not ready to proceed has drawn disproportionate attention to assessors. This paper proposes a model for a more systems-based view so that the value of the assessor's judgement is incorporated while preserving the value and robustness of collective decision-making. Our experience has shown it can facilitate robust decisions on some of the more difficult areas, such as professionalism.
Collapse
|
24
|
Yilmaz Y, Jurado Nunez A, Ariaeinejad A, Lee M, Sherbino J, Chan TM. Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education. JMIR MEDICAL EDUCATION 2022; 8:e30537. [PMID: 35622398 PMCID: PMC9187970 DOI: 10.2196/30537] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/19/2021] [Revised: 12/05/2021] [Accepted: 04/30/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. OBJECTIVE This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings. METHODS NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported. RESULTS The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%). CONCLUSIONS The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.
Collapse
Affiliation(s)
- Yusuf Yilmaz
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Department of Medical Education, Ege University, Izmir, Turkey
- Program for Faculty Development, Office of Continuing Professional Development, McMaster University, Hamilton, ON, Canada
- Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Alma Jurado Nunez
- Department of Medicine and Masters in eHealth Program, McMaster University, Hamilton, ON, Canada
| | - Ali Ariaeinejad
- Department of Medicine and Masters in eHealth Program, McMaster University, Hamilton, ON, Canada
| | - Mark Lee
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Jonathan Sherbino
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Division of Emergency Medicine, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Division of Education and Innovation, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Teresa M Chan
- McMaster Education Research, Innovation, and Theory Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Program for Faculty Development, Office of Continuing Professional Development, McMaster University, Hamilton, ON, Canada
- Division of Emergency Medicine, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Division of Education and Innovation, Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| |
Collapse
|
25
|
Castanelli DJ, Weller JM, Molloy E, Bearman M. How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:704-710. [PMID: 34732657 PMCID: PMC9028297 DOI: 10.1097/acm.0000000000004501] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/30/2023]
Abstract
PURPOSE In competency-based medical education, workplace-based assessment provides trainees with an opportunity for guidance and supervisors the opportunity to judge the trainees' clinical practice. Learning from assessment is enhanced when trainees reveal their thinking and are open to critique, which requires trust in the assessor. If supervisors knew more about how trainees come to trust them in workplace-based assessment, they could better engender trainee trust and improve trainees' learning experience. METHOD From August 2018 to September 2019, semistructured interviews were conducted with 17 postgraduate anesthesia trainees across Australia and New Zealand. The transcripts were analyzed using constructivist grounded theory methods sensitized by a sociocultural view of learning informed by Wenger's communities of practice theory. RESULTS Participants described a continuum from a necessary initial trust to an experience-informed dynamic trust. Trainees assumed initial trust in supervisors based on accreditation, reputation, and a perceived obligation of trustworthiness inherent in the supervisor's role. With experience and time, trainees' trust evolved based on supervisor actions. Deeper levels of trainee trust arose in response to perceived supervisor investment and allowed trainees to devote more emotional and cognitive resources to patient care and learning rather than impression management. Across the continuum from initial trust to experience-informed trust, trainees made rapid trust judgments that were not preceded by conscious deliberation; instead, they represented a learned "feel for the game." CONCLUSIONS While other factors are involved, our results indicate that the trainee behavior observed in workplace-based assessment is a product of supervisor invitation. Supervisor trustworthiness and investment in trainee development invite trainees to work and present in authentic ways in workplace-based assessment. This authentic engagement, where learners "show themselves" to supervisors and take risks, creates assessment for learning.
Collapse
Affiliation(s)
- Damian J. Castanelli
- D.J. Castanelli is senior lecturer, School of Clinical Sciences at Monash Health, Monash University, consultant anesthetist, Department of Anaesthesia and Perioperative Medicine, Monash Health, Clayton, Victoria, Australia, and PhD candidate, Centre for Research in Assessment and Digital Learning, Deakin University, Geelong, Victoria, Australia; ORCID: https://orcid.org/0000-0002-5377-809X
| | - Jennifer M. Weller
- J.M. Weller is professor and head, Centre for Medical and Health Sciences Education, University of Auckland, and specialist anesthetist, Department of Anaesthesia, Auckland City Hospital, Auckland, New Zealand; ORCID: https://orcid.org/0000-0003-3029-1390
| | - Elizabeth Molloy
- E. Molloy is professor of work-integrated learning, Department of Medical Education, Melbourne Medical School, University of Melbourne, Melbourne, Victoria, Australia; ORCID: https://orcid.org/0000-0001-9457-9348
| | - Margaret Bearman
- M. Bearman is research professor, Centre for Research in Assessment and Digital Learning, Deakin University, Geelong, Victoria, Australia; ORCID: https://orcid.org/0000-0002-6862-9871
| |
Collapse
|
26
|
de Jonge LPJWM, Minkels FNE, Govaerts MJB, Muris JWM, Kramer AWM, van der Vleuten CPM, Timmerman AA. Supervisory dyads' communication and alignment regarding the use of workplace-based observations: a qualitative study in general practice residency. BMC MEDICAL EDUCATION 2022; 22:330. [PMID: 35484573 PMCID: PMC9052511 DOI: 10.1186/s12909-022-03395-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2021] [Accepted: 04/21/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND In medical residency, performance observations are considered an important strategy to monitor competence development, provide feedback and warrant patient safety. The aim of this study was to gain insight into whether and how supervisor-resident dyads build a working repertoire regarding the use of observations, and how they discuss and align goals and approaches to observation in particular. METHODS We used a qualitative, social constructivist approach to explore if and how supervisory dyads work towards alignment of goals and preferred approaches to performance observations. We conducted semi-structured interviews with supervisor-resident dyads, performing a template analysis of the data thus obtained. RESULTS The supervisory dyads did not frequently communicate about the use of observations, except at the start of training and unless they were triggered by internal or external factors. Their working repertoire regarding the use of observations seemed to be primarily driven by patient safety goals and institutional assessment requirements rather than by providing developmental feedback. Although intended as formative, the institutional test was perceived as summative by supervisors and residents, and led to teaching to the test rather than educating for purposes of competence development. CONCLUSIONS To unlock the full educational potential of performance observations, and to foster the development of an educational alliance, it is essential that supervisory dyads and the training institute communicate clearly about these observations and the role of assessment practices of- and for learning, in order to align their goals and respective approaches.
Collapse
Affiliation(s)
- Laury P J W M de Jonge
- Department of General Practice, Maastricht University, P.O. Box 616, 6200, MD, Maastricht, The Netherlands.
| | - Floor N E Minkels
- Department of General Practice, Maastricht University, P.O. Box 616, 6200, MD, Maastricht, The Netherlands
| | - Marjan J B Govaerts
- Department of Educational Research and Development, Maastricht University, Maastricht, The Netherlands
| | - Jean W M Muris
- Department of General Practice, Maastricht University, P.O. Box 616, 6200, MD, Maastricht, The Netherlands
| | - Anneke W M Kramer
- Department of Family Medicine, Leiden University, Leiden, The Netherlands
| | - Cees P M van der Vleuten
- Department of Educational Research and Development, Maastricht University, Maastricht, The Netherlands
| | - Angelique A Timmerman
- Department of General Practice, Maastricht University, P.O. Box 616, 6200, MD, Maastricht, The Netherlands
| |
Collapse
|
27
|
Luckoski J, Thelen A, Russell D, George B, Krumm A. Feedback-Seeking Behavior and Practice Readiness for General Surgery. JOURNAL OF SURGICAL EDUCATION 2022; 79:295-301. [PMID: 34838471 DOI: 10.1016/j.jsurg.2021.10.003] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2021] [Revised: 06/11/2021] [Accepted: 10/02/2021] [Indexed: 06/13/2023]
Abstract
OBJECTIVE To analyze the relationship between feedback-seeking behavior, operationalized as the number of trainee-requested evaluations, with ratings of surgical trainees' operative autonomy and performance. DESIGN We analyzed operative assessment data using the System for Improving and Measuring Procedural Learning's smartphone-based assessment app called Society for Improving Medical Professional Learning (SIMPL) OR. Using cross-classified mixed effects models, we analyzed the association between trainee-requested SIMPL OR app evaluations and both trainee performance and autonomy ratings. Models included covariates for requested evaluations, PGY-year, month of the academic year, and patient-related case complexity. Random effects for program, procedure, rater, and trainee were also included to account for correlations among evaluations. Only ratings for procedures deemed Core to general surgery were included. SETTING Operative assessment data using the SIMPL OR app requested by categorical U.S. general surgery residents between September 2015 to April 2021. PARTICIPANTS A total of 61 general surgery residency programs, encompassing 2190 categorical general surgery residents. RESULTS A total of 58,104 SIMPL app operative assessments were analyzed. Autonomy scores were weakly but positively associated with number of trainee-requested evaluations (B = 0.002, p < 0.001). Trainee-requested evaluations were also statistically associated with operative performance scores ( B = 0.002, p < 0.001). CONCLUSIONS The propensity of a resident to seek feedback using the SIMPL app was weakly associated with higher operative autonomy ratings and higher operative performance ratings. While regular feedback is important for monitoring performance over time, more direct approaches related to the quality of feedback that trainees receive may be needed to better assess the relationships between feedback-seeking behavior and operative autonomy as well as performance.
Collapse
Affiliation(s)
- John Luckoski
- Department of Surgery, University of Michigan, Ann Arbor, Michigan.
| | - Angela Thelen
- Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Dylan Russell
- Department of Surgery, Tripler Army Medical Center, Honolulu, Hawaii
| | - Brian George
- Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Andrew Krumm
- Department of Learning Health Sciences, University of Michigan, Ann Arbor, Michigan
| |
Collapse
|
28
|
Castanelli DJ, Weller JM, Molloy E, Bearman M. Trust, power and learning in workplace-based assessment: The trainee perspective. MEDICAL EDUCATION 2022; 56:280-291. [PMID: 34433230 PMCID: PMC9292503 DOI: 10.1111/medu.14631] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/01/2021] [Revised: 08/09/2021] [Accepted: 08/17/2021] [Indexed: 05/22/2023]
Abstract
For trainees to participate meaningfully in workplace-based assessment (WBA), they must have trust in their assessor. However, the trainee's dependent position complicates such trust. Understanding how power and trust influence WBAs may help us make them more effective learning opportunities. We conducted semi-structured interviews with 17 postgraduate anaesthesia trainees across Australia and New Zealand. Sensitised by notions of power, we used constructivist grounded theory methodology to examine trainees' experiences with trusting their supervisors in WBAs. In our trainee accounts, we found that supervisors held significant power to mediate access to learning opportunities and influence trainee progress in training. All episodes where supervisors could observe trainees, from simply working together to formal WBAs, were seen to generate assessment information with potential consequences. In response, trainees actively acquiesced to a deferential role, which helped them access desirable expertise and minimise the risk of reputational harm. Trainees granted trust based on how they anticipated a supervisor would use the power inherent in their role. Trainees learned to ration exposure of their authentic practice to supervisors in proportion to their trust in them. Trainees were more trusting and open to learning when supervisors used their power for the trainee's benefit and avoided WBAs with supervisors they perceived as less trustworthy. If assessment for learning is to flourish, then the trainee-supervisor power dynamic must evolve. Enhancing supervisor behaviour through reflection and professional development to better reward trainee trust would invite more trainee participation in assessment for learning. Modifying the assessment system design to nudge the power balance towards the trainee may also help. Modifications could include designated formative and summative assessments or empowering trainees to select which assessments count towards progress decisions. Attending to power and trust in WBA may stimulate progress towards the previously aspirational goal of assessment for learning in the workplace.
Collapse
Affiliation(s)
- Damian J. Castanelli
- School of Clinical Sciences at Monash HealthMonash UniversityClaytonVictoriaAustralia
- Department of Anaesthesia and Perioperative MedicineMonash HealthClaytonVictoriaAustralia
- Centre for Research and Assessment in Digital Learning (CRADLE)Deakin UniversityGeelongVictoriaAustralia
| | - Jennifer M. Weller
- Centre for Medical and Health Sciences Education, School of MedicineUniversity of AucklandAucklandNew Zealand
- Department of AnaesthesiaAucklandAucklandNew Zealand
| | - Elizabeth Molloy
- Department of Medical Education, Melbourne Medical SchoolUniversity of MelbourneMelbourneVictoriaAustralia
| | - Margaret Bearman
- Centre for Research and Assessment in Digital Learning (CRADLE)Deakin UniversityGeelongVictoriaAustralia
| |
Collapse
|
29
|
Liang Y, Noble LM. Chinese doctors' views on workplace-based assessment: trainee and supervisor perspectives of the mini-CEX. MEDICAL EDUCATION ONLINE 2021; 26:1869393. [PMID: 33380291 PMCID: PMC7782920 DOI: 10.1080/10872981.2020.1869393] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/03/2020] [Revised: 12/15/2020] [Accepted: 12/22/2020] [Indexed: 05/28/2023]
Abstract
Purpose: This study investigated whether the mini-clinical evaluation exercise (mini-CEX) has been successfully integrated into the Chinese context, following its introduction as part of the national general training programme. Materials and methods: Online questionnaires (N = 91) and interviews (N = 22) were conducted with Year 1 trainee doctors and clinical supervisors at a cancer hospital in China to explore users' experiences, attitudes and opinions of the mini-CEX. Results" Trainees were more likely than supervisors to report understanding the purpose of the mini-CEX and agree that it encouraged reflection and helped improve overall performance. Both trainees and supervisors felt that it provided a framework for learning, that it was useful in identifying underperformance, and that it informed learning progression. Groups were equally positive about the commitment of their counterpart in the process and valued the focus on detailed feedback. It was perceived as cultivating the learner-teacher relationship. Overall, both groups felt they 'bought in' to using the mini-CEX. However, concerns were raised about subjectivity of ratings and lack of benchmarking with expected standards of care. Conclusions: Chinese trainees and supervisors generally perceived the mini-CEX as an acceptable and valuable medical training tool, although both groups suggested enhancements to improve its efficacy.
Collapse
Affiliation(s)
- Yuying Liang
- Department of Medical Education, Affiliated Cancer Hospital and Institute of Guangzhou Medical University, Guangzhou, China
- UCL Medical School, University College London, London, UK
| | | |
Collapse
|
30
|
Duggan N, Curran VR, Fairbridge NA, Deacon D, Coombs H, Stringer K, Pennell S. Using mobile technology in assessment of entrustable professional activities in undergraduate medical education. PERSPECTIVES ON MEDICAL EDUCATION 2021; 10:373-377. [PMID: 33095399 PMCID: PMC8633342 DOI: 10.1007/s40037-020-00618-9] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
BACKGROUND The adoption of competency-based medical education requires objective assessments of a learner's capability to carry out clinical tasks within workplace-based learning settings. This study involved an evaluation of the use of mobile technology to record entrustable professional activity assessments in an undergraduate clerkship curriculum. APPROACH A paper-based form was adapted to a mobile platform called eClinic Card. Students documented workplace-based assessments throughout core clerkship and preceptors confirmed accuracy via mobile phones. Assessment scores for the 2017-2018 academic year were collated and analyzed for all core rotations, and preceptors and students were surveyed regarding the mobile assessment experience. EVALUATION The mobile system enabled 80 students and 624 preceptors to document 6850 assessment submissions across 47 clinical sites over a 48-week core clerkship curriculum. Students' scores demonstrated progressive improvement across all entrustable professional activities with stage-appropriate levels of independence reported by end of core clerkship. Preceptors and students were satisfied with ease of use and dependability of the mobile assessment platform; however, students felt quality of formative coaching feedback could be improved. REFLECTION Our preliminary evaluation suggests the use of mobile technology to assess entrustable professional activity achievement across a core clerkship curriculum is a feasible and acceptable modality for workplace-based assessment. The use of mobile technology supported a programmatic assessment approach. However, meaningful coaching feedback, as well as faculty development and support, emerged as key factors influencing successful adoption and usage of entrustable professional activities within an undergraduate medical curriculum.
Collapse
Affiliation(s)
- Norah Duggan
- Faculty of Medicine, Memorial University, St. John's, Newfoundland, Canada
| | - Vernon R Curran
- Faculty of Medicine, Memorial University, St. John's, Newfoundland, Canada.
| | | | - Diana Deacon
- Faculty of Medicine, Memorial University, St. John's, Newfoundland, Canada
| | - Heidi Coombs
- Faculty of Medicine, Memorial University, St. John's, Newfoundland, Canada
| | - Katherine Stringer
- Faculty of Medicine, Memorial University, St. John's, Newfoundland, Canada
| | - Stephen Pennell
- Faculty of Medicine, Memorial University, St. John's, Newfoundland, Canada
| |
Collapse
|
31
|
Anderson HL, Kurtz J, West DC. Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S164-S174. [PMID: 34406132 DOI: 10.1097/acm.0000000000004366] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
PURPOSE Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions. METHOD The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders. RESULTS In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain. CONCLUSIONS Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.
Collapse
Affiliation(s)
- Hannah L Anderson
- H.L. Anderson is research associate, Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0002-9435-1535
| | - Joshua Kurtz
- J. Kurtz is a first-year resident, Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - Daniel C West
- D.C. West is professor of pediatrics, The Perelman School of Medicine at the University of Pennsylvania, and associate chair for education and senior director of medical education, Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0002-0909-4213
| |
Collapse
|
32
|
Phinney LB, Fluet A, O'Brien BC, Seligman L, Hauer KE. Using Activity Theory to Explore How Changes in a Work-Based Assessment Tool Can Alter Feedback Systems in Clerkships. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S212-S213. [PMID: 34705714 DOI: 10.1097/acm.0000000000004322] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
- Lauren B Phinney
- Author affiliations: L.B. Phinney, B.C. O'Brien, University of California, San Francisco
| | - Angelina Fluet
- A. Fluet, K.E. Hauer, University of California, San Francisco, School of Medicine
| | - Bridget C O'Brien
- Author affiliations: L.B. Phinney, B.C. O'Brien, University of California, San Francisco
| | - Lee Seligman
- L. Seligman New York-Presbyterian Hospital, Columbia University Irving Medical Center
| | - Karen E Hauer
- A. Fluet, K.E. Hauer, University of California, San Francisco, School of Medicine
| |
Collapse
|
33
|
Rhodes CSM, Lane H, Kumar K, Etherton-Beer C. 'Everyone's nice, but …': A qualitative exploration of trainees' experiences of postgraduate geriatric medicine training in Australia. Australas J Ageing 2021; 41:e16-e22. [PMID: 34617660 DOI: 10.1111/ajag.13006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2020] [Revised: 05/03/2021] [Accepted: 09/11/2021] [Indexed: 11/27/2022]
Abstract
OBJECTIVE To understand the experiences of trainees who undertake geriatric medicine advanced training in Australia. METHODS In this qualitative study, data were collected through semi-structured interviews with geriatric medicine advanced trainees. Data were analysed using a thematic analysis approach and data collection ceased when thematic saturation was achieved. RESULTS Thirteen trainees participated. Trainees enjoyed the training program, supported each other and felt prepared for independent practice as a geriatrician. However, they noted the variability of supervision and feedback practices, and educational opportunities afforded to them across different clinical learning environments. They felt unable to give feedback on their training. The research project and concerns about employment after training were also substantial sources of stress. CONCLUSION While geriatric medicine advanced trainees reported positively on much of their training, areas for improvement could include improved training for supervisors, improved support for research projects and more robust mechanisms for providing feedback.
Collapse
Affiliation(s)
- Caroline Sarah Marie Rhodes
- Department of Geriatric Medicine, Royal Perth Hospital, Perth, WA, Australia.,Prideaux Centre for Research in Health Professions Education, College of Medicine and Public Health, Flinders University, Bedford Park, SA, Australia
| | - Heather Lane
- Geriatric, Acute and Rehabilitation Medicine, Sir Charles Gairdner Hospital, Nedlands, WA, Australia
| | - Koshila Kumar
- Prideaux Centre for Research in Health Professions Education, College of Medicine and Public Health, Flinders University, Bedford Park, SA, Australia
| | - Christopher Etherton-Beer
- Department of Geriatric Medicine, Royal Perth Hospital, Perth, WA, Australia.,WA Centre for Health and Ageing, The University of Western Australia, Perth, WA, Australia
| |
Collapse
|
34
|
Piotrkowicz A, Wang K, Hallam J, Dimitrova V. Data-driven Exploration of Engagement with Workplace-based Assessment in the Clinical Skills Domain. INTERNATIONAL JOURNAL OF ARTIFICIAL INTELLIGENCE IN EDUCATION 2021. [DOI: 10.1007/s40593-021-00264-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
AbstractThe paper presents a multi-faceted data-driven computational approach to analyse workplace-based assessment (WBA) of clinical skills in medical education. Unlike formal university-based part of the degree, the setting of WBA can be informal and only loosely regulated, as students are encouraged to take every opportunity to learn from the clinical setting. For clinical educators and placement coordinators it is vital to follow and analyse students’ engagement with WBA while on placements, in order to understand how students are participating in the assessment, and what improvements can be made. We analyse digital data capturing the students’ WBA attempts and comments on how the assessments went, using process mining and text analytics. We compare Year 1 cohorts across three years, focusing on differences between primary vs. secondary care placements. The main contribution of the work presented in this paper is the exploration of computational approaches for multi-faceted, data-driven assessment analytics for workplace learning which includes:(i) a set of features for analysing clinical skills WBA data, (ii) analysis of the temporal aspects ofthat data using process mining, and (iii) utilising text analytics to compare student reflections on WBA. We show how assessment data captured during clinical placements can provide insights about the student engagement and inform the medical education practice. Our work is inspired by Jim Greer’s vision that intelligent methods and techniques should be adopted to address key challenges faced by educational practitioners in order to foster improvement of learning and teaching. In the broader AI in Education context, the paper shows the application of AI methods to address educational challenges in a new informal learning domain - practical healthcare placements in higher education medical training.
Collapse
|
35
|
Jamieson J, Hay M, Gibson S, Palermo C. Implementing programmatic assessment transforms supervisor attitudes: An explanatory sequential mixed methods study. MEDICAL TEACHER 2021; 43:709-717. [PMID: 33705668 DOI: 10.1080/0142159x.2021.1893678] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
INTRODUCTION Programmatic assessment (PA) is an increasingly popular approach to competency-based assessment (CBA), yet evaluation evidence is limited. This study aimed to identify and explore supervisor attitudes before and after implementing a novel PA using a sequential explanatory mixed methods design. In phase one, a survey was used to identify supervisor perspectives on work-based placements, PA and CBA. Survey results were then applied to develop focus group questions to further explore supervisor attitudes. RESULTS PA was found to improve supervisor-student relationships by removing high-stakes assessment decisions and creating greater capacity for feedback and teaching, leading to a productive learning environment. Assessment was perceived as an important role and supervisors wanted to feel valued and heard within PA. Trust was conceptualised as a triad between supervisor, student and university, and enabled supervisors to engage with PA which was important for success. Supervisor learning of PA was experiential and often supported by students, highlighting the need for hands-on training. CONCLUSION Participants reported a high level of agreement with PA and CBA principles which may have made them amenable to educational change. Further research is needed to explore the experience of all stakeholders and to understand how worldviews and culture influence assessment initiatives.
Collapse
Affiliation(s)
- Janica Jamieson
- Department of Nutrition and Dietetics, Monash University, Melbourne, Australia
- School of Medical and Health Sciences, Edith Cowan University, Perth, Australia
| | - Margaret Hay
- Department of Nutrition and Dietetics, Monash University, Melbourne, Australia
| | - Simone Gibson
- Department of Nutrition and Dietetics, Monash University, Melbourne, Australia
| | - Claire Palermo
- Department of Nutrition and Dietetics, Monash University, Melbourne, Australia
| |
Collapse
|
36
|
Young JQ, Frank JR, Holmboe ES. Advancing Workplace-Based Assessment in Psychiatric Education: Key Design and Implementation Issues. Psychiatr Clin North Am 2021; 44:317-332. [PMID: 34049652 DOI: 10.1016/j.psc.2021.03.005] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
Abstract
With the adoption of competency-based medical education, assessment has shifted from traditional classroom domains of knows and knows how to the workplace domain of doing. This workplace-based assessment has 2 purposes; assessment of learning (summative feedback) and the assessment for learning (formative feedback). What the trainee does becomes the basis for identifying growth edges and determining readiness for advancement and ultimately independent practice. High-quality workplace-based assessment programs require thoughtful choices about the framework of assessment, the tools themselves, the platforms used, and the contexts in which the assessments take place, with an emphasis on direct observation.
Collapse
Affiliation(s)
- John Q Young
- Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell and, Zucker Hillside Hospital at Northwell Health, 75-59 263rd Street, Kaufman Building, Glen Oaks, NY 11004, USA.
| | - Jason R Frank
- Department of Emergency Medicine, University of Ottawa, Royal College of Physicians and Surgeons of Canada, 774 Echo Drive, Ottawa, Ontario K15 5NB, Canada
| | - Eric S Holmboe
- Accreditation Council for Graduate Medical Education, ACGME, 401 North Michigan Avenue, Chicago, IL 60611, USA
| |
Collapse
|
37
|
Ochsendorf F, Welzel J. [Competence-oriented specialist training in dermatology]. DER HAUTARZT; ZEITSCHRIFT FUR DERMATOLOGIE, VENEROLOGIE, UND VERWANDTE GEBIETE 2021; 72:535-548. [PMID: 33999214 DOI: 10.1007/s00105-021-04820-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Subscribe] [Scholar Register] [Accepted: 03/25/2021] [Indexed: 11/24/2022]
Abstract
The new M WBO (Musterweiterbildungsordnung) has been developed for 6 years and will be put into force by most LÄK (Landesärztekammern) on 01.07.2020. Future training to become a dermatologist is competency-based. This is to enable observable, successful problem solving in practice. Thus, the acquisition of competence is only dependent on whether the goal has been achieved. New procedures are required to determine achievement of the respective competencies, including annual continuing education interviews, workplace-based examinations, and an eLogbuch (electronic logbook). Minimum inpatient periods will be eliminated in the future. In principle, continuing education can take place entirely on an outpatient basis. The M WBO Dermatology comprises 14 thematic blocks with differentiated description of cognitive and methodological or action competencies. The guideline numbers of dermatological additional training courses have been shifted considerably into the period of basic training. Additional training courses can be acquired while working. Fachlich empfohlene Weiterbildungspläne (FEWP) are the concrete implementation regulations of the M WBO. They are not part of the WBO and can be adapted. In connection with this new approach, numerous questions are currently still open, such as documentation in practice or financing.
Collapse
Affiliation(s)
- Falk Ochsendorf
- Klinik für Dermatologie, Venerologie und Allergologie, Universitätsklinikum, Theodor-Stern-Kai 7, 60590, Frankfurt am Main, Deutschland.
| | - Julia Welzel
- Klinik für Dermatologie und Allergologie, Universitätsklinikum Augsburg Medizincampus Süd, Sauerbruchstr. 6, 86179, Augsburg, Deutschland
| |
Collapse
|
38
|
Amir Rad FA, Otaki F, AlGurg R, Khan E, Davis D. A qualitative study of trainer and trainee perceptions and experiences of clinical assessment in post-graduate dental training. EUROPEAN JOURNAL OF DENTAL EDUCATION : OFFICIAL JOURNAL OF THE ASSOCIATION FOR DENTAL EDUCATION IN EUROPE 2021; 25:215-224. [PMID: 32813939 DOI: 10.1111/eje.12593] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/25/2020] [Revised: 07/13/2020] [Accepted: 08/07/2020] [Indexed: 05/08/2023]
Abstract
BACKGROUND The implementation of workplace-based assessment (WBA) needs to ensure the achievement of pre-set competences but may look different across varying contexts, such as in post-graduate dental education. The purpose of this study is to explore the perception of residents, faculty members and alumni concerning their experience with clinical assessment, and what configurations they consider as optimal to maximise the entailed learning experience. METHODS This study relied on a qualitative descriptive design using two data collection tools: focus group sessions, and semi-structured, one-to-one interviews. Data were triangulated from three sources: residents, faculty members and alumni. The data were inductively analysed based on constructivist epistemology. This was done using the Thematic Analysis approach, facilitated by NVivo software. RESULTS The analysis revealed two mutually exclusive themes: process and people. Within process, variables related to quality, workflow and feedback surfaced. As for the people theme, the main two group of stakeholders referred to in the related analysis were the trainees and the trainers. DISCUSSION There are many variables that need to be considered when developing an evidence-driven WBA. In addition, factoring into the design of the WBA the perception of the main stakeholders will enable contextualisation which is expected to raise the reliability of the adapted tools. CONCLUSION This study introduced a framework that could support post-graduate universities in their journey towards developing context-specific WBA.
Collapse
Affiliation(s)
- Fatemeh A Amir Rad
- Hamdan Bin Mohammed College of Dental Medicine, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates
- Masters in Medical Education Programme, Centre for Medical Education, School of Medicine, University of Dundee, Nethergate, Dundee, UK
| | - Farah Otaki
- Strategy and Institutional Excellence, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates
| | - Reem AlGurg
- Strategy and Institutional Excellence, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates
- College of Medicine, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates
| | - Erum Khan
- College of Medicine, Ajman University, Ajman, United Arab Emirates
| | - Dave Davis
- Centre for Outcomes and Research in Education, Mohammed Bin Rashid University of Medicine and Health Sciences, Dubai, United Arab Emirates
- Department of Family and Community Medicine, University of Toronto, Toronto, Canada
| |
Collapse
|
39
|
Pinilla S, Kyrou A, Klöppel S, Strik W, Nissen C, Huwendiek S. Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study. BMC MEDICAL EDUCATION 2021; 21:223. [PMID: 33882926 PMCID: PMC8059233 DOI: 10.1186/s12909-021-02637-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/17/2020] [Accepted: 03/27/2021] [Indexed: 06/12/2023]
Abstract
BACKGROUND Entrustable professional activities (EPAs) in competency-based, undergraduate medical education (UME) have led to new formative workplace-based assessments (WBA) using entrustment-supervision scales in clerkships. We conducted an observational, prospective cohort study to explore the usefulness of a WBA designed to assess core EPAs in a psychiatry clerkship. METHODS We analyzed changes in self-entrustment ratings of students and the supervisors' ratings per EPA. Timing and frequencies of learner-initiated WBAs based on a prospective entrustment-supervision scale and resultant narrative feedback were analyzed quantitatively and qualitatively. Predictors for indirect supervision levels were explored via regression analysis, and narrative feedback was coded using thematic content analysis. Students evaluated the WBA after each clerkship rotation. RESULTS EPA 1 ("Take a patient's history"), EPA 2 ("Assess physical & mental status") and EPA 8 ("Document & present a clinical encounter") were most frequently used for learner-initiated WBAs throughout the clerkship rotations in a sample of 83 students. Clinical residents signed off on the majority of the WBAs (71%). EPAs 1, 2, and 8 showed the largest increases in self-entrustment and received most of the indirect supervision level ratings. We found a moderate, positive correlation between self-entrusted supervision levels at the end of the clerkship and the number of documented entrustment-supervision ratings per EPA (p < 0.0001). The number of entrustment ratings explained 6.5% of the variance in the supervisors' ratings for EPA 1. Narrative feedback was documented for 79% (n = 214) of the WBAs. Most narratives addressed the Medical Expert role (77%, n = 208) and used reinforcement (59%, n = 161) as a feedback strategy. Students perceived the feedback as beneficial. CONCLUSIONS Using formative WBAs with an entrustment-supervision scale and prompts for written feedback facilitated targeted, high-quality feedback and effectively supported students' development toward self-entrusted, indirect supervision levels.
Collapse
Affiliation(s)
- Severin Pinilla
- University Hospital of Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland.
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland.
| | - Alexandra Kyrou
- University Hospital of Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Stefan Klöppel
- University Hospital of Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Werner Strik
- University Hospital of Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Christoph Nissen
- University Hospital of Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Sören Huwendiek
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| |
Collapse
|
40
|
Johansen RF, Nielsen RB, Malling BV, Storm H. Can case-based discussions in a group setting be used to assess residents' clinical skills? INTERNATIONAL JOURNAL OF MEDICAL EDUCATION 2021; 12:64-73. [PMID: 33840646 PMCID: PMC8411343 DOI: 10.5116/ijme.606a.eb39] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/26/2020] [Accepted: 04/05/2021] [Indexed: 06/12/2023]
Abstract
OBJECTIVES The purpose of this study was to explore residents' and assessors' perception of a new group assessment concept. METHODS This qualitative study consists of observations of four group assessment sessions, followed by semi-structured interviews with six residents and four assessors (specialists in internal medicine), who all volunteered to be interviewed. All residents at a medical department (eleven to fifteen each time) and four assessors participated in four group assessments, where the residents' clinical skills were assessed through case-based discussions. An external consultant (an anthropologist) performed the observations and the interviews. Notes from the observations and the interviews were analyzed using an inductive approach. RESULTS Eight of the ten interviewed participants preferred group assessment to individual assessment. Results from the interviews suggested that the group assessments were more consistent and that the level of discussion was perceived to be higher in the group discussions compared to the one-to-one discussions. All residents indicated that they had acquired new knowledge during their assessment and reported having learned from listening to the assessment of their peers. Assessors similarly reported gaining new knowledge. CONCLUSIONS The residents and assessors expressed very favourable attitudes toward the new group assessment concept. The assessment process was perceived to be higher in quality and more consistent, contributing to learning for all participating doctors in the department. Group assessment is feasible and acceptable, and provides a promising tool for assessment of clinical skills in the future.
Collapse
Affiliation(s)
| | | | - Bente V. Malling
- Department of Clinical Medicine, Health, Aarhus University, Denmark
| | - Hanne Storm
- Diagnostic Center, Regional Hospital Silkeborg, Regional Hospital Central, Jutland, Denmark
| |
Collapse
|
41
|
Boet S, Etherington N, Lam S, Lê M, Proulx L, Britton M, Kenna J, Przybylak-Brouillard A, Grimshaw J, Grantcharov T, Singh S. Implementation of the Operating Room Black Box Research Program at the Ottawa Hospital Through Patient, Clinical, and Organizational Engagement: Case Study. J Med Internet Res 2021; 23:e15443. [PMID: 33724199 PMCID: PMC8074833 DOI: 10.2196/15443] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2019] [Revised: 12/11/2019] [Accepted: 02/26/2020] [Indexed: 11/13/2022] Open
Abstract
Background A large proportion of surgical patient harm is preventable; yet, our ability to systematically learn from these incidents and improve clinical practice remains limited. The Operating Room Black Box was developed to address the need for comprehensive assessments of clinical performance in the operating room. It captures synchronized audio, video, patient, and environmental clinical data in real time, which are subsequently analyzed by a combination of expert raters and software-based algorithms. Despite its significant potential to facilitate research and practice improvement, there are many potential implementation challenges at the institutional, clinician, and patient level. This paper summarizes our approach to implementation of the Operating Room Black Box at a large academic Canadian center. Objective We aimed to contribute to the development of evidence-based best practices for implementing innovative technology in the operating room for direct observation of the clinical performance by using the case of the Operating Room Black Box. Specifically, we outline the systematic approach to the Operating Room Black Box implementation undertaken at our center. Methods Our implementation approach included seeking support from hospital leadership; building frontline support and a team of champions among patients, nurses, anesthesiologists, and surgeons; accounting for stakeholder perceptions using theory-informed qualitative interviews; engaging patients; and documenting the implementation process, including barriers and facilitators, using the consolidated framework for implementation research. Results During the 12-month implementation period, we conducted 23 stakeholder engagement activities with over 200 participants. We recruited 10 clinician champions representing nursing, anesthesia, and surgery. We formally interviewed 15 patients and 17 perioperative clinicians and identified key themes to include in an information campaign run as part of the implementation process. Two patient partners were engaged and advised on communications as well as grant and protocol development. Many anticipated and unanticipated challenges were encountered at all levels. Implementation was ultimately successful, with the Operating Room Black Box installed in August 2018, and data collection beginning shortly thereafter. Conclusions This paper represents the first step toward evidence-guided implementation of technologies for direct observation of performance for research and quality improvement in surgery. With technology increasingly being used in health care settings, the health care community should aim to optimize implementation processes in the best interest of health care professionals and patients.
Collapse
Affiliation(s)
- Sylvain Boet
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada.,Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada.,Department of Innovation in Medical Education, University of Ottawa, Ottawa, ON, Canada.,Francophone Affairs, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Nicole Etherington
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada.,Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
| | - Sandy Lam
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada.,Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
| | - Maxime Lê
- Patient and Family Advisory Council, The Ottawa Hospital, Ottawa, ON, Canada
| | - Laurie Proulx
- Patient and Family Advisory Council, The Ottawa Hospital, Ottawa, ON, Canada
| | - Meghan Britton
- Main Operating Room, The Ottawa Hospital, Ottawa, ON, Canada
| | - Julie Kenna
- Main Operating Room, The Ottawa Hospital, Ottawa, ON, Canada
| | - Antoine Przybylak-Brouillard
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada.,Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
| | - Jeremy Grimshaw
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
| | - Teodor Grantcharov
- Department of General Surgery, University of Toronto, Toronto, ON, Canada.,Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, ON, Canada
| | - Sukhbir Singh
- Department of Obstetrics, Gynecology, and Newborn Care, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
42
|
Boursicot K, Kemp S, Wilkinson T, Findyartini A, Canning C, Cilliers F, Fuller R. Performance assessment: Consensus statement and recommendations from the 2020 Ottawa Conference. MEDICAL TEACHER 2021; 43:58-67. [PMID: 33054524 DOI: 10.1080/0142159x.2020.1830052] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
INTRODUCTION In 2011 the Consensus Statement on Performance Assessment was published in Medical Teacher. That paper was commissioned by AMEE (Association for Medical Education in Europe) as part of the series of Consensus Statements following the 2010 Ottawa Conference. In 2019, it was recommended that a working group be reconvened to review and consider developments in performance assessment since the 2011 publication. METHODS Following review of the original recommendations in the 2011 paper and shifts in the field across the past 10 years, the group identified areas of consensus and yet to be resolved issues for performance assessment. RESULTS AND DISCUSSION This paper addresses developments in performance assessment since 2011, reiterates relevant aspects of the 2011 paper, and summarises contemporary best practice recommendations for OSCEs and WBAs, fit-for-purpose methods for performance assessment in the health professions.
Collapse
Affiliation(s)
- Katharine Boursicot
- Department of Assessment and Progression, Duke-National University of Singapore, Singapore, Singapore
| | - Sandra Kemp
- Curtin Medical School, Curtin University, Perth, Australia
| | - Tim Wilkinson
- Dean's Department, University of Otago, Christchurch, New Zealand
| | - Ardi Findyartini
- Department of Medical Education, Universitas Indonesia, Jakarta, Indonesia
| | - Claire Canning
- Department of Assessment and Progression, Duke-National University of Singapore, Singapore, Singapore
| | - Francois Cilliers
- Department of Health Sciences Education, University of Cape Town, Cape Town, South Africa
| | | |
Collapse
|
43
|
Young JQ, Sugarman R, Schwartz J, O'Sullivan PS. Faculty and Resident Engagement With a Workplace-Based Assessment Tool: Use of Implementation Science to Explore Enablers and Barriers. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:1937-1944. [PMID: 32568853 DOI: 10.1097/acm.0000000000003543] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
PURPOSE Implementation of workplace-based assessment programs has encountered significant challenges. Faculty and residents alike often have a negative view of these programs as "tick-box" or "jump through the hoops" exercises. A number of recommendations have been made to address these challenges. To understand the experience with a workplace-based assessment tool that follows many of these recommendations, the authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with the tool. METHOD The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) is a direct observation tool designed to assess resident performance during a psychiatric medication management visit. From August 2017 to February 2018, the P-SCO was implemented in the outpatient continuity clinics for second- and third-year residents at Zucker Hillside Hospital/Northwell Health. In February and March 2019, the authors conducted semistructured interviews of participating faculty and residents. Interview guides based on the CFIR were used to capture the enablers and barriers to engagement. Interview transcripts were independently coded. Codes were then organized into themes relevant to the domains of the CFIR. RESULTS Ten faculty and 10 residents were interviewed. Overall, participants had a positive experience with the P-SCO. Enabling factors for faculty and residents included the ongoing training, design features of the P-SCO, predisposing beliefs, dedicated faculty time, and the perception that the P-SCO improved verbal feedback quality. Barriers for faculty included checklist length and discomfort with feedback that threatens identity, and barriers for residents included faculty variability in timeliness and quality of feedback and minimal review of the feedback after initial receipt. CONCLUSIONS This study demonstrates that the negative experience of faculty and residents with workplace-based assessment tools shown in prior studies can be overcome, at least in part, when specific implementation strategies are pursued. The findings provide guidance for future research and implementation efforts.
Collapse
Affiliation(s)
- John Q Young
- J.Q. Young is professor of psychiatry and vice chair for education, Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, and Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York
| | - Rebekah Sugarman
- R. Sugarman is a research assistant, Department of Psychiatry, Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York
| | - Jessica Schwartz
- J. Schwartz is a resident, Department of Psychiatry, Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York
| | - Patricia S O'Sullivan
- P.S. O'Sullivan is professor, Department of Medicine, and director of research and development in medical education, University of California, San Francisco, School of Medicine, San Francisco, California
| |
Collapse
|
44
|
Ossenberg C, Henderson A, Mitchell M. The use of factor analysis and abductive inference to explore students' and practitioners' perspectives of feedback: divergent or congruent understanding? BMC MEDICAL EDUCATION 2020; 20:466. [PMID: 33238974 PMCID: PMC7687844 DOI: 10.1186/s12909-020-02378-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/14/2020] [Accepted: 11/11/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND The importance of feedback in workplace-based settings cannot be underestimated. Approaches that evaluate feedback reflect either the sender's or receiver's viewpoint in isolation of each other. This study investigated prevailing student and practitioner views of feedback resulting from development and testing of a survey about feedback. METHOD This study used a cross-sectional design, incorporating use of expert consultation and factor analysis of surveys. Fifty-two items based on identified attributes for effective feedback from current research were developed and reviewed through expert consultation. Surveys developed from the items were completed by students (n = 209) and practitioners (n = 145). The juxtaposition of items based on students' and practitioners' responses to the surveys were examined through use of exploratory factor analysis. RESULTS Separate student and practitioner surveys resulted. Each survey contained 23 items that clustered into factors. The item statements were different across practitioner and student groups Only nine items were shared across factors identified for both groups. The resulting factors represented different notions of feedback-namely, practitioners had a process-oriented focus in comparison with students' outcome focus. CONCLUSION While students and practitioners view feedback differently this does not necessarily mean they are incongruous.
Collapse
Affiliation(s)
- Christine Ossenberg
- Griffith University, School of Nursing and Midwifery, Kessels Road, Nathan, Queensland 4111 Australia
- Central Queensland University, Ann Street, Brisbane, Queensland 4000 Australia
| | - Amanda Henderson
- Griffith University, School of Nursing and Midwifery, Kessels Road, Nathan, Queensland 4111 Australia
- Central Queensland University, Ann Street, Brisbane, Queensland 4000 Australia
| | - Marion Mitchell
- Griffith University, School of Nursing and Midwifery, Kessels Road, Nathan, Queensland 4111 Australia
- Central Queensland University, Ann Street, Brisbane, Queensland 4000 Australia
- Menzies Health Institute Queensland, Griffith University, Brisbane, 4111 Australia
| |
Collapse
|
45
|
Prentice S, Benson J, Kirkpatrick E, Schuwirth L. Workplace-based assessments in postgraduate medical education: A hermeneutic review. MEDICAL EDUCATION 2020; 54:981-992. [PMID: 32403200 DOI: 10.1111/medu.14221] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/10/2019] [Revised: 03/30/2020] [Accepted: 05/06/2020] [Indexed: 06/11/2023]
Abstract
OBJECTIVES Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education. METHOD This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the United Kingdom, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature in order to answer three scoping questions concerning acceptability, effectiveness and assessment training. As each round was coded, themes were refined and questions added until saturation was reached. RESULTS Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy, disagreement with definitions and frameworks, and inadequate summative use of WBAs. Effectiveness is influenced by user (eg, engagement and assessment literacy) and tool attributes (eg, definitions and scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessors' assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline. CONCLUSIONS The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learned regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.
Collapse
Affiliation(s)
- Shaun Prentice
- GPEx Ltd., Adelaide, South Australia, Australia
- School of Psychology, University of Adelaide, Adelaide, South Australia, Australia
| | - Jill Benson
- GPEx Ltd., Adelaide, South Australia, Australia
- Health in Human Diversity Unit, School of Medicine, University of Adelaide, Adelaide, South Australia, Australia
- Prideaux Centre, Flinders University, Adelaide, South Australia, Australia
| | - Emily Kirkpatrick
- GPEx Ltd., Adelaide, South Australia, Australia
- School of Medicine, University of Adelaide, Adelaide, South Australia, Australia
| | - Lambert Schuwirth
- Prideaux Centre, Flinders University, Adelaide, South Australia, Australia
- Maastrich University, Maastricht, the Netherlands
- Uniformed University for the Health Sciences, Bethesda, Maryland, USA
| |
Collapse
|
46
|
McEllistrem B, Barrett A, Hanley K. Performance in practice; exploring trainer and trainee experiences of user-designed formative assessment tools. EDUCATION FOR PRIMARY CARE 2020; 32:27-33. [PMID: 33094687 DOI: 10.1080/14739879.2020.1815085] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Abstract
INTRODUCTION General Practice training in Ireland currently has various methods of formative assessment and feedback delivered to trainees. In 2018 the Irish College of General Practitioners commissioned the generation of two new user-designed formative feedback tools that would allow trainee feedback to drive learning. These tools became known as the Performance in Practice (PiP) tools. AIMS To explore the experiences of General Practice (GP) trainers and trainees having completed a pilot of using the PiP tools for 4 months. METHODS An explorative phenomenological approach was taken to understand the experiences of trainers and trainees. One to one interviews were conducted, and the transcripts analysed for themes and sub-theme via Template analysis. RESULTS User experiences focused on two main areas; educational value and acceptability. In relation to educational value, the PiP tools were seen as an improvement over established forms of formative feedback, as they were centred around the curriculum and therefore reflected the unique multifaceted requirements of an independently practising GP. Acceptability primarily focused around data governance and structures, as well as practical issues such as ease of software use. CONCLUSIONS Overall, the experience of using the PiP tools was positive for both trainers and trainees. Future plans to further explore implementation of the PiP tools have been significantly informed by this research.
Collapse
Affiliation(s)
- B McEllistrem
- General Practice Training Unit, Irish College of General Practitioners, Dublin, Ireland
| | - A Barrett
- General Practice Training Unit, Irish College of General Practitioners, Dublin, Ireland
| | - K Hanley
- General Practice Training Unit, Irish College of General Practitioners, Dublin, Ireland
| |
Collapse
|
47
|
Young JQ, Sugarman R, Schwartz J, O'Sullivan PS. Overcoming the Challenges of Direct Observation and Feedback Programs: A Qualitative Exploration of Resident and Faculty Experiences. TEACHING AND LEARNING IN MEDICINE 2020; 32:541-551. [PMID: 32529844 DOI: 10.1080/10401334.2020.1767107] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
Problem: Prior studies have reported significant negative attitudes amongst both faculty and residents toward direct observation and feedback. Numerous contributing factors have been identified, including insufficient time for direct observation and feedback, poorly understood purpose, inadequate training, disbelief in the formative intent, inauthentic resident-patient clinical interactions, undermining of resident autonomy, lack of trust between the faculty-resident dyad, and low-quality feedback information that lacks credibility. Strategies are urgently needed to overcome these challenges and more effectively engage faculty and residents in direct observation and feedback. Otherwise, the primary goals of supporting both formative and summative assessment will not be realized and the viability of competency-based medical education will be threatened. Intervention: Toward this end, recent studies have recommended numerous strategies to overcome these barriers: protected time for direct observation and feedback; ongoing faculty and resident training on goals and bidirectional, co-constructed feedback; repeated direct observations and feedback within a longitudinal resident-supervisor relationship; utilization of assessment tools with evidence for validity; and monitoring for engagement. Given the complexity of the problem, it is likely that bundling multiple strategies together will be necessary to overcome the challenges. The Direct Observation Structured Feedback Program (DOSFP) incorporated many of the recommended features, including protected time for direct observation and feedback within longitudinal faculty-resident relationships. Using a qualitative thematic approach the authors conducted semi-structured interviews, during February and March, 2019, with 10 supervisors and ten residents. Participants were asked to reflect on their experiences. Interview guide questions explored key themes from the literature on direct observation and feedback. Transcripts were anonymized. Two authors independently and iteratively coded the transcripts. Coding was theory-driven and differences were discussed until consensus was reached. The authors then explored the relationships between the codes and used a semantic approach to construct themes. Context: The DOSFP was implemented in a psychiatry continuity clinic for second and third year residents. Impact: Faculty and residents were aligned around the goals. They both perceived the DOSFP as focused on growth rather than judgment even though residents understood that the feedback had both formative and summative purposes. The DOSFP facilitated educational alliances characterized by trust and respect. With repeated practice within a longitudinal relationship, trainees dropped the performance orientation and described their interactions with patients as authentic. Residents generally perceived the feedback as credible, described feedback quality as high, and valued the two-way conversation. However, when receiving feedback with which they did not agree, residents demurred or, at most, would ask a clarifying question, but then internally discounted the feedback. Lessons Learned: Direct observation and structured feedback programs that bundle recent recommendations may overcome many of the challenges identified by previous research. Yet, residents discounted disagreeable feedback, illustrating a significant limitation and the need for other strategies that help residents reconcile conflict between external data and one's self-appraisal.
Collapse
Affiliation(s)
- John Q Young
- Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, USA
| | - Rebekah Sugarman
- Department of Psychiatry, The Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York, USA
| | - Jessica Schwartz
- Department of Psychiatry, The Zucker Hillside Hospital at Northwell Health, Glen Oaks, New York, USA
| | - Patricia S O'Sullivan
- Office of Medical Education, University of California San Francisco, San Francisco, California, USA
| |
Collapse
|
48
|
Young JQ, Sugarman R, Schwartz J, McClure M, O'Sullivan PS. A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement. PERSPECTIVES ON MEDICAL EDUCATION 2020; 9:210-219. [PMID: 32504446 PMCID: PMC7459074 DOI: 10.1007/s40037-020-00587-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2023]
Abstract
INTRODUCTION Mobile apps that utilize the framework of entrustable professional activities (EPAs) to capture and deliver feedback are being implemented. If EPA apps are to be successfully incorporated into programmatic assessment, a better understanding of how they are experienced by the end-users will be necessary. The authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with an EPA app. METHODS Structured interviews of faculty and residents were conducted with an interview guide based on the CFIR. Transcripts were independently coded by two study authors using directed content analysis. Differences were resolved via consensus. The study team then organized codes into themes relevant to the domains of the CFIR. RESULTS Eight faculty and 10 residents chose to participate in the study. Both faculty and residents found the app easy to use and effective in facilitating feedback immediately after the observed patient encounter. Faculty appreciated how the EPA app forced brief, distilled feedback. Both faculty and residents expressed positive attitudes and perceived the app as aligned with the department's philosophy. Barriers to engagement included faculty not understanding the EPA framework and scale, competing clinical demands, residents preferring more detailed feedback and both faculty and residents noting that the app's feedback should be complemented by a tool that generates more systematic, nuanced, and comprehensive feedback. Residents rarely if ever returned to the feedback after initial receipt. DISCUSSION This study identified key enablers and barriers to engagement with the EPA app. The findings provide guidance for future research and implementation efforts focused on the use of mobile platforms to capture direct observation feedback.
Collapse
Affiliation(s)
- John Q Young
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell and the Zucker Hillside Hospital at Northwell Health, Hempstead, NY, USA.
| | - Rebekah Sugarman
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell and the Zucker Hillside Hospital at Northwell Health, Hempstead, NY, USA
| | - Jessica Schwartz
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell and the Zucker Hillside Hospital at Northwell Health, Hempstead, NY, USA
| | - Matthew McClure
- Donald and Barbara Zucker School of Medicine at Hofstra/Northwell and the Zucker Hillside Hospital at Northwell Health, Hempstead, NY, USA
| | - Patricia S O'Sullivan
- Department of Medicine, University of California at San Francisco School of Medicine, San Francisco, USA
| |
Collapse
|
49
|
Ossenberg C, Mitchell M, Henderson A. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian 2020. [DOI: 10.1016/j.colegn.2019.10.005] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
50
|
Burm S, Sebok-Syer SS, Van Koughnett JA, Watling CJ. Are we generating more assessments without added value? Surgical trainees' perceptions of and receptiveness to cross-specialty assessment. PERSPECTIVES ON MEDICAL EDUCATION 2020; 9:201-209. [PMID: 32504448 PMCID: PMC7459015 DOI: 10.1007/s40037-020-00594-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
INTRODUCTION Competency-based medical education (CBME) hinges on robust assessment. However, integrating regular workplace-based assessment within demanding and sometimes chaotic clinical environments remains challenging. Many faculty lack assessment expertise, and some programs lack the infrastructure and faculty numbers to fulfill CBME's mandate. Recognizing this, we designed and implemented an assessment innovation that trains and deploys a cadre of faculty to assess in specialties outside their own. Specifically, we explored trainees' perceptions of and receptiveness to this novel assessment approach. METHODS Within Western University's Surgical Foundations program, 27 PGY‑1 trainees were formatively assessed by trained non-surgeons on a basic laparoscopic surgical skill. These assessments did not impact trainees' progression. Four focus groups were conducted to gauge residents' sentiments about the experience of cross-specialty assessment. Data were then analyzed using a thematic analysis approach. RESULTS While a few trainees found the experience motivating, more often trainees questioned the feedback they received and the practicality of this assessment approach to advance their procedural skill acquisition. What trainees wanted were strategies for improvement, not merely an assessment of performance. DISCUSSION Trainees' trepidation at the idea of using outside assessors to meet increased assessment demands appeared grounded in their expectations for assessment. What trainees appeared to desire was a coach-someone who could break their performance into its critical individual components-as opposed to an assessor whose role was limited to scoring their performance. Understanding trainees' receptivity to new assessment approaches is crucial; otherwise training programs run the risk of generating more assessments without added value.
Collapse
Affiliation(s)
- Sarah Burm
- Division of Medical Education, Dalhousie University, Halifax, Nova Scotia, Canada.
| | | | - Julie Ann Van Koughnett
- Divisions of General Surgery and Surgical Oncology, Western University, London, Ontario, Canada
| | - Christopher J Watling
- Departments of Oncology and Clinical Neurological Sciences, Centre for Education Research and Innovation, Western University, London, Ontario, Canada
| |
Collapse
|