1
|
L'Huillier JC, Woodward JM, Lund S, Gan CY, Moreci R, Silvestri C, Brian R, Zarate Rodriguez JG, Roshal J, White BAA. Is it gossip or feedback? Surgical attendings' perceptions of gossip within residency. JOURNAL OF SURGICAL EDUCATION 2024; 81:1362-1373. [PMID: 39173427 DOI: 10.1016/j.jsurg.2024.07.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/22/2024] [Revised: 06/07/2024] [Accepted: 07/12/2024] [Indexed: 08/24/2024]
Abstract
OBJECTIVE Gossip-evaluative talk about an absent third party-exists in surgical residency programs. Attending surgeons may engage in gossip to provide residents with feedback on performance, which may contribute to bias. Nevertheless, the perspectives of attending surgeons on gossip has not been studied. DESIGN In this qualitative study, semi-structured interviews about gossip in surgical training were conducted with attending surgeons. We performed a reflexive thematic analysis of transcripts with a grounded theory approach to describe attendings' perceptions of their role in gossip within surgical residency. SETTING Interviews were conducted from September 23, 2023, to November 27, 2023 via Zoom™. PARTICIPANTS Eighteen surgery attendings associated with 7 surgical training programs were interviewed. RESULTS Six themes were developed: 1) Attendings typically view gossip with a negative lens; thus, well-intended conversations about resident performance that meet the academic definition of gossip are not perceived as gossip; 2) Gossip can damage attendings' reputations as surgeons and educators; 3) Mitigating the negative impacts of gossip by maintaining accurate and objective standards of honest communication is hard; 4) Attendings express concerns about hearing other attendings' impressions of residents prior to formulating their own opinion; 5) The surgical hierarchy restricts the volume and content of gossip that reaches attendings, which may limit their knowledge of program culture; and 6) It is very difficult to mitigate gossip at the program level. Ultimately, attendings utilize gossip (e.g. triangulating their experience) with the goal of providing residents feedback. CONCLUSIONS Defining important conversations about resident performance as gossip should not discourage these critically important conversations but rather underscore the importance of combating harmful gossip through 3 behaviors: 1) committing to objective communication; 2) limiting or reframing information about resident performance that is shared with attendings who have yet to formulate their own opinions; and 3) regulating gossip in particular high-stakes microenvironments (e.g. the operating room).
Collapse
Affiliation(s)
- Joseph C L'Huillier
- Department of Surgery, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo, Buffalo, NY; Department of Epidemiology and Environmental Health, Division of Health Services Policy and Practice, School of Public Health and Health Professions, University at Buffalo, Buffalo, NY; Department of Health Professions Education, MGH Institute for Health Professions Education, Boston, MA.
| | - John M Woodward
- Department of Surgery, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo, Buffalo, NY; Department of Health Professions Education, MGH Institute for Health Professions Education, Boston, MA
| | - Sarah Lund
- Department of Surgery, Mayo Clinic, Rochester, MN
| | - Connie Y Gan
- Department of Surgery, Oregon Health & Science University, Portland, OR; Department of Surgery, Stanford University, Stanford, CA
| | - Rebecca Moreci
- Department of Surgery, Louisiana State University Health Sciences Center, New Orleans, LA; Department of Surgery, University of Michigan, Ann Arbor, MI
| | - Caitlin Silvestri
- Department of Surgery, New York Presbyterian/Columbia University Irving Medical Center, New York, NY
| | - Riley Brian
- Department of Surgery, University of California San Francisco, San Francisco, CA
| | | | - Joshua Roshal
- Department of Surgery, University of Texas Medical Branch, Galveston, TX; Department of Surgery, Brigham and Women's Hospital, Boston, MA
| | - Bobbie Ann Adair White
- Department of Health Professions Education, MGH Institute for Health Professions Education, Boston, MA; Department of Surgery, Baylor Scott & White Health, Temple, TX
| |
Collapse
|
2
|
Hanmore T, Moon CC, Curtis R, Hopman W, Baxter S. Is time really of the essence? Timeliness of narrative feedback in ophthalmology CBME assessments. MEDICAL TEACHER 2024; 46:705-710. [PMID: 37910021 DOI: 10.1080/0142159x.2023.2274286] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/03/2023]
Abstract
PURPOSE Competency-based medical education relies on a strong program of assessment, and quality comments play a vital role in ensuring its success. The goal of this study is to determine the effect of the timeliness of assessment completion on the quality of the feedback. MATERIALS AND METHODS Using the Quality of Assessment for Learning (QuAL) score 2478 assessments were evaluated. The assessments included those completed between July 2017 and December 2020 for 18 ophthalmology residents. Spearman correlation, Mann-Whitney U and Kruskal-Wallis tests were used to assess variations in QuAL scores based on the timeliness of assessment completion. RESULTS The timeliness of assessment completion ranged from 0 to 299 d with the mean time for completion being 3 d. As the delay increased, the QuAL score decreased. Feedback provided 4, 5, and 14 d post-encounter demonstrated statistically significant differences in the QuAL score. Additionally, there was a significant difference in the timeliness of feedback when there is no written comment. CONCLUSIONS This study demonstrates that the timeliness of assessment completion might have an effect on the quality of written feedback. Written feedback should be completed within 14 d of the encounter to optimize quantity and quality.
Collapse
Affiliation(s)
- Tessa Hanmore
- Department of Ophthalmology, Queen's University, Kingston, Canada
- Department of Physical Medicine and Rehabilitation, Queen's University, Kingston, Canada
- Department of Psychiatry, Queen's University, Kingston, Canada
| | | | - Rachel Curtis
- Department of Ophthalmology, Queen's University, Kingston, Canada
| | - Wilma Hopman
- Department of Public Health Sciences, Queen's University, Kingston, Canada
| | - Stephanie Baxter
- Department of Ophthalmology, Queen's University, Kingston, Canada
| |
Collapse
|
3
|
Montgomery KB, Mellinger JD, Lindeman B. Entrustable Professional Activities in Surgery: A Review. JAMA Surg 2024; 159:571-577. [PMID: 38477902 PMCID: PMC11260519 DOI: 10.1001/jamasurg.2023.8107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/14/2024]
Abstract
Importance Entrustable professional activities (EPAs) compose a competency-based education (CBE) assessment framework that has been increasingly adopted across medical specialties as a workplace-based assessment tool. EPAs focus on directly observed behaviors to determine the level of entrustment a trainee has for a given activity of that specialty. In this narrative review, we highlight the rationale for EPAs in general surgery, describe current evidence supporting their use, and outline some of the practical considerations for EPAs among residency programs, faculty, and trainees. Observations An expanding evidence base for EPAs in general surgery has provided moderate validity evidence for their use as well as practical recommendations for implementation across residency programs. Challenges to EPA use include garnering buy-in from individual faculty and residents to complete EPA microassessments and engage in timely, specific feedback after a case or clinical encounter. When successfully integrated into a program's workflow, EPAs can provide a more accurate picture of residents' competence for a fundamental surgical task or activity compared with other assessment methods. Conclusions and Relevance EPAs represent the next significant shift in the evaluation of general surgery residents as part of the overarching progression toward CBE among all US residency programs. While pragmatic challenges to the implementation of EPAs remain, the best practices from EPA and other CBE assessment literature summarized in this review may assist individuals and programs in implementing EPAs. As EPAs become more widely used in general surgery resident training, further analysis of barriers and facilitators to successful and sustainable EPA implementation will be needed to continue to optimize and advance this new assessment framework.
Collapse
|
4
|
Marty AP, Linsenmeyer M, George B, Young JQ, Breckwoldt J, Ten Cate O. Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154. MEDICAL TEACHER 2023; 45:1203-1213. [PMID: 36706225 DOI: 10.1080/0142159x.2023.2168527] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.
Collapse
Affiliation(s)
| | - Machelle Linsenmeyer
- West Virginia School of Osteopathic Medicine, Lewisburg, WV, United States of America
| | - Brian George
- Surgery and Learning Health Sciences, University of Michigan, Ann Arbor, Michigan, United States of America
| | - John Q Young
- Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell &, Zucker Hillside Hospital, NY, United States of America
| | - Jan Breckwoldt
- Institute of Anesthesia at the University Hospital Zurich, Switzerland
| | - Olle Ten Cate
- Utrecht Center for Research and Development of Health Professions Education at UMC Utrecht, The Netherlands
| |
Collapse
|
5
|
Faber DA, Hinman JM, Knauer EM, Hechenbleikner EM, Badell IR, Lin E, Srinivasan JK, Chahine AA, Papandria DJ. Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees. J Surg Res 2023; 291:574-585. [PMID: 37540975 DOI: 10.1016/j.jss.2023.07.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Revised: 06/29/2023] [Accepted: 07/02/2023] [Indexed: 08/06/2023]
Abstract
INTRODUCTION Assessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance. MATERIALS AND METHODS A web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments. RESULTS The Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority. CONCLUSIONS This novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.
Collapse
Affiliation(s)
- David A Faber
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Johanna M Hinman
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Eric M Knauer
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | | | - I Raul Badell
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Edward Lin
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | | | - A Alfred Chahine
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia
| | - Dominic J Papandria
- Department of Surgery, Emory University School of Medicine, Atlanta, Georgia.
| |
Collapse
|
6
|
Cox ML, Weaver ML, Johnson C, Chen X, Carter T, Yee CC, Coleman DM, Sgroi MD, George BC, Smith BK. Early findings and strategies for successful implementation of SIMPL workplace-based assessments within vascular surgery residency and fellowship programs. J Vasc Surg 2023; 78:806-814.e2. [PMID: 37164236 DOI: 10.1016/j.jvs.2023.04.039] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Revised: 04/13/2023] [Accepted: 04/30/2023] [Indexed: 05/12/2023]
Abstract
OBJECTIVE As medical education systems increasingly move toward competency-based training, it is important to understand the tools available to assess competency and how these tools are utilized. The Society for Improving Medical Professional Learning (SIMPL) offers a smart phone-based assessment system that supports workplace-based assessment of residents' and fellows' operative autonomy, performance, and case complexity. The purpose of this study was to characterize implementation of the SIMPL app within vascular surgery integrated residency (0+5) and fellowship (5+2) training programs. METHODS SIMPL operative ratings recorded between 2018 and 2022 were collected from all participating vascular surgery training institutions (n = 9 institutions with 5+2 and 0+5 programs; n = 4 institutions with 5+2 program only). The characteristics of programs, trainees, faculty, and SIMPL operative assessments were evaluated using descriptive statistics. RESULTS Operative assessments were completed for 2457 cases by 85 attendings and 86 trainees, totaling 4615 unique operative assessment ratings. Attendings included dictated feedback in 52% of assessments. Senior-level residents received more assessments than junior-level residents (postgraduate year [PGY]1-3, n = 439; PGY4-5, n = 551). Performance ratings demonstrated increases from junior to senior trainees for both resident and fellow cohorts with "performance-ready" or "exceptional performance" ratings increasing by nearly two-fold for PGY1 to PGY5 residents (28.1% vs 40.6%), and from first- to second-year fellows (PGY6, 46.7%; PGY7, 60.3%). Similar gains in autonomy were demonstrated as trainees progressed through training. Senior residents were more frequently granted autonomy with "supervision only" than junior residents (PGY1, 8.7%; PGY5, 21.6%). "Supervision only" autonomy ratings were granted to 21.8% of graduating fellows. Assessment data included a greater proportion of complex cases for senior compared with junior fellows (PGY6, 20.9% vs PGY7, 26.5%). Program Directors felt that faculty and trainee buy-in were the main barriers to implementation of the SIMPL assessment app. CONCLUSIONS This is the first description of the SIMPL app as an operative assessment tool within vascular surgery that has been successfully implemented in both residency and fellowship programs. The assessment data demonstrates expected progressive gains in trainees' autonomy and performance, as well as increasing case complexity, across PGY years. Given the selection of SIMPL as the assessment platform for required American Board of Surgery and Vascular Surgery Board Entrustable Professional Activities assessments, understanding facilitators and barriers to implementation of workplace-based assessments using this app is imperative, particularly as we move toward competency-based medical education.
Collapse
Affiliation(s)
- Morgan L Cox
- Division of Vascular Surgery and Endovascular Therapy, Department of Surgery, University of Florida, Gainesville, FL.
| | - M Libby Weaver
- Vascular and Endovascular Surgery Division, University of Virginia Department of Surgery, University of Virginia, Charlottesville, VA
| | - Cali Johnson
- Division of Vascular Surgery, Department of Surgery, University of Utah School of Medicine, Salt Lake City, UT
| | - Xilin Chen
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI
| | - Taylor Carter
- Division of General Surgery, Department of Surgery, University of North Carolina, Chapel Hill, NC
| | - Chia Chye Yee
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI
| | - Dawn M Coleman
- Division of Vascular and Endovascular Surgery, Department of Surgery, Duke University, Durham, NC
| | - Michael D Sgroi
- Division of Vascular and Endovascular Surgery, Stanford University, Stanford, CA
| | - Brian C George
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI
| | - Brigitte K Smith
- Division of Vascular Surgery, Department of Surgery, University of Utah School of Medicine, Salt Lake City, UT
| |
Collapse
|
7
|
Atkinson RB, Sidey-Gibbons C, Smink DS, Askari R, Pusic AL, Cho NL, Robertson JM, Rangel EL. Real-Time Student Feedback on the Surgical Learning Environment: Use of a Mobile Application. JOURNAL OF SURGICAL EDUCATION 2023; 80:817-825. [PMID: 36973156 DOI: 10.1016/j.jsurg.2023.02.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Revised: 01/31/2023] [Accepted: 02/23/2023] [Indexed: 05/23/2023]
Abstract
OBJECTIVE Improvements to the medical student surgical learning environment are limited by lack of granular data and recall bias on end-of-clerkship evaluations. The purpose of this study was to identify specific areas for intervention using a novel real-time mobile application. DESIGN An application was designed to obtain real-time feedback from medical students regarding the learning environment on their surgical clerkship. Thematic analysis of student experiences was performed at the conclusion of 4 consecutive 12-week rotation blocks. SETTING Brigham and Women's Hospital/Harvard Medical School, Boston, Massachusetts. RESULTS Fifty-four medical students at a single institution were asked to participate during their primary clerkship experience. Students submitted 365 responses over 48 weeks. Multiple themes emerged which were dichotomized into positive and negative emotions centered on specific student priorities. Approximately half of responses were associated with positive emotions (52.9%) and half with negative emotions (47.1%). Student priorities included the desire to feel included in the surgical team (resulting in feeling engaged/ignored), to have a positive relationship with members of the team (perceiving kind/rude interactions), to witness compassionate patient care (observing empathy/disrespect for patients), to have a well-planned surgical rotation (experiencing organization/disorganization within teams), and to feel that student well-being is prioritized (reporting opportunities/disregard for student wellness). CONCLUSION A novel, user-friendly mobile application identified several areas to improve the experience and engagement of students on their surgery clerkship. Allowing clerkship directors and other educational leaders to collect longitudinal data in real time may allow for more targeted, timely improvements to the medical student surgical learning environment.
Collapse
Affiliation(s)
- Rachel B Atkinson
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | | | - Douglas S Smink
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Reza Askari
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Andrea L Pusic
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Nancy L Cho
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Jamie M Robertson
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Erika L Rangel
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts.
| |
Collapse
|
8
|
Neal CJ, Durning SJ, Dharmapurikar R, McDaniel KE, Lad SP, Haglund MM. From Their Eyes: What Constitutes Quality Formative Written Feedback for Neurosurgery Residents. JOURNAL OF SURGICAL EDUCATION 2023; 80:323-330. [PMID: 36280588 DOI: 10.1016/j.jsurg.2022.10.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 08/06/2022] [Accepted: 10/05/2022] [Indexed: 06/16/2023]
Abstract
BACKGROUND The characteristics of quality feedback from the neurosurgery resident's perspective are not fully elucidated. The Surgical Autonomy Program is an intraoperative assessment tool based on Vygotsky's Zone of Proximal Development (ZPD). SAP facilitates assessment of a resident's operative performance accompanied by written feedback. OBJECTIVE The goal of this study was twofold: to identify themes from the written feedback of SAP operative assessments and to examine if these themes influenced the neurosurgery residents' perception of feedback quality. METHODS In 2021, SAP data from 2019-2021 at two neurosurgery programs were reviewed. Feedback quality from the SAP was determined by the resident at the time of their assessment. Using a constant comparative technique, the feedback was coded using a thematic analysis. The quality of feedback within each code was analyzed. RESULTS There were 2968 SAP entries evaluated. When the ZPD concept was fully used, residents reported high quality feedback 91.4% of the time compared to 58.6% when ZPD was not used (p < 0.001). Qualitative analysis of the written feedback revealed five themes: Non-Specific, Specific General Observations, Key Points, Next Steps, and Independent Practice. Feedback in the Specific General Observations, Key Points, and Independent Practice categories were associated with higher level feedback than leaving the space blank (p < 0.001) or writing Non-Specific comments (p < 0.001). CONCLUSIONS Providing comments that discuss the resident's specific performance in the case, key learning points, or their progress towards independence, results in high quality feedback. Utilizing a theory-based tool such as the SAP can provide meaningful feedback to neurosurgical residents.
Collapse
Affiliation(s)
- Chris J Neal
- Division of Neurosurgery, Walter Reed National Military Medical Center, Bethesda MD, Uniformed Services University, Bethesda, Maryland.
| | - Steven J Durning
- Center for Health Professions Education, Department of Medicine, Uniformed Services University, Bethesda, Maryland
| | | | | | - Shivanand P Lad
- Department of Neurosurgery, Duke University, Durham, North Carolina
| | | |
Collapse
|
9
|
Oberoi KPS, Caine AD, Schwartzman J, Rab S, Turner AL, Merchant AM, Kunac A. A Resident-Driven Mobile Evaluation System Can Be Used to Augment Traditional Surgery Rotation Evaluations. Am Surg 2023; 89:137-144. [PMID: 33881951 DOI: 10.1177/00031348211011130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
BACKGROUND The Accreditation Council for Graduate Medical Education requires residents to receive milestone-based evaluations in key areas. Shortcomings of the traditional evaluation system (TES) are a low completion rate and delay in completion. We hypothesized that adoption of a mobile evaluation system (MES) would increase the number of evaluations completed and improve their timeliness. METHODS Traditional evaluations for a general surgery residency program were converted into a web-based form via a widely available, free, and secure application and implemented in August 2017. After 8 months, MES data were analyzed and compared to that of our TES. RESULTS 122 mobile evaluations were completed; 20% were solicited by residents. Introduction of the MES resulted in an increased number of evaluations per resident (P = .0028) and proportion of faculty completing evaluations (P = .0220). Timeliness also improved, with 71% of evaluations being completed during one's clinical rotation. CONCLUSIONS A resident-driven MES is an inexpensive and effective method to augment traditional end-of-rotation evaluations.
Collapse
Affiliation(s)
| | - Akia D Caine
- Department of Surgery, 12286Rutgers - New Jersey Medical School, Newark, NJ, USA
| | - Jacob Schwartzman
- Department of Surgery, 12286Rutgers - New Jersey Medical School, Newark, NJ, USA
| | - Sayeeda Rab
- Department of Surgery, 12286Rutgers - New Jersey Medical School, Newark, NJ, USA
| | - Amber L Turner
- Department of Surgery, 24056Saint Barnabas Medical Center, Livingston, NJ, USA
| | - Aziz M Merchant
- Division of General Surgery, Department of Surgery, 12286Rutgers - New Jersey Medical School, Newark, NJ, USA
| | - Anastasia Kunac
- Division of Trauma and Surgical Critical Care, Department of Surgery, 12286Rutgers - New Jersey Medical School, Newark, NJ, USA
| |
Collapse
|
10
|
Lillemoe HA, Hanna DN, Baregamian N, Solórzano CC, Terhune KP, Geevarghese SK, Kiernan CM. The use of an educational time-out in thyroid and parathyroid surgery to move the needle in periprocedural education. Surgery 2023; 173:84-92. [PMID: 36216620 DOI: 10.1016/j.surg.2022.07.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2022] [Revised: 05/19/2022] [Accepted: 07/19/2022] [Indexed: 12/13/2022]
Abstract
BACKGROUND As surgical training shifts toward a competency-based paradigm, deliberate practice for procedures must be a point of focus. The purpose of this study was to assess the impact of an educational time-out intervention on educational experience and operative performance in endocrine surgery. METHODS For 12 months, third-year general surgery residents used the educational time-out to establish an operative step of focus for thyroidectomy and parathyroidectomy procedures. Data were collected using the System for Improving and Measuring Procedural Learning application and post-rotation surveys. The Zwisch scale was used to classify supervision, with meaningful autonomy defined as passive help or supervision only. RESULTS Eight residents and 3 attending surgeons performed the educational time-out for a total of 211 operations (93% completion rate). At the end of each rotation, there was improvement in the frequency of goal setting. There was strong agreement (90%) that the intervention strengthened the educational experience. For most cases (52%), the residents were rated at active help. Residents performed a median of 3/6 thyroidectomy steps at meaningful autonomy and a median of 2/5 parathyroidectomy steps at meaningful autonomy. Review of the qualitative data revealed that optimal feedback was provided in 46% of cases. CONCLUSION The educational time-out strengthened educational experiences. Stepwise procedural data revealed the varying levels of supervision that exist within an operation. Broader implementation of this intervention could facilitate competency-based procedural education.
Collapse
Affiliation(s)
- Heather A Lillemoe
- Department of Surgery, Vanderbilt University Medical Center, Nashville, TN; Department of Surgical Oncology, The University of Texas at MD Anderson Cancer Center, Houston, TX.
| | - David N Hanna
- Department of Surgery, Vanderbilt University Medical Center, Nashville, TN
| | - Naira Baregamian
- Department of Surgery, Vanderbilt University Medical Center, Nashville, TN
| | - Carmen C Solórzano
- Department of Surgery, Vanderbilt University Medical Center, Nashville, TN
| | - Kyla P Terhune
- Department of Surgery, Vanderbilt University Medical Center, Nashville, TN
| | | | - Colleen M Kiernan
- Department of Surgery, Vanderbilt University Medical Center, Nashville, TN
| |
Collapse
|
11
|
Best Practices for Remediation in Pulmonary and Critical Care Medicine Fellowship Training. ATS Sch 2022; 3:485-500. [PMID: 36312805 PMCID: PMC9590524 DOI: 10.34197/ats-scholar.2022-0007re] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Accepted: 06/03/2022] [Indexed: 12/03/2022] Open
Abstract
Background Remediation of struggling learners in pulmonary and critical care fellowship
programs is a challenge, even for experienced medical educators. Objective This evidence-based narrative review provides a framework program leaders may
use to address fellows having difficulty achieving competency during
fellowship training. Methods The relevant evidence for approaches on the basis of each learner’s
needs is reviewed and interpreted in the context of fellowship training in
pulmonary medicine and critical care. Issues addressed include bias in
fellow assessments and remediation, the impacts of the severe acute
respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the specific
challenges of pulmonary and critical care fellowship programs, a brief
review of relevant legal issues, guidance on building and leveraging program
resources, and a discussion of learner outcomes. Results This results in a concise, evidence-based toolkit for program leaders based
around four pillars: early identification, fellow assessment, collaborative
intervention, and reassessment. Important concepts also include the need for
documentation, clear and written communication, and fellow-directed
approaches to the creation of achievable goals. Conclusion Evidence-based remediation helps struggling learners in pulmonary and
critical care fellowship to improve their ability to meet Accreditation
Council for Graduate Medical Education (ACGME) milestones.
Collapse
|
12
|
Miles VP, Schroll RW, Beaty JS. Empowering the Resident Learner. JOURNAL OF SURGICAL EDUCATION 2022; 79:909-917. [PMID: 35490137 DOI: 10.1016/j.jsurg.2022.03.009] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/06/2021] [Revised: 03/07/2022] [Accepted: 03/22/2022] [Indexed: 06/14/2023]
Abstract
Surgical resident autonomy is an essential element of the transition from supervised training to independent practice. However, in an age of duty hour restrictions, legislative constraints, increased litigation, and heightened societal expectations, training an autonomous resident proves increasingly difficult. To tackle these barriers to training successful surgeons, the American College of Surgeons Committee on Resident Education initiated the Resident Mentored Autonomy Project. As a subdivision of this project, the Empowered Learner research team here presents a framework for training the self-directed and empowered surgical resident learner. There are many strategies by which surgical faculty, program directors and chairs, and residents themselves may engage to improve resident operative autonomy.
Collapse
Affiliation(s)
- Victoria P Miles
- Department of Surgery, University of Tennessee College of Medicine Chattanooga, Chattanooga, Tennessee.
| | - Rebecca W Schroll
- Department of Surgery, Tulane University School of Medicine, New Orleans, Louisiana
| | | |
Collapse
|
13
|
Ahle SL, Eskender M, Schuller M, Carnes E, Chen X, Koehler J, Willey G, Latif A, Doyle J, Wnuk G, Fryer JP, Mellinger JD, George BC. The Quality of Operative Performance Narrative Feedback: A Retrospective Data Comparison Between End of Rotation Evaluations and Workplace-based Assessments. Ann Surg 2022; 275:617-620. [PMID: 32511125 DOI: 10.1097/sla.0000000000003907] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
OBJECTIVE To describe the quality of operative performance feedback using evaluation tools commonly used by general surgery residency training programs. SUMMARY OF BACKGROUND DATA The majority of surgical training programs administer an evaluation through which faculty members may rate and comment on trainee operative performance at the end of the rotation (EOR). Many programs have also implemented the system for improving and measuring procedural learning (SIMPL), a workplace-based assessment tool with which faculty can rate and comment on a trainee's operative performance immediately after a case. It is unknown how the quality of narrative operative performance feedback delivered with these tools compares. METHODS The authors collected EOR evaluations and SIMPL narrative comments on trainees' operative performance from 3 university-based surgery training programs during the 2016-2017 academic year. Two surgeon raters categorized comments relating to operative skills as being specific or general and as encouraging and/or corrective. Comments were then classified as effective, mediocre, ineffective, or irrelevant. The frequencies with which comments were rated as effective were compared using Chi-square analysis. RESULTS The authors analyzed a total of 600 comments. 10.7% of EOR and 58.3% of SIMPL operative performance evaluation comments were deemed effective (P < 0.0001). CONCLUSIONS Evaluators give significantly higher quality operative performance feedback when using workplace-based assessment tools rather than EOR evaluations.
Collapse
Affiliation(s)
- Samantha L Ahle
- Department of Surgery, Yale School of Medicine, New Haven, CT
| | - Mickyas Eskender
- Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL
| | - Mary Schuller
- Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL
| | - Emily Carnes
- Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL
| | - Xilin Chen
- Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, MI
| | - Jeanne Koehler
- Department of Medical Education, Southern Illinois University School of Medicine, Springfield, IL
| | | | - Ahmed Latif
- Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, MI
| | - Jennifer Doyle
- Department of Surgery, Massachusetts General Hospital, Boston MA
| | - Gregory Wnuk
- Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, MI
| | - Jonathan P Fryer
- Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL
| | - John D Mellinger
- Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL
| | - Brian C George
- Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, MI
| |
Collapse
|
14
|
Chen JX, Miller LE, Filimonov A, Shuman EA, Marchiano E, George BC, Thorne M, Pletcher SD, Platt M, Teng M, Kozin ED, Gray ST. Factors affecting operative autonomy and performance during otolaryngology training: A multicenter trial. Laryngoscope Investig Otolaryngol 2022; 7:404-408. [PMID: 35434323 PMCID: PMC9008171 DOI: 10.1002/lio2.750] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Accepted: 01/20/2022] [Indexed: 11/05/2022] Open
Abstract
Objective Surgical education is changing in an era of new regulations and evolving training cultures. We sought to understand the factors that affect operative experiences during otolaryngology residency. Methods From December 2019 to December 2020, five otolaryngology training programs used the SIMPL OR smartphone application to evaluate residents after each operation. Residents and attendings rated the trainee's autonomy on a 4‐level Zwisch scale, performance on a 5‐level scale, and case complexity on a 3‐level scale. We examined associations between ratings of autonomy and performance with variables including postgraduate year (PGY), case complexity, gender, week of the academic year (AY), and whether multiple procedures were logged. Results 78 attendings and 92 residents logged 2984 evaluations. PGY level and week of the AY were positively associated with attending ratings of autonomy and performance (PGY3 vs. PGY2: B = 0.63, p < .001 for autonomy and B = 1.05, p < .001 for performance; week of the AY: B = 0.013, p = .002 for autonomy; B = 0.025, p < .001 for performance). Multiple procedures logged and increasing case complexity were negatively associated with attending ratings (multiple procedures: B = −0.19, p = .04 for autonomy and B = −0.48, p < .001 for performance; hardest vs. easiest 1/3 of cases: B = −1.01, p < .001 for autonomy and B = −0.59, p < .001 for performance). Attending and trainee genders were not associated with attending ratings of autonomy or performance. Conclusion Resident autonomy and performance were positively associated with PGY level and week of the academic year, and negatively associated with case complexity and multiple procedures. These findings highlight the need to align training level with case complexity to promote quality operative experiences. Level of Evidence 2.
Collapse
Affiliation(s)
- Jenny X. Chen
- Department of Otolaryngology – Head and Neck Surgery Massachusetts Eye and Ear Boston Massachusetts USA
- Department of Otolaryngology – Head and Neck Surgery Harvard Medical School Boston Massachusetts USA
| | - Lauren E. Miller
- Department of Otolaryngology – Head and Neck Surgery Massachusetts Eye and Ear Boston Massachusetts USA
- Department of Otolaryngology – Head and Neck Surgery Harvard Medical School Boston Massachusetts USA
| | - Andrey Filimonov
- Department of Otolaryngology – Head and Neck Surgery Mount Sinai Hospital New York City New York USA
| | - Elizabeth A. Shuman
- Department of Otolaryngology – Head and Neck Surgery University of California San Francisco San Francisco California USA
| | - Emily Marchiano
- Department of Otolaryngology – Head and Neck Surgery Michigan Medicine Ann Arbor Michigan USA
| | - Brian C. George
- Center for Surgical Training and Research, Department of Surgery Michigan Medicine Ann Arbor Michigan USA
| | - Marc Thorne
- Department of Otolaryngology – Head and Neck Surgery Michigan Medicine Ann Arbor Michigan USA
| | - Steven D. Pletcher
- Department of Otolaryngology – Head and Neck Surgery University of California San Francisco San Francisco California USA
| | - Michael Platt
- Department of Otolaryngology – Head and Neck Surgery Boston University Medical Center Boston Massachusetts USA
| | - Marita Teng
- Department of Otolaryngology – Head and Neck Surgery Mount Sinai Hospital New York City New York USA
| | - Elliott D. Kozin
- Department of Otolaryngology – Head and Neck Surgery Massachusetts Eye and Ear Boston Massachusetts USA
- Department of Otolaryngology – Head and Neck Surgery Harvard Medical School Boston Massachusetts USA
| | - Stacey T. Gray
- Department of Otolaryngology – Head and Neck Surgery Massachusetts Eye and Ear Boston Massachusetts USA
- Department of Otolaryngology – Head and Neck Surgery Harvard Medical School Boston Massachusetts USA
| |
Collapse
|
15
|
Chen JX, Deng F, Filimonov A, Shuman EA, Marchiano E, George BC, Thorne M, Pletcher SD, Platt M, Teng MS, Kozin ED, Gray ST. Multi-institutional Study of Otolaryngology Resident Intraoperative Experiences for Key Indicator Procedures. Otolaryngol Head Neck Surg 2021; 167:268-273. [PMID: 34609936 DOI: 10.1177/01945998211050350] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
OBJECTIVE There is concern that current otolaryngology residents may not receive adequate surgical training. We aimed to characterize residents' surgical experiences at 5 academic centers performing the 14 key indicator procedures (KIPs) outlined by the Accreditation Council for Graduate Medical Education. STUDY DESIGN Prospective study. SETTING Five otolaryngology training programs. METHODS Data were gathered from December 2019 to December 2020 with a smartphone application from the Society for Improving Medical Professional Learning. After each operation, residents and faculty rated trainee autonomy on a 4-level Zwisch scale and performance on a 5-level modified Dreyfus scale. RESULTS Residents and attendings (n = 92 and 78, respectively) logged 2984 evaluations. Attending ratings of resident autonomy and performance increased with training level (P < .001). Resident self-assessments of autonomy and performance were lower than paired attending assessments (P < .001). Among attending evaluations of KIPs performed by senior residents (postgraduate year 4 or 5), 55% of cases were performed with meaningful autonomy (passive help or supervision only). Similarly, attendings rated 55% of these cases as a practice-ready or exceptional performance. Senior residents had meaningful autonomy for ≥50% of cases for most KIPs, with the exception of flaps and grafts (40%), pediatric/adult airway (39%), and stapedectomy/ossiculoplasty (33%). Similarly, senior residents received practice-ready or exceptional performance ratings for ≥50% of cases across all KIPs other than pediatric/adult airway (42%) and stapedectomy/ossiculoplasty (33%). CONCLUSION In this multicenter study, resident surgical autonomy and performance varied across otolaryngology KIPs. The development of nationwide benchmarks will help programs and residents set educational goals. LEVEL OF EVIDENCE 2.
Collapse
Affiliation(s)
- Jenny X Chen
- Department of Otolaryngology-Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA.,Department of Otolaryngology-Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| | - Francis Deng
- Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Andrey Filimonov
- Department of Otolaryngology-Head and Neck Surgery, Mount Sinai Hospital, New York City, New York, USA
| | - Elizabeth A Shuman
- Department of Otolaryngology-Head and Neck Surgery, University of California San Francisco, San Francisco, California, USA
| | - Emily Marchiano
- Department of Otolaryngology-Head and Neck Surgery, Michigan Medicine, Ann Arbor, Michigan, USA
| | - Brian C George
- Center for Surgical Training and Research, Department of Surgery, Michigan Medicine, Ann Arbor, Michigan, USA
| | - Marc Thorne
- Department of Otolaryngology-Head and Neck Surgery, Michigan Medicine, Ann Arbor, Michigan, USA
| | - Steven D Pletcher
- Department of Otolaryngology-Head and Neck Surgery, University of California San Francisco, San Francisco, California, USA
| | - Michael Platt
- Department of Otolaryngology-Head and Neck Surgery, Boston University Medical Center, Boston, Massachusetts, USA
| | - Marita S Teng
- Department of Otolaryngology-Head and Neck Surgery, Mount Sinai Hospital, New York City, New York, USA
| | - Elliott D Kozin
- Department of Otolaryngology-Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA.,Department of Otolaryngology-Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| | - Stacey T Gray
- Department of Otolaryngology-Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA.,Department of Otolaryngology-Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
16
|
Evaluating chief resident readiness for the teaching assistant role: The Teaching Evaluation assessment of the chief resident (TEACh-R) instrument. Am J Surg 2021; 222:1112-1119. [PMID: 34600735 DOI: 10.1016/j.amjsurg.2021.09.026] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2021] [Revised: 09/23/2021] [Accepted: 09/24/2021] [Indexed: 11/21/2022]
Abstract
BACKGROUND The American Board of Surgery has mandated chief residents complete 25 cases in the teaching assistant (TA) role. We developed a structured instrument, the Teaching Evaluation and Assessment of the Chief Resident (TEACh-R), to determine readiness and provide feedback for residents in this role. METHODS Senior (PGY3-5) residents were scored on technical and teaching performance by faculty observers using the TEACh-R instrument in the simulation lab. Residents were provided with their TEACh-R scores and surveyed on their experience. RESULTS Scores in technical (p < 0.01) and teaching (p < 0.01) domains increased with PGY. Higher technical, but not teaching, scores correlated with attending-rated readiness for operative independence (p 0.02). Autonomy mismatch was inversely correlated with teaching competence (p < 0.01). Residents reported satisfaction with TEACh-R feedback and desire for use of this instrument in operating room settings. CONCLUSION Our TEACh-R instrument is an effective way to assess technical and teaching performance in the TA role.
Collapse
|
17
|
Valeriano A, Kim A, Katsoulas E, Sanfilippo A, Wang L, Rajaram A. Perspectives of Recent Graduates on Clerkship Procedural Skill Training at a Canadian Medical School: an Exploratory Study. MEDICAL SCIENCE EDUCATOR 2021; 31:1361-1367. [PMID: 34457978 PMCID: PMC8368422 DOI: 10.1007/s40670-021-01313-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/11/2021] [Indexed: 06/13/2023]
Abstract
The implementation of competency-based medical education in Canada has presented both unique opportunities and challenges for improving undergraduate procedural skills curricula. Despite the recognized importance of procedural skills, there remains a lack of national congruency in procedural training across medical schools that must be addressed. When undertaking such curricular development, obtaining learner feedback is a crucial step that can facilitate practical changes and address disparities. The purpose of the current study is to explore the perspectives and insights of recent medical graduates surrounding the clerkship procedural skills curriculum at a Canadian medical school. Six residents from a variety of program specialties participated in a semi-structured focus group interview discussing key aspects of procedural skill training. The focus group was later transcribed and qualitatively analyzed for themes. The results highlight barriers to competency-based procedural skill training involving time constraints and obtaining required evaluations, and the ability of students to self-advocate for learning opportunities. Participants note few opportunities to practice nasogastric tube insertion and casting in particular. Recommendations for curricular improvement are discussed, including options for curricular remediation and resident perspectives on which procedural skills undergraduate trainees should achieve competency in by graduation.
Collapse
Affiliation(s)
| | - Andrew Kim
- School of Medicine, Queen’s University, Kingston, ON Canada
| | | | - Anthony Sanfilippo
- School of Medicine, Queen’s University, Kingston, ON Canada
- Department of Medicine, Queen’s University, Kingston, ON Canada
| | - Louie Wang
- School of Medicine, Queen’s University, Kingston, ON Canada
- Department of Anesthesiology and Perioperative Medicine, Queen’s University, Kingston, ON Canada
| | - Akshay Rajaram
- Department of Family Medicine, Queen’s University, Kingston, ON Canada
| |
Collapse
|
18
|
Abbott KL, Kwakye G, Kim GJ, Luckoski JL, Krumm AE, Clark M, Chen X, Gupta T, Weiser TG, George BC. US general surgical trainee performance for representative global surgery procedures. Am J Surg 2021; 223:224-228. [PMID: 34119330 DOI: 10.1016/j.amjsurg.2021.05.016] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2021] [Revised: 05/06/2021] [Accepted: 05/31/2021] [Indexed: 01/08/2023]
Abstract
BACKGROUND Many US general surgery residents are interested in global surgery, but their competence with key procedures is unknown. METHODS Using a registry managed by the Society for Improving Medical Professional Learning (SIMPL), we extracted longitudinal operative performance ratings data for a national cohort of US general surgery residents. Operative performance at the time of graduation was estimated via a Bayesian generalized linear mixed model. RESULTS Operative performance ratings for 12,976 procedures performed by 1584 residents in 52 general surgery programs were analyzed. These spanned 17 of 31 (55%) procedures deemed important for global surgical practice. For these procedures, the probability of a graduating resident being deemed competent to perform a procedure was 0.95 (95% confidence interval 0.86-1.00) but was less than 0.9 for 3 observed procedures. CONCLUSION Our results highlight gaps in the preparedness of US general surgery trainees to perform procedures deemed most important for global surgery settings.
Collapse
Affiliation(s)
- Kenneth L Abbott
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA; University of Michigan Medical School, Ann Arbor, MI, USA
| | - Gifty Kwakye
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA
| | - Grace J Kim
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA
| | - John L Luckoski
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA
| | - Andrew E Krumm
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA; Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI, USA
| | - Michael Clark
- Consulting for Statistics, Computing, and Analytics Research, University of Michigan, Ann Arbor, MI, USA
| | - Xilin Chen
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA
| | - Tanvi Gupta
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA
| | - Thomas G Weiser
- Stanford University School of Medicine, Department of Surgery, Division of General Surgery, Section of Trauma & Critical Care, Stanford, CA, USA; Department of Clinical Surgery, University of Edinburgh, Edinburgh, UK
| | - Brian C George
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, MI, USA.
| | | |
Collapse
|
19
|
Abstract
OBJECTIVE The aim of this study was to propose an evidence-based blueprint for training, assessment, and certification of operative performance for surgical trainees. SUMMARY BACKGROUND DATA Operative skill is a critical aspect of surgical performance. High-quality assessment of operative skill therefore has profound implications for training, accreditation, certification, and the public trust of the profession. Current methods of operative skill assessment for surgeons rely heavily on global assessment strategies across a very broad domain of procedures. There is no mechanism to assure technical competence for individual procedures. The science and scalability of operative skill assessment has progressed significantly in recent decades, and can inform a much more meaningful strategy for competency-based assessment of operative skill than has been previously achieved. METHODS The present article reviews the current status and science of operative skill assessment and proposes a template for competency-based assessment which could be used to update training, accreditation, and certification processes. The proposal is made in reference to general surgery but is more generally applicable to other procedural specialties. RESULTS Streamlined, routine assessment of every procedure performed by surgical trainees is feasible and would enable a more competency-based educational paradigm. In light of the constraints imposed by both clinical volume and assessment bias, trainees should be expected to become proficient and be measured against a mastery learning standard only for the most important and highest-frequency procedures. For less frequently observed procedures, performance can be compared to a norm-referenced standard and, to provide an overall trajectory of performance, analyzed in aggregate. Key factors in implementing this approach are the number of evaluations, the number of raters, the timeliness of evaluation, and evaluation items. CONCLUSIONS A competency-based operative skill assessment can be incorporated into surgical training, assessment, and certification. The time has come to develop a systematic approach to this issue as a means of demonstrating professional standards worthy of the public trust.
Collapse
|
20
|
Chen XP, Cochran A, Harzman AE, Ellison EC. Predicting prospective resident entrustment: From evaluation to action. Am J Surg 2021; 222:536-540. [PMID: 33485620 DOI: 10.1016/j.amjsurg.2021.01.020] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2020] [Revised: 12/27/2020] [Accepted: 01/11/2021] [Indexed: 11/18/2022]
Abstract
OBJECTIVE We aimed to identify potential variables predictive of a resident achieving faculty future entrustment as a way to enhance attending surgeons' planning of teaching in the operating room leading to improved resident operative autonomy in practice. METHODS We reviewed 273 resident performance evaluations from 91 surgical cases that were collected from 11 general surgery chief residents and 16 attending surgeons between April 2018 and June 2019 using a validated evaluation instrument. The primary outcome measure was prospective resident entrustment estimated by the rater for future similar cases. We used descriptive statistics and the boosted tree analysis model to find potential predictors for the outcome measure and examine test-retest reliability by procedure. RESULTS Step-specific guidance (r = 0.77, p < 0.0001) was the variable most highly associated with prospective resident entrustment in bivariate linear analysis. The boosted tree analysis demonstrated step-specific guidance was the strongest predictor for prospective resident entrustment in the OR, and its predictive importance was much higher than the overall guidance (0.64 > 0.18). Test-retest reliability was from 0.93 to 0.98 across procedures, indicating the likelihood that attending surgeons granted future autonomy complied with their evaluation of prospective resident entrustment was high. CONCLUSIONS By assessing step-specific guidance, attending surgeons can reliably judge residents' future entrustment and potentially better plan for operative teaching/supervision that may lead to granting a surgical resident operative autonomy on similar cases in the future. Our findings provide insight into prospective faculty development of surgical teaching aimed at improving resident readiness for independent practice.
Collapse
|
21
|
Naples R, French JC, Thomas JD, Khandelwal C, Rosen MJ, Lipman JM. Utilization of a quality reporting system to increase faculty participation in resident operative assessment. Surgery 2020; 169:483-487. [PMID: 33328137 DOI: 10.1016/j.surg.2020.11.014] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2020] [Revised: 10/07/2020] [Accepted: 11/10/2020] [Indexed: 11/29/2022]
Abstract
BACKGROUND A quality collaborative across our hospital system was initiated to track surgical outcomes. We sought to determine whether incorporating a resident operative performance assessment into this quality collaborative would increase the quantity and quality of these assessments and impact relevant milestones. METHODS A resident operative assessment was added to a quality reporting system required to be completed by faculty at the completion of 2 operations. Three milestones directly related to operative performance were analyzed-Patient Care 3, Medical Knowledge 2, and Interpersonal and Communication Skills 3. Residents were divided in 2 groups: quality collaborative (≥10 operative assessments) and no quality collaborative (<10 operative assessments). Milestones from Spring 2019 and Fall 2019 were analyzed. RESULTS Faculty participation was 86% with 407 assessments completed from February to October 2019. A difference in the rate of change in resident performance for Patient Care 3 (+0.95 vs +0.55; P = .04) and Interpersonal and Communication Skills 3 (+1.05 vs +0.52; P = .02) was observed for those residents in the quality collaborative group (n = 20) compared with baseline data. CONCLUSION Addition of an operative assessment to a mandatory quality collaborative increases faculty participation and impacts resident milestone determination. These findings highlight opportunities to find innovative and efficient methods to improve faculty engagement.
Collapse
Affiliation(s)
- Robert Naples
- Department of General Surgery, Cleveland Clinic, Cleveland, OH.
| | - Judith C French
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| | - Jonah D Thomas
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| | | | - Michael J Rosen
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| | - Jeremy M Lipman
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| |
Collapse
|
22
|
Albright JB, Meier AH, Ruangvoravat L, VanderMeer TJ. Association Between Entrustable Professional Activities and Milestones Evaluations: Real-time Assessments Correlate With Semiannual Reviews. JOURNAL OF SURGICAL EDUCATION 2020; 77:e220-e228. [PMID: 32747323 DOI: 10.1016/j.jsurg.2020.07.027] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/22/2020] [Revised: 06/09/2020] [Accepted: 07/16/2020] [Indexed: 06/11/2023]
Abstract
OBJECTIVE Entrustable professional activities (EPAs) have been developed to refine competency-based education. The American Board of Surgery has initiated a 2-year pilot study to evaluate the impact of EPAs on the evaluation and feedback of surgical residents. The ACGME Milestones in Surgery is a semiannual competency-based evaluation program to measure resident progression through 16 professional attributes across 8 practice domains. The correlation between these 2 evaluation tools remains unclear. The purpose of this study is to evaluate this correlation through comparison of an EPA with the corresponding elements of the ACGME Milestones. DESIGN From July, 2018 to October, 2019, all residents submitting EPA evaluations for gall bladder disease were evaluated for preoperative, intraoperative, and/or postoperative entrustability. The ratings were converted to a numerical rank from 0 to 4. Milestones scores from May 2019 and November 2019 were obtained for each resident, with scores ranging from 0 to 4. The gall bladder EPA incorporates the operative PC3 and MK2 and nonoperative PC1, PC2, and ICS3 components. Spearman rank correlation was conducted to evaluate the association between each resident's median EPA ranking and his/her milestones scores. SETTING SUNY Upstate Medical University, Syracuse, NY, a university-based hospital. PARTICIPANTS General surgery residents. RESULTS Among 24 residents, 106 intraoperative EPA evaluations were. For both the May and November milestones, significant positive correlations were noted for PC3 (correlation coefficient ρ = 0.690, p < 0.001; ρ = 0.876, p < 0.001). Similarly, for MK2, a significant positive correlation was noted (ρ = 0.882, p < 0.001; ρ = 0.759, p < 0.001). Interestingly, significant positive correlations were also identified between the 3 nonoperative milestones and the intraoperative entrustability ranking. CONCLUSIONS We observed significant correlations between EPAs for cholecystectomy and associated milestones evaluation scores. These findings indicate that EPAs may provide more timely and specific feedback than existing tools and, on aggregate, may improve upon existing formative feedback practices provided through the biannual evaluation of surgical residents.
Collapse
Affiliation(s)
- Jeffrey B Albright
- State University of New York Upstate Medical University, Department of Surgery, Syracuse, New York.
| | - Andreas H Meier
- State University of New York Upstate Medical University, Department of Surgery, Syracuse, New York
| | - Lucy Ruangvoravat
- Yale School of Medicine, Department of Surgery, New Haven, Connecticut
| | - Thomas J VanderMeer
- State University of New York Upstate Medical University, Department of Surgery, Syracuse, New York
| |
Collapse
|
23
|
Nicolas JD, Huang R, Teitelbaum EN, Bilimoria KY, Hu YY. Constructing Learning Curves to Benchmark Operative Performance of General Surgery Residents Against a National Cohort of Peers. JOURNAL OF SURGICAL EDUCATION 2020; 77:e94-e102. [PMID: 33109492 DOI: 10.1016/j.jsurg.2020.10.001] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/17/2020] [Revised: 08/27/2020] [Accepted: 10/02/2020] [Indexed: 06/11/2023]
Abstract
OBJECTIVE No method or data exist to allow surgical trainees or their programs to contextualize their technical progress. The objective of this study was to create peer benchmarks for Cumulative Sum (CUSUM) charts based upon operative evaluations from a national cohort of general surgery residents. DESIGN, SETTING, PARTICIPANTS In 2016-2018, faculty from 26 general surgery residency programs nationwide rated 328 residents' operative performance on a case-by-case basis using a validated 5-point Likert scale. An individual case was considered a "misstep" if scoring below the national median score for that procedure in that postgraduate year (PGY). We constructed 2-sided observed-expected CUSUM charts to capture each resident's cumulative performance over time relative to the national medians. Upper (failure) and lower (positive outlier) benchmarks were established based on the PGY-specific 75th percentile and median misstep rates; consistent/repeated missteps are reflected by crossing of the upper boundary. Procedures with ≤10 observations and residents who were evaluated <10 times for each PGY were excluded. RESULTS Around 8,161 evaluations on 76 procedure types were analyzed. The individual misstep rate was lowest among PGY-3s at 13.3% and highest among PGY-4s at 28.6%. No interns had curves that crossed the failure boundary. 8.7% of PGY-2s and 8.9% of PGY-3s finished the year past the failure boundary. PGY-2s had the most positive outliers, with 28.3% of them demonstrating an outlying success performance beyond the lower boundary for at least once. PGY-5s most frequently failed, with 16.7% ever crossing the upper boundary and 11.1% remaining above it at graduation. CONCLUSIONS CUSUM is a valid statistical approach for benchmarking individual residents' operative performance against national peers as they progress through the year in real-time. With further validation, CUSUM could be used to set progression and/or graduation standards and objectively identify residents who might benefit from remediation.
Collapse
Affiliation(s)
- Joseph D Nicolas
- Surgical Outcomes and Quality Improvement Center (SOQIC), Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Reiping Huang
- Surgical Outcomes and Quality Improvement Center (SOQIC), Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Ezra N Teitelbaum
- Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Karl Y Bilimoria
- Surgical Outcomes and Quality Improvement Center (SOQIC), Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois; Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Yue-Yung Hu
- Surgical Outcomes and Quality Improvement Center (SOQIC), Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois; Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago, Illinois; Division of Pediatric Surgery, Ann and Robert H. Lurie Children's Hospital, Chicago, Illinois.
| |
Collapse
|
24
|
Koduri S, Altshuler DB, Khalsa SSS, Maher CO, Wnuk G, Tong D, George BC, Szerlip N. Using a Mobile Application for Evaluation of Procedural Learning in Neurosurgery. World Neurosurg 2020; 138:e124-e150. [DOI: 10.1016/j.wneu.2020.02.049] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2019] [Revised: 02/06/2020] [Accepted: 02/07/2020] [Indexed: 11/16/2022]
|
25
|
Crannell WC, Brasel KJ. Dealing with the struggling learner. Surgery 2020; 167:523-527. [DOI: 10.1016/j.surg.2019.06.013] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2019] [Revised: 05/21/2019] [Accepted: 06/04/2019] [Indexed: 11/17/2022]
|
26
|
Lillemoe HA, Stonko DP, George BC, Schuller MC, Fryer JP, Sullivan ME, Terhune KP, Geevarghese SK. A Preoperative Educational Time-Out is Associated with Improved Resident Goal Setting and Strengthens Educational Experiences. JOURNAL OF SURGICAL EDUCATION 2020; 77:18-26. [PMID: 31327734 DOI: 10.1016/j.jsurg.2019.07.004] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/30/2019] [Revised: 06/15/2019] [Accepted: 07/06/2019] [Indexed: 06/10/2023]
Abstract
OBJECTIVE The purpose of this study was to assess the impact of a preoperative Educational Time-Out (ETO) with structured postoperative feedback on resident preoperative goal-setting and the educational experience of a clinical rotation. DESIGN A preoperative ETO was developed during which trainees and faculty jointly identified an operative goal and discussed the trainee's operative autonomy. Postoperative feedback with a smartphone application was encouraged. From November 2016 to October 2017, the intervention was piloted with 1 surgical service. Outcomes included ETO completion rate, goal setting rate, and subjects' perception of the impact of the ETO on identification of performance deficits, trainee autonomy, and receipt of feedback. Data were analyzed using descriptive statistics. SETTING This study was performed in an institutional hospital setting. PARTICIPANTS Third-year general surgery residents and surgical faculty in the Department of Hepatobiliary Surgery and Liver Transplantation at Vanderbilt University Medical Center took part in the intervention. RESULTS Seven residents and 7 attending surgeons participated in this study. Residents performed a median of 15 procurements during an average of 6.5 weeks each on service. The ETO completion rate was 83%. Resident-reported preoperative goal setting increased after the intervention (from 36% to 78%, p = 0.015). Subjects reported a positive impact of the intervention, with high resident agreement that the ETO helped identify deficits (82% median agreement), increased autonomy (82% median agreement), and increased receipt of feedback (84% median agreement). Residents and attendings agreed that the educational experience was stronger due to the ETO (median 81% and 77%, respectively). CONCLUSIONS The ETO intervention improved rates of resident preoperative goal setting and strengthened perceived educational experiences. Resident participants also reported improvements in autonomy and rates of postoperative feedback. Broader implementation of this brief preoperative pause is an easy way to emphasize procedural education in the operating room.
Collapse
Affiliation(s)
- Heather A Lillemoe
- Department of Surgery, Vanderbilt University Medical Center, Nashville, Tennessee.
| | - David P Stonko
- Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Brian C George
- Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Mary C Schuller
- Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Jonathan P Fryer
- Department of Surgery, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Maura E Sullivan
- Department of Surgery, Keck School of Medicine at the University of Southern California, Los Angeles, California
| | - Kyla P Terhune
- Department of Surgery, Vanderbilt University Medical Center, Nashville, Tennessee
| | - Sunil K Geevarghese
- Department of Surgery, Vanderbilt University Medical Center, Nashville, Tennessee
| |
Collapse
|
27
|
Assessment of Operative Autonomy and Readiness for Independent Practice Among Pediatric Surgery Fellows. J Pediatr Surg 2020; 55:117-121. [PMID: 31679774 DOI: 10.1016/j.jpedsurg.2019.09.061] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/14/2019] [Accepted: 09/29/2019] [Indexed: 11/23/2022]
Abstract
PURPOSE The purpose of this study was to evaluate the implementation of the System for Improving and Measuring Procedural Learning (SIMPL) "app" in a pediatric surgery training program. SIMPL is a novel workplace-based operative performance assessment platform which allows case by case assessment of trainees using three scales (autonomy, performance, and case complexity) and takes seconds to complete. METHODS A pediatric surgery-specific procedural taxonomy for SIMPL was developed from the ACGME procedural library. SIMPL was piloted and implemented in a single pediatric surgery training program. A descriptive analysis of the operative assessments was undertaken. RESULTS In the year following SIMPL implementation, 565 (39% of cases) assessments were completed by 21 faculty for 3 fellows for 148 unique procedures within a median of 8.5 h. Recorded feedback accompanied 61% of assessments. 2nd year fellows were more likely to be deemed autonomous (81% ratings) and practice ready (84%) vs. 1st year fellows (50%; p < 0.001 both), with improvements over time. CONCLUSION Within pediatric surgery, this is the first implementation of a workplace-based operative performance assessment "app". With its ease of use, SIMPL drastically increased the volume of operative evaluations and diversified the mix of cases evaluated while SIMPL ratings demonstrated expected and graduated performance improvements over time. TYPE OF STUDY Education. LEVEL OF EVIDENCE N/A.
Collapse
|
28
|
Ji S, Hwang C, Karmakar M, Matusko N, Thompson-Burdine J, Williams AM, Leininger L, Minter RM, Sandhu G. Association of intraoperative entrustment with clinical competency amongst general surgery residents. Am J Surg 2019; 219:245-252. [PMID: 31870532 DOI: 10.1016/j.amjsurg.2019.12.012] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2019] [Revised: 11/06/2019] [Accepted: 12/05/2019] [Indexed: 10/25/2022]
Abstract
BACKGROUND Lack of transparency and meaningful assessment in surgical residency has led to inconsistent intraoperative entrustment and highly variable trainee competence at graduation. The relationship between faculty entrustment and resident entrustability on clinical competency remains unclear. We sought to evaluate the dynamic between entrustment/entrustability and clinical competency in general surgery residency. METHODS Intraoperative observations were conducted across a 22-month period at an academic tertiary center. Entrustment/entrustability were measured using OpTrust. Clinical competencies were appraised via ACGME Milestones and Objective Structured Assessment of Technical Skill (OSATS) scores. Mixed effects linear regression was used to investigate the relationship among overall ACGME Milestone scores, OSATS domain scores, and overall OpTrust scores. RESULTS Overall OpTrust scores significantly correlated with overall Milestone scores and multiple OSATS score domains. CONCLUSIONS OpTrust demonstrated a positive association between ACGME general surgery Milestones and OSATS scores. Overall, OpTrust may help optimize intraoperative faculty entrustment and resident entrustability, facilitating surgical trainee success during residency.
Collapse
Affiliation(s)
- Sunjong Ji
- University of Michigan Medical School, 7300 Medical Science Building I - A Wing, 1301 Catherine Street, Ann Arbor, MI, 48109, USA
| | - Charles Hwang
- University of Michigan Medical School, 7300 Medical Science Building I - A Wing, 1301 Catherine Street, Ann Arbor, MI, 48109, USA
| | - Monita Karmakar
- Department of Surgery, Michigan Medicine, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA
| | - Niki Matusko
- Department of Surgery, Michigan Medicine, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA
| | - Julie Thompson-Burdine
- Department of Surgery, Michigan Medicine, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA
| | - Aaron M Williams
- Department of Surgery, Michigan Medicine, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA
| | - Lisa Leininger
- Department of Surgery, Michigan Medicine, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA
| | - Rebecca M Minter
- Department of Surgery, University of Wisconsin, 600 Highland Avenue, Madison, WI, 53792, USA
| | - Gurjit Sandhu
- Department of Surgery, Michigan Medicine, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA; Department of Learning Health Sciences, Michigan Medicine, 209 Victor Vaughan Building, 1111 E. Catherine Street, Ann Arbor, MI, 48109, USA.
| |
Collapse
|
29
|
Ahle SL, Schuller M, Clark MJ, Williams RG, Wnuk G, Fryer JP, George BC. Do End-of-Rotation Evaluations Adequately Assess Readiness to Operate? ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2019; 94:1946-1952. [PMID: 31397708 DOI: 10.1097/acm.0000000000002936] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
PURPOSE Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based microassessments and (2) which factors most influence EOR evaluations and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method. METHOD In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment. RESULTS Ratings of operative performance from EOR evaluations vs workplace-based microassessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: Model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74). CONCLUSIONS EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared with other trainees in the same PGY of training. Workplace-based microassessments may be better for assessing a trainee's ability to perform specific procedures autonomously, thus perhaps providing more insight into a trainee's true readiness for operative independence.
Collapse
Affiliation(s)
- Samantha L Ahle
- S.L. Ahle is a general surgery resident, Yale School of Medicine, New Haven, Connecticut. M. Schuller is manager, Surgical Education, Department of Surgery, Northwestern University, Chicago, Illinois. M.J. Clark is lead statistician, Consulting for Statistics, Computing and Analytics Research, University of Michigan, Ann Arbor, Michigan. R.G. Williams was, at the time of the research reported here, adjunct research professor, Indiana University School of Medicine, Indianapolis, Indiana. G. Wnuk is program manager, Center for Surgical Training and Research, University of Michigan, Ann Arbor, Michigan. J.P. Fryer is vice chair of education, Department of Surgery, Northwestern University, Chicago, Illinois. B.C. George is assistant professor and director, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan; ORCID: https://orcid.org/0000-0002-9404-5255
| | | | | | | | | | | | | |
Collapse
|
30
|
Chen JX, Kozin E, Bohnen J, George B, Deschler D, Emerick K, Gray ST. Tracking operative autonomy and performance in otolaryngology training using smartphone technology: A single institution pilot study. Laryngoscope Investig Otolaryngol 2019; 4:578-586. [PMID: 31890874 PMCID: PMC6929585 DOI: 10.1002/lio2.323] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2019] [Revised: 10/04/2019] [Accepted: 10/18/2019] [Indexed: 11/05/2022] Open
Abstract
BACKGROUND In the era of duty hour restrictions, otolaryngology residents may not gain the operative experience necessary to function autonomously by the end of training. This study quantifies residents' autonomy during key indicator cases, defined by the Accreditation Council for Graduate Medical Education. STUDY DESIGN Prospective cohort study. METHODS Faculty and residents at a large academic institution were surveyed on the surgical autonomy trainees should achieve for otolaryngology key indicator surgeries at each training level. Residents and faculty used the mobile application "System for Improving and Measuring Procedural Learning" (SIMPL) between December 2017 and July 2018 to log trainees' operative autonomy during cases on a validated four-level Zwisch scale, from "show and tell" to "supervision only." RESULTS The study included 40 participants (23 residents and 17 attendings). The survey response rate was 83%. In surveys, residents overestimated the autonomy PGY5 residents should achieve for parotidectomy, rhinoplasty, thyroid/parathyroidectomy, and airway procedures compared with faculty (P < .05). Using SIMPL, 833 evaluations were logged of which 253 were paired evaluations for key indicator cases. Comparing survey predictions with actual cases logged in SIMPL, residents and faculty overestimated the autonomy achieved by senior trainees performing mastoidectomy (PGY5, P < .05) and ethmoidectomy (PGY4/5, P < .05); both felt that senior residents should operate with between "passive help" and "supervision only" whereas residents actually had "passive help." Residents overestimated their autonomy during rhinoplasty (PGY5, P = .017) and parotidectomy (PGY5, P = .007) while attendings accurately expected chief residents to have "passive help." CONCLUSIONS Resident surgical autonomy varies across otolaryngology procedures. Multicenter studies are needed to elucidate national trends. LEVEL OF EVIDENCE 2.
Collapse
Affiliation(s)
- Jenny X. Chen
- Department of Otolaryngology–Head and Neck SurgeryMassachusetts Eye and Ear/Harvard Medical SchoolBostonMassachusetts
| | - Elliott Kozin
- Department of Otolaryngology–Head and Neck SurgeryMassachusetts Eye and Ear/Harvard Medical SchoolBostonMassachusetts
| | - Jordan Bohnen
- Department of General SurgeryMassachusetts General HospitalBostonMassachusetts
| | - Brian George
- Department of General SurgeryUniversity of MichiganAnn ArborMichigan
| | - Daniel Deschler
- Department of Otolaryngology–Head and Neck SurgeryMassachusetts Eye and Ear/Harvard Medical SchoolBostonMassachusetts
| | - Kevin Emerick
- Department of Otolaryngology–Head and Neck SurgeryMassachusetts Eye and Ear/Harvard Medical SchoolBostonMassachusetts
| | - Stacey T. Gray
- Department of Otolaryngology–Head and Neck SurgeryMassachusetts Eye and Ear/Harvard Medical SchoolBostonMassachusetts
| |
Collapse
|
31
|
George BC, Bohnen JD, Schuller MC, Fryer JP. Using smartphones for trainee performance assessment: A SIMPL case study. Surgery 2019; 167:903-906. [PMID: 31668358 DOI: 10.1016/j.surg.2019.09.011] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2019] [Revised: 06/27/2019] [Accepted: 09/17/2019] [Indexed: 11/25/2022]
Abstract
Workplace-based assessments are used by raters to evaluate observed performance of trainees in actual clinical practice. These types of assessments are of growing interest, especially because observed performance is prioritized within the larger competency-based medical educational movement. Implementation of workplace-based assessments has, however, been challenging. This article describes the motivations and implications for workplace-based assessments that leverage smartphone technology. It does so in reference to an app called SIMPL (System for Improving and Measuring Procedural Learning) in order to highlight some of the challenges and benefits one might encounter during implementation of similar systems.
Collapse
Affiliation(s)
- Brian C George
- Center for Surgical Training and Research, Michigan Medicine, University of Michigan, Ann Arbor, MI.
| | - Jordan D Bohnen
- Department of Surgery, Massachusetts General Hospital, Boston, MA
| | - Mary C Schuller
- Department of Surgery, Northwestern University, Evanston, IL
| | | |
Collapse
|
32
|
Chen JX, Kozin E, Bohnen J, George B, Deschler DG, Emerick K, Gray ST. Assessments of Otolaryngology Resident Operative Experiences Using Mobile Technology: A Pilot Study. Otolaryngol Head Neck Surg 2019; 161:939-945. [DOI: 10.1177/0194599819868165] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Objectives Surgical education has shifted from the Halstedian model of “see one, do one, teach one” to a competency-based model of training. Otolaryngology residency programs can benefit from a fast and simple system to assess residents’ surgical skills. In this quality initiative, we hypothesize that a novel smartphone application called System for Improving and Measuring Procedural Learning (SIMPL) could be applied in an otolaryngology residency to facilitate the assessment of resident operative experiences. Methods The Plan Do Study Act method of quality improvement was used. After researching tools of surgical assessment and trialing SIMPL in a resident-attending pair, we piloted SIMPL across an otolaryngology residency program. Faculty and residents were trained to use SIMPL to rate resident operative performance and autonomy with a previously validated Zwisch Scale. Results Residents (n = 23) and faculty (n = 17) were trained to use SIMPL using a standardized curriculum. A total of 833 assessments were completed from December 1, 2017, to June 30, 2018. Attendings completed a median 20 assessments, and residents completed a median 14 self-assessments. All evaluations were resident initiated, and attendings had a 78% median response rate. Evaluations took residents a median 22 seconds to complete; 126 unique procedures were logged, representing all 14 key indicator cases for otolaryngology. Discussion This is the first residency-wide application of a mobile platform to track the operative experiences of otolaryngology residents. Implications for Practice We adapted and implemented a novel assessment tool in a large otolaryngology program. Future multicenter studies will benchmark resident operative experiences nationwide.
Collapse
Affiliation(s)
- Jenny X. Chen
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA
- Department of Otolaryngology–Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| | - Elliott Kozin
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA
- Department of Otolaryngology–Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| | - Jordan Bohnen
- Department of General Surgery Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Brian George
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan, USA
| | - Daniel G. Deschler
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA
- Department of Otolaryngology–Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| | - Kevin Emerick
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA
- Department of Otolaryngology–Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| | - Stacey T. Gray
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Massachusetts, USA
- Department of Otolaryngology–Head and Neck Surgery, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
33
|
Holmstrom AL, Meyerson SL. Obtaining Meaningful Assessment in Thoracic Surgery Education. Thorac Surg Clin 2019; 29:239-247. [PMID: 31235292 DOI: 10.1016/j.thorsurg.2019.03.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
Abstract
Training in thoracic surgery has evolved immensely over the past decade due to the advent of integrated programs, technological innovations, and regulations on resident duty hours, decreasing the time trainees have to learn. These changes have made assessment of thoracic surgical trainees even more important. Shifts in medical education have increasingly emphasized competency, which has led to novel competency-based assessment tools for clinical and operative assessment. These novel tools take advantage of simulation and modern technology to provide more frequent and comprehensive assessment of the surgical trainee to ensure competence.
Collapse
Affiliation(s)
- Amy L Holmstrom
- Department of Surgery, Northwestern University Feinberg School of Medicine, 676 North Saint Clair Street, Suite 2320, Chicago, IL 60611, USA
| | - Shari L Meyerson
- Department of Surgery, University of Kentucky, 740 South Limestone, Suite A301, Lexington, KY 40536, USA.
| |
Collapse
|
34
|
Eaton M, Scully R, Schuller M, Yang A, Smink D, Williams RG, Bohnen JD, George BC, Fryer JP, Meyerson SL. Value and Barriers to Use of the SIMPL Tool for Resident Feedback. JOURNAL OF SURGICAL EDUCATION 2019; 76:620-627. [PMID: 30770304 DOI: 10.1016/j.jsurg.2019.01.012] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/03/2018] [Revised: 01/14/2019] [Accepted: 01/19/2019] [Indexed: 06/09/2023]
Abstract
OBJECTIVE The System for Improving and Measuring Procedural Learning (SIMPL) is a smart-phone application used to provide residents with an evaluation of operative autonomy and feedback. This study investigated the perceived benefits and barriers to app use. DESIGN A database of previously performed SIMPL evaluations was analyzed to identify high, low, and never users. Potential predisposing factors to use were explored. A survey investigating key areas of value and barriers to use for the SIMPL application was sent to resident and faculty users. Respondents were asked to self-identify how often they used the app. The perceived benefits and barriers were correlated with the level of usage. Qualitative analysis of free text responses was used to determine strategies to increase usage. SETTING General surgery training programs who are members of the Procedural Learning and Safety Collaborative. PARTICIPANTS Surgical residents and faculty. RESULTS At least 1 SIMPL evaluation was created for 411 residents and 524 faculty. Thirty percent of both faculty and residents were high-frequency users. Thirty percent of faculty were never users. One hundred eighty-eight residents and 207 faculty (response rate 46%) completed the survey. High-frequency resident users were more likely to perceive a benefit for both numerical evaluations (76% vs 30%) and dictated feedback (92% vs 30%). Faculty and residents commonly blamed each other for not creating and completing evaluations regularly (87% of residents, 81% of faculty). Suggested strategies to increase usage included reminders and integration with existing data systems. CONTRIBUTIONS Frequent users perceive value from the application, particularly from dictated feedback and see a positive impact on feedback in their programs. Faculty engagement represents a major barrier to adoption. Mechanisms which automatically remind residents to initiate an evaluation will help improve utilization but programs must work to enhance faculty willingness to respond and dictate feedback.
Collapse
Affiliation(s)
- Megan Eaton
- University of Texas Health San Antonio, San Antonio, Texas
| | - Rebecca Scully
- Massachusetts General Hospital, Department of Surgery, Boston, Massachusetts
| | - Mary Schuller
- Northwestern University, Department of Surgery, Chicago, Illinois
| | - Amy Yang
- Northwestern University, Department of Surgery, Chicago, Illinois
| | - Douglas Smink
- Massachusetts General Hospital, Department of Surgery, Boston, Massachusetts
| | - Reed G Williams
- Indiana University, Department of Surgery, Bloomington, Indiana
| | - Jordan D Bohnen
- Massachusetts General Hospital, Department of Surgery, Boston, Massachusetts
| | - Brian C George
- University of Michigan, Department of Surgery, Ann Arbor, Michigan
| | - Jonathan P Fryer
- Northwestern University, Department of Surgery, Chicago, Illinois
| | - Shari L Meyerson
- University of Kentucky, Department of Surgery, Lexington, Kentucky.
| |
Collapse
|
35
|
Saliken D, Dudek N, Wood TJ, MacEwan M, Gofton WT. Comparison of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to a Single-Item Performance Score. TEACHING AND LEARNING IN MEDICINE 2019; 31:146-153. [PMID: 30514128 DOI: 10.1080/10401334.2018.1503961] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/06/2018] [Revised: 06/22/2018] [Accepted: 07/12/2018] [Indexed: 06/09/2023]
Abstract
UNLABELLED Construct: We compared a single-item performance score with the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for their ability in assessing surgical competency. BACKGROUND Surgical programs are adopting competency-based frameworks. The adoption of these frameworks for assessment requires tools that produce accurate and valid assessments of knowledge and technical performance. An assessment tool that is quick to complete could improve feasibility, reduce delays, and result in a higher volume of assessments of learners. Previous work demonstrated that the 9-item O-SCORE can produce valid results; the goal of this study was to determine if a single-item performance rating (Is candidate competent to independently complete procedure: yes or no) completed at a separate viewing would correlate to the O-SCORE, thus increasing feasibility of procedural competence assessment. APPROACH Nineteen residents and 2 staff orthopedic surgeons from the University of Ottawa volunteered for a 2-part OSCE-style station including a written questionnaire and videotaped simulated open reduction and internal fixation midshaft radius fracture. Each performance was rated independently by 3 orthopedic surgeons using a single-item performance score (Time 1). The performances were assessed again 6 weeks later by the 3 raters using the O-SCORE (Time 2). Correlation between the single-item performance score and the O-SCORE were evaluated. RESULTS Three orthopedic surgeons completed 21 ratings each resulting in 63 orthopedic ratings. There was a high level of correlation and agreement between the single-item performance score at Time 1 and Time 2 (κ correlation =0.72-1.00; p < .001; percentage agreement =90%-100%). The reliability of the O-SCORE at Time 2 with three raters was 0.83 and the internal consistency was 0.89. There was a tendency for each rater to assign more yes responses to the more senior trainees. CONCLUSIONS A single-item performance score correlated highly with the O-SCORE in an orthopedic setting. A single-item score could be used to supplement a multi-item score with similar results in orthopedics. There is still benefit in completing multi-item scores such as the O-SCORE evaluations to guide specific areas of improvement and direct feedback.
Collapse
Affiliation(s)
- David Saliken
- a Department of Surgery , RebalanceMD , Victoria , British Columbia , Canada
| | - Nancy Dudek
- b Department of Surgery , University of Ottawa , Ottawa , Ontario , Canada
| | - Timothy J Wood
- c Department of Innovation in Medical Education , University of Ottawa , Ottawa , Ontario , Canada
| | - Matthew MacEwan
- d Department of Orthopedic Surgery , University of Ottawa , Ottawa , Ontario , Canada
- e Departments of Surgery and Innovation in Medical Education , University of Ottawa , Ottawa , Ontario , Canada
| | - Wade T Gofton
- c Department of Innovation in Medical Education , University of Ottawa , Ottawa , Ontario , Canada
| |
Collapse
|
36
|
How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency? Ann Surg 2019; 269:377-382. [DOI: 10.1097/sla.0000000000002554] [Citation(s) in RCA: 28] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/23/2023]
|
37
|
Bello RJ, Meyer ML, Cooney DS, Rosson GD, Lifchez SD, Cooney CM. The reliability of operative rating tool evaluations: How late is too late to provide operative performance feedback? Am J Surg 2018; 216:1052-1055. [DOI: 10.1016/j.amjsurg.2018.04.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2018] [Revised: 03/31/2018] [Accepted: 04/09/2018] [Indexed: 11/30/2022]
|
38
|
Abdelsattar JM, AlJamal YN, Ruparel RK, Rowse PG, Heller SF, Farley DR. Correlation of Objective Assessment Data With General Surgery Resident In-Training Evaluation Reports and Operative Volumes. JOURNAL OF SURGICAL EDUCATION 2018; 75:1430-1436. [PMID: 29773409 DOI: 10.1016/j.jsurg.2018.04.016] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/12/2017] [Revised: 03/30/2018] [Accepted: 04/22/2018] [Indexed: 06/08/2023]
Abstract
OBJECTIVE Faculty evaluations, ABSITE scores, and operative case volumes often tell little about true resident performance. We developed an objective structured clinical examination called the Surgical X-Games (5 rooms, 15 minutes each, 12-15 tests total, different for each postgraduate [PGY] level). We hypothesized that performance in X-Games will prove more useful in identifying areas of strength or weakness among general surgery (GS) residents than faculty evaluations, ABSITE scores, or operative cases volumes. DESIGN PGY 2 to 5 GS residents (n = 35) were tested in a semiannual X-Games assessment using multiple simulation tasks: laparoscopic skills, bowel anastomosis, CT/CXR analysis, chest tube placement, etc. over 1 academic year. Resident scores were compared to their ABSITE, in-training evaluation reports, and operating room case numbers. SETTING Academic medical center. PARTICIPANTS PGY-2, 3, 4, and 5 GS residents at Mayo Clinic in Rochester, MN. RESULTS Results varied greatly within each class except for staff evaluations: in-training evaluation reports medians for PGY-2s were 5.3 (range: 5.0-6.0), PGY-3s 5.9 (5.5-6.3), PGY-4s 5.6 (5.0-6.0), and PGY-5s were 6.1 (5.6-6.9). Although ABSITE and operating room case volumes fluctated greatly with each PGY class, only X-Games scores (median: PGY-2 = 82, PGY-3 = 61, PGY-4 = 76, and PGY-5 = 60) correlated positively (p < 0.05) with operative case volume and negatively (p < 0.05) with staff evaluations. CONCLUSIONS X-Games assessment generated wide differentiation of resident performance quickly, inexpensively, and objectively. Although "Minnesota-nice" surgical staff may feel all GS trainees are "above average," objective assessment tells us otherwise.
Collapse
Affiliation(s)
- Jad M Abdelsattar
- Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota
| | - Yazan N AlJamal
- Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota
| | - Raaj K Ruparel
- Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota
| | - Phillip G Rowse
- Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota
| | - Stephanie F Heller
- Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota
| | - David R Farley
- Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota.
| |
Collapse
|
39
|
Shekar R, George BC, Bohnen JD, Villa A. A SIMPL application to assess oral medicine residents' performance and level of autonomy. Oral Surg Oral Med Oral Pathol Oral Radiol 2018; 126:246-251. [PMID: 29937360 DOI: 10.1016/j.oooo.2018.05.017] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2017] [Revised: 05/09/2018] [Accepted: 05/18/2018] [Indexed: 11/25/2022]
Abstract
OBJECTIVES The aim of this study was to evaluate a smartphone application, SIMPL, to optimize faculty guidance and oral medicine residents' performance, and to measure resident-faculty agreement for performance and supervision levels. STUDY DESIGN Raters used the 5-level "Performance" scale to assess resident's readiness for independent practice and the 4-level Zwisch scale to assess faculty guidance, between June 2016 and June 2017. Junior residents were trainees with less than 18 months of training, and senior residents had more than 18 months of training. Descriptive statistics were used to analyze supervision and autonomy. Performance and supervision agreement was estimated as percentage-agreement and measured using κ and 95% confidence intervals. RESULTS A total of 660 evaluations were performed by 6 residents and 6 faculty members. Senior residents received higher performance scores compared with junior residents (P for trend < .01). In terms of complexity of cases and level of supervision, there were no significant differences between juniors and seniors (P = .69 and P = .39, respectively). Residents were "practice ready" or greater than for 80% cases. Residents received meaningful autonomy for 64.8% cases. Faculty-resident concordance was 86.1% for performance (with κ of 0.77 [95% CI 66.1%-87.6%]) and 92.4% for supervision (with κ of 0.84 [95% CI 80%-88%]). CONCLUSIONS SIMPL can feasibly be used for real-time assessment of residents' performance and autonomy.
Collapse
Affiliation(s)
- Revathi Shekar
- Division of Oral Medicine and Dentistry, Brigham and Women's Hospital, Department of Oral Medicine, Infection and Immunity, Harvard School of Dental Medicine, Boston, MA, USA.
| | - Brian C George
- Department of Surgery, University of Michigan, Ann Arbor, MI, USA
| | - Jordan D Bohnen
- Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
| | - Alessandro Villa
- Division of Oral Medicine and Dentistry, Brigham and Women's Hospital, Department of Oral Medicine, Infection and Immunity, Harvard School of Dental Medicine, Boston, MA, USA
| |
Collapse
|
40
|
Fahim C, Wagner N, Nousiainen MT, Sonnadara R. Assessment of Technical Skills Competence in the Operating Room: A Systematic and Scoping Review. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2018; 93:794-808. [PMID: 28953567 DOI: 10.1097/acm.0000000000001902] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
PURPOSE While academic accreditation bodies continue to promote competency-based medical education (CBME), the feasibility of conducting regular CBME assessments remains challenging. The purpose of this study was to identify evidence pertaining to the practical application of assessments that aim to measure technical competence for surgical trainees in a nonsimulated, operative setting. METHOD In August 2016, the authors systematically searched Medline, Embase, and the Cochrane Database of Systematic Reviews for English-language, peer-reviewed articles published in or after 1996. The title, abstract, and full text of identified articles were screened. Data regarding study characteristics, psychometric and measurement properties, implementation of assessment, competency definitions, and faculty training were extracted. The findings from the systematic review were supplemented by a scoping review to identify key strategies related to faculty uptake and implementation of CBME assessments. RESULTS A total of 32 studies were included. The majority of studies reported reasonable scores of interrater reliability and internal consistency. Seven articles identified minimum scores required to establish competence. Twenty-five articles mentioned faculty training. Many of the faculty training interventions focused on timely completion of assessments or scale calibration. CONCLUSIONS There are a number of diverse tools used to assess competence for intraoperative technical skills and a lack of consensus regarding the definition of technical competence within and across surgical specialties. Further work is required to identify when and how often trainees should be assessed and to identify strategies to train faculty to ensure timely and accurate assessment.
Collapse
Affiliation(s)
- Christine Fahim
- C. Fahim is a PhD candidate, Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada. N. Wagner is a PhD candidate, Department of Psychology, Neuroscience and Behaviour, McMaster University, Hamilton, Ontario, Canada. M.T. Nousiainen is orthopedic surgeon and assistant professor, Sunnybrook Hospital, Department of Surgery, University of Toronto, Toronto, Ontario, Canada. R. Sonnadara is director of education science and associate professor, Department of Surgery, McMaster University, Hamilton, Ontario, Canada, and associate professor, Department of Surgery, University of Toronto, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0001-8318-5714
| | | | | | | |
Collapse
|
41
|
Linsenmeyer M, Wimsatt L, Speicher M, Powers J, Miller S, Katsaros E. Assessment Considerations for Core Entrustable Professional Activities for Entering Residency. J Osteopath Med 2018; 118:243-251. [DOI: 10.7556/jaoa.2018.049] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
Abstract
Abstract
Context
In the process of analyzing entrustable professional activities (EPAs) for use in medical education, ten Cate and others identified challenges, including the need for valid and reliable EPA assessment strategies.
Objective
To provide osteopathic medical schools with a database of assessment tools compiled from the literature to assist them with the development and implementation of robust, evidence-based assessment methods.
Methods
MEDLINE, ERIC, PubMed, and other relevant databases were searched using MeSH keywords for articles outlining robust, evidence-based assessment tools that could be used in designing assessments for EPAs 1 through 6.
Results
A total of 55 publications were included in content analysis and reporting. All but 2 of the assessment articles were conducted in an undergraduate or graduate medical education setting. The majority of the 55 articles related to assessment of competencies affiliated with EPA 2 (16 articles) and EPA 4 (15 articles). Four articles focused on EPA 3.
Conclusion
Osteopathic medical schools can use this database of assessment tools to support the development of EPA-specific assessment plans that match the unique context and needs of their institution.
Collapse
|
42
|
Fryer JP, Teitelbaum EN, George BC, Schuller MC, Meyerson SL, Theodorou CM, Kang J, Yang A, Zhao L, DaRosa DA. Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment. JOURNAL OF SURGICAL EDUCATION 2018; 75:333-343. [PMID: 28363675 DOI: 10.1016/j.jsurg.2016.11.018] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2016] [Revised: 10/25/2016] [Accepted: 11/28/2016] [Indexed: 06/07/2023]
Abstract
OBJECTIVE We have previously demonstrated the feasibility and validity of a smartphone-based system called Procedural Autonomy and Supervision System (PASS), which uses the Zwisch autonomy scale to facilitate assessment of the operative performances of surgical residents and promote progressive autonomy. To determine whether the use of PASS in a general surgery residency program is associated with any negative consequences, we tested the null hypothesis that PASS implementation at our institution would not negatively affect resident or faculty satisfaction in the operating room (OR) nor increase mean OR times for cases performed together by residents and faculty. METHODS Mean OR times were obtained from the electronic medical record at Northwestern Memorial Hospital for the 20 procedures most commonly performed by faculty members with residents before and after PASS implementation. OR times were compared via two-sample t-test. The OR Educational Environment Measure tool was used to assess OR satisfaction with all clinically active general surgery residents (n = 31) and full-time general surgery faculty members (n = 27) before and after PASS implementation. Results were compared using the Mann-Whitney rank sum test. RESULTS A significant prolongation in mean OR time between control and study period was found for only 1 of the 20 operative procedures performed at least 20 times by participating faculty members with residents. Based on the overall survey score, no significant differences were found between resident and faculty responses to the OR Educational Environment Measure survey before and after PASS implementation. When individual survey items were compared, while no differences were found with resident responses, differences were noted with faculty responses for 7 of the 35 items addressed although after Bonferroni correction none of these differences remained significant. CONCLUSIONS Our data suggest that PASS does not increase mean OR times for the most commonly performed procedures. Resident OR satisfaction did not significantly change during PASS implementation, whereas some changes in faculty satisfaction were noted suggesting that PASS implementation may have had some negative effect with them. Although the effect on faculty satisfaction clearly requires further investigation, our findings support that use of an autonomy-based OR performance assessment system such as PASS does not appear to have a major negative influence on OR times nor OR satisfaction.
Collapse
Affiliation(s)
- Jonathan P Fryer
- Department of Surgery, Northwestern University, Chicago, Illinois.
| | | | - Brian C George
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Mary C Schuller
- Department of Surgery, Northwestern University, Chicago, Illinois
| | - Shari L Meyerson
- Department of Surgery, Northwestern University, Chicago, Illinois
| | | | - Joseph Kang
- Department of Surgery, Northwestern University, Chicago, Illinois
| | - Amy Yang
- Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Lihui Zhao
- Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Debra A DaRosa
- Department of Surgery, Northwestern University, Chicago, Illinois
| |
Collapse
|
43
|
Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents. Surgery 2017; 163:488-494. [PMID: 29277387 DOI: 10.1016/j.surg.2017.10.011] [Citation(s) in RCA: 42] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2017] [Revised: 10/06/2017] [Accepted: 10/10/2017] [Indexed: 12/23/2022]
Abstract
BACKGROUND Concerns exist regarding the competency of general surgery graduates with performing core general surgery procedures. Current competence assessment incorporates minimal procedural numbers requirements. METHODS Based on the Zwisch scale we evaluated the level of autonomy achieved by categorical PGY1-5 general surgery residents at 14 U.S. general surgery resident training programs between September 1, 2015 and December 31, 2016. With 5 of the most commonly performed core general surgery procedures, we correlated the level of autonomy achieved by each resident with the number of procedures they had performed before the evaluation period, with the intent of identifying specific target numbers that would correlate with the achievement of meaningful autonomy for each procedure with most residents. RESULTS Whereas a definitive target number was identified for laparoscopic appendectomy (i.e. 25), for the other 4 procedures studied (i.e. laparoscopic cholecystectomy, 52; open inguinal hernia repair, 42; ventral hernia repair, 35; and partial colectomy, 60), target numbers identified were less definitive and/or were higher than many residents will experience during their surgical residency training. CONCLUSIONS We conclude that procedural target numbers are generally not effective in predicting procedural competence and should not be used as the basis for determining residents' readiness for independent practice.
Collapse
|
44
|
|
45
|
Kaban LB, Cappetta A, George BC, Lahey ET, Bohnen JD, Troulis MJ. Evaluation of Oral and Maxillofacial Surgery Residents' Operative Skills: Feasibility and Engagement Study Using SIMPL Software for a Mobile Phone. J Oral Maxillofac Surg 2017; 75:2041-2047. [DOI: 10.1016/j.joms.2017.05.036] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2017] [Revised: 05/05/2017] [Accepted: 05/30/2017] [Indexed: 11/30/2022]
|
46
|
Peshkepija AN, Basson MD, Davis AT, Ali M, Haan PS, Gupta RN, Hardaway JC, Nebeker CA, McLeod MK, Osmer RL, Anderson CI. Perioperative self-reflection among surgical residents. Am J Surg 2017; 214:564-570. [PMID: 28259204 DOI: 10.1016/j.amjsurg.2016.12.007] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2016] [Revised: 12/05/2016] [Accepted: 12/19/2016] [Indexed: 11/30/2022]
|
47
|
Do female surgeons learn or teach differently? Am J Surg 2017; 213:282-287. [PMID: 28139201 DOI: 10.1016/j.amjsurg.2016.10.010] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2016] [Revised: 08/24/2016] [Accepted: 10/20/2016] [Indexed: 11/21/2022]
|
48
|
Kozin ED, Bohnen JD, George BC, Justicz N, Colaianni CA, Duarte M, Gray ST. Novel Mobile App Allows for Fast and Validated Intraoperative Assessment of Otolaryngology Residents. OTO Open 2017; 1:2473974X16685705. [PMID: 30480172 PMCID: PMC6239054 DOI: 10.1177/2473974x16685705] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2016] [Accepted: 12/02/2016] [Indexed: 11/17/2022] Open
Abstract
Evaluation of resident operative skills is challenging in the fast-paced
operating room environment and limited by lack of validated assessment metrics.
We describe a smartphone-based app that enables rapid assessment of operative
skills. Accreditation Council for Graduate Medical Education (ACGME)
otolaryngology taxonomy surgical procedures (n = 593) were uploaded to the
software platform. The app was piloted over 1 month. Outcomes included (1)
completion of evaluation, (2) time spent completing the evaluation, and (3)
quantification of case complexity, operative autonomy, and performance. During
the study, 12 of 12 procedures, corresponding to 3 paired evaluated by the
resident/attending dyad. Mean ± SD time of evaluation completion was 98.0 ± 24.2
and 123.0 ± 14.0 seconds for the resident and attending, respectively. Mean time
between resident and attending evaluation completion was 27.9 ± 26.8 seconds.
Resident and attending scores for case complexity, operative autonomy, and
performance were strongly correlated (P < .0001). Rapid
evaluation of resident intraoperative performance is feasible using
smartphone-based technology.
Collapse
Affiliation(s)
- Elliott D Kozin
- Department of Otolaryngology, Massachusetts Eye and Ear Infirmary, Boston, Massachusetts, USA.,Department of Otolaryngology, Harvard Medical School, Boston, Massachusetts, USA
| | - Jordan D Bohnen
- Department of General Surgery, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Brian C George
- Department of General Surgery, Massachusetts General Hospital, Boston, Massachusetts, USA.,Department of Surgery, University of Michigan Health System, Ann Arbor, Michigan, USA
| | - Natalie Justicz
- Department of Otolaryngology, Massachusetts Eye and Ear Infirmary, Boston, Massachusetts, USA.,Department of Otolaryngology, Harvard Medical School, Boston, Massachusetts, USA
| | - C Alessandra Colaianni
- Department of Otolaryngology, Massachusetts Eye and Ear Infirmary, Boston, Massachusetts, USA.,Department of Otolaryngology, Harvard Medical School, Boston, Massachusetts, USA
| | - Maria Duarte
- Department of Otolaryngology, Massachusetts Eye and Ear Infirmary, Boston, Massachusetts, USA.,Department of Otolaryngology, Harvard Medical School, Boston, Massachusetts, USA
| | - Stacey T Gray
- Department of Otolaryngology, Massachusetts Eye and Ear Infirmary, Boston, Massachusetts, USA.,Department of Otolaryngology, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
49
|
Hardaway JC, Basson MD, Ali M, Davis AT, Haan PS, Gupta RN, Peshkepija AN, Nebeker CA, McLeod MK, Osmer RL, Anderson CI. A taxonomy of perioperative surgical learning: Trending resident skill acquisition. Am J Surg 2017; 213:260-267. [PMID: 28062076 DOI: 10.1016/j.amjsurg.2016.09.045] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2016] [Revised: 08/18/2016] [Accepted: 09/24/2016] [Indexed: 11/24/2022]
Abstract
BACKGROUND Resident and curriculum evaluation require tracking surgical resident operative performance, yet what and when to measure remains unclear. METHODS From a multi-institutional database, we reviewed 611 resident/surgeon-paired assessments of ACGME Milestones and modified OPRS ratings for different cases and postgraduate years. RESULTS Faculty Milestone ratings increased with each PGY (p=<0.001) and correlated with resident self-ratings (ICC = 0.83). Mean OPRS scores increased in small increments with substantial intra-year variability. Progression among individual OPRS subcategories was not apparent from more global analyses. Interestingly, male faculty offered lower ratings than female faculty. CONCLUSIONS Milestones and modified mean OPRS ratings suggest residents are learning, yet lack sufficient discrimination for promotion or curricular analysis. Differential progression through OPRS subcategories suggests a taxonomy of surgical learning that can be tailored to focus on different skills at each point in the training continuum. The effect of faculty gender on resident ratings awaits further study.
Collapse
Affiliation(s)
- John C Hardaway
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Marc D Basson
- University of North Dakota School of Medicine & Health Sciences, 501 North Columbia Road, Stop 9037, Grand Forks, ND, 58202, USA
| | - Muhammad Ali
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Alan T Davis
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA; Grand Rapids Medical Education Partners Research Department, 945 Ottawa Ave. NW, Grand Rapids, MI, 49503, USA
| | - Pam S Haan
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Rama N Gupta
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Andi N Peshkepija
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Cody A Nebeker
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Michael K McLeod
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Robert L Osmer
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA
| | - Cheryl I Anderson
- College of Human Medicine, Department of Surgery, Michigan State University, 1200 E. Michigan Avenue, Suite 655, Lansing, MI, 48912, USA.
| | | |
Collapse
|
50
|
Strandbygaard J, Scheele F, Sørensen JL. Twelve tips for assessing surgical performance and use of technical assessment scales. MEDICAL TEACHER 2017; 39:32-37. [PMID: 27678279 DOI: 10.1080/0142159x.2016.1231911] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
Using validated assessment scales for technical competence can help structure and standardize assessment and feedback for both the trainee and the supervisor and thereby avoid bias and drive learning. Correct assessment of operative skills can establish learning curves and allow adequate monitoring. However, the assessment of surgical performance is not an easy task, since it includes many proxy parameters, which are hard to measure. Although numerous technical assessment scales exist, both within laparoscopic and open surgery, the validity evidence is often sparse, and this can raise doubts about reliability and educational outcome. Furthermore, the implementation of technical assessment scales varies due to several obstacles and doubts about accurate use. In this 12-tips article, we aim to give the readers a critical and useful appraisal of some of the common questions and misunderstandings regarding the use of surgical assessment scales and provide tips to ease and overcome potential pitfalls.
Collapse
Affiliation(s)
- Jeanett Strandbygaard
- a Department of Obstetrics and Gynaecology, Juliane Marie Centre, Centre for Children, Women and Reproduction , University Hospital of Copenhagen , Copenhagen , Denmark
| | - Fedde Scheele
- b School of Medical Sciences VUmc and Athena Institute for Transdisciplinary Research , VU University , Amsterdam , Netherlands
| | - Jette Led Sørensen
- c Juliane Marie Centre, Centre for Children, Women and Reproduction , University Hospital of Copenhagen , Copenhagen , Denmark
| |
Collapse
|