1
|
Spencer BL, Krumm A, Izadi S, Hirschl RB, Modi BP, Ehrlich P, Newman EA, Zendejas B. How Many Operative Performance Ratings Does a Pediatric Surgery Fellow Need to Be Deemed Practice Ready? J Pediatr Surg 2024; 59:31-36. [PMID: 37845126 DOI: 10.1016/j.jpedsurg.2023.09.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Accepted: 09/06/2023] [Indexed: 10/18/2023]
Abstract
PURPOSE Identifying the number of cases required for a fellow to achieve competence has been challenging. Workplace-based assessment (WBA) systems make collecting performance data practical and create the opportunity to translate WBA ratings into probabilistic statements about a fellow's likelihood of performing to a given standard on a subsequent assessment opportunity. METHODS We compared data from two pediatric surgery training programs that used the performance rating scale from the Society for Improving Medical Professional Learning (SIMPL). We used a Bayesian generalized linear mixed effects model to examine the relationship past and future performance for three procedures: Laparoscopic Inguinal Hernia Repair, Laparoscopic Gastrostomy Tube Placement, and Pyloromyotomy. RESULTS For site one, 26 faculty assessed 9 fellows on 16 procedures yielding 1094 ratings, of which 778 (71%) earned practice-ready ratings. For site two, 25 faculty rated 3 fellows on 4 unique procedures yielding 234 ratings of which 151 (65%) were deemed practice-ready. We identified similar model-based future performance expectations, with prior practice-ready ratings having a similar average effect across both sites (Site one, B = 0.25; Site two, B = 0.25). Similar prior practice-ready ratings were needed for Laparoscopic G-Tube Placement (Site one = 13; Site two = 14), while greater differences were observed for Laparoscopic Inguinal Hernia Repair (Site one = 10; Site two = 15) and Pyloromyotomy (Site one = 10; Site two = 15). CONCLUSION Our approach to modeling operative performance data is effective at determining future practice readiness of pediatric surgery fellows across multiple faculty and fellow groups. This method could be used to establish minimum case number requirements. TYPE OF STUDY Original manuscript, Study of Diagnostic Test. LEVEL OF EVIDENCE II.
Collapse
Affiliation(s)
- Brianna L Spencer
- Section of Pediatric Surgery, Department of Surgery, University of Michigan Medical School, C.S. Mott Children's Hospital, 1540 E Hospital Dr, Ann Arbor, MI, 48109, USA.
| | - Andrew Krumm
- Department of Learning Health Sciences, University of Michigan Medical School, 221 Victor Vaughan Building, 1111 E. Catherine St, Ann Arbor, MI, 48109, USA
| | - Shawn Izadi
- Department of Surgery, Boston Children's Hospital, 300 Longwood Ave, Boston, MA, 02115, USA
| | - Ronald B Hirschl
- Section of Pediatric Surgery, Department of Surgery, University of Michigan Medical School, C.S. Mott Children's Hospital, 1540 E Hospital Dr, Ann Arbor, MI, 48109, USA
| | - Biren P Modi
- Department of Surgery, Boston Children's Hospital, 300 Longwood Ave, Boston, MA, 02115, USA
| | - Peter Ehrlich
- Section of Pediatric Surgery, Department of Surgery, University of Michigan Medical School, C.S. Mott Children's Hospital, 1540 E Hospital Dr, Ann Arbor, MI, 48109, USA
| | - Erika A Newman
- Section of Pediatric Surgery, Department of Surgery, University of Michigan Medical School, C.S. Mott Children's Hospital, 1540 E Hospital Dr, Ann Arbor, MI, 48109, USA
| | - Benjamin Zendejas
- Department of Surgery, Boston Children's Hospital, 300 Longwood Ave, Boston, MA, 02115, USA
| |
Collapse
|
2
|
Haas MRC, Davis MG, Harvey CE, Huang R, Scott KW, George BC, Wnuk GM, Burkhardt J. Implementation of the SIMPL (Society for Improving Medical Professional Learning) performance assessment tool in the emergency department: A pilot study. AEM EDUCATION AND TRAINING 2023; 7:e10842. [PMID: 36777102 PMCID: PMC9899600 DOI: 10.1002/aet2.10842] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 12/01/2022] [Accepted: 12/18/2022] [Indexed: 06/18/2023]
Abstract
Background Feedback and assessment are difficult to provide in the emergency department (ED) setting despite their critical importance for competency-based education, and traditional end-of-shift evaluations (ESEs) alone may be inadequate. The SIMPL (Society for Improving Medical Professional Learning) mobile application has been successfully implemented and studied in the operative setting for surgical training programs as a point-of-care tool that incorporates three assessment scales in addition to dictated feedback. SIMPL may represent a viable tool for enhancing workplace-based feedback and assessment in emergency medicine (EM). Methods We implemented SIMPL at a 4-year EM residency program during a pilot study from March to June 2021 for observable activities such as medical resuscitations and related procedures. Faculty and residents underwent formal rater training prior to launch and were asked to complete surveys regarding the SIMPL app's content, usability, and future directions at the end of the pilot. Results A total of 36/58 (62%) of faculty completed at least one evaluation, for a total of 190 evaluations and an average of three evaluations per faculty. Faculty initiated 130/190 (68%) and residents initiated 60/190 (32%) evaluations. Ninety-one percent included dictated feedback. A total of 45/54 (83%) residents received at least one evaluation, with an average of 3.5 evaluations per resident. Residents generally agreed that SIMPL increased the quality of feedback received and that they valued dictated feedback. Residents generally did not value the numerical feedback provided from SIMPL. Relative to the residents, faculty overall responded more positively toward SIMPL. The pilot generated several suggestions to inform the optimization of the next version of SIMPL for EM training programs. Conclusions The SIMPL app, originally developed for use in surgical training programs, can be implemented for use in EM residency programs, has positive support from faculty, and may provide important adjunct information beyond current ESEs.
Collapse
Affiliation(s)
- Mary R. C. Haas
- Department of Emergency MedicineUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| | - Mallory G. Davis
- Department of Emergency MedicineUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| | - Carrie E. Harvey
- Department of Emergency MedicineUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| | - Rob Huang
- Department of Emergency MedicineUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| | - Kirstin W. Scott
- University of Michigan Emergency Medicine Residency ProgramAnn ArborMichiganUSA
| | - Brian C. George
- Center for Surgical Training and Research, Department of SurgeryUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| | - Gregory M. Wnuk
- Center for Surgical Training and Research, Department of SurgeryUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| | - John Burkhardt
- Departments of Emergency Medicine and Learning Health SciencesUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| |
Collapse
|
3
|
Kraft KH. Advancements in Surgical Performance Evaluation and Feedback. Curr Urol Rep 2023; 24:11-15. [PMID: 36394773 DOI: 10.1007/s11934-022-01134-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/02/2022] [Indexed: 11/18/2022]
Abstract
PURPOSE OF REVIEW How today's urology trainees acquire surgical skills has changed dramatically due to multiple forces placing strain on the graduate medical education mission. The development of workplace-based assessments that deliver feedback while capturing performance data has led to a paradigm shift toward individualized learning. RECENT FINDINGS Delivering feedback that drives surgical skill development requires the educator to provide a meaningful assessment of the learner after an operative experience. Workplace-based assessment involves direct observation of routine clinical practice and has become a central component of competency-based medical education. Urology has the chance to fully embrace competency-based medical education, employing robust feedback mechanisms and workplace-based assessments. We must first define what it means to be a proficient urologist and design an assessment system that captures this collective sentiment. This can only be done through effective engagement and collaboration with stakeholders across our specialty.
Collapse
Affiliation(s)
- Kate H Kraft
- Department of Urology, University of Michigan, 1500 E. Medical Center Drive, SPC 5330, Ann Arbor, MI, 48105, USA.
| |
Collapse
|
4
|
Response to the Comment on "A Proposed Blueprint for Operative Performance Training, Assessment, and Certification". Ann Surg 2021; 274:e938-e939. [PMID: 34784685 DOI: 10.1097/sla.0000000000005082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
5
|
French JC, Pien LC. A Document Analysis of Nationally Available Faculty Assessment Forms of Resident Performance. J Grad Med Educ 2021; 13:833-840. [PMID: 35070096 PMCID: PMC8672836 DOI: 10.4300/jgme-d-21-00289.1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Revised: 06/28/2021] [Accepted: 09/07/2021] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Written feedback by faculty of resident performance is valuable when it includes components based on assessment for learning. However, it is not clear how often assessment forms include these components for summative and formative feedback. OBJECTIVE To analyze prompts used in forms for faculty assessment of resident performance, guided by best practices in survey research methodology, self-regulation theory, and competency-based assessment. METHODS A document analysis, which is a qualitative approach used to analyze content and structure of texts, was completed on assessment forms nationally available in MedHub. Due to the number of forms available, only internal medicine and surgery specialties were included. A document summary form was created to analyze the assessments. The summary form guided researchers through the analysis. RESULTS Forty-eight forms were reviewed, each from a unique residency program. All forms provided a textbox for comments, and 54% made this textbox required for assessment completion. Eighty-three percent of assessments placed the open textbox at the end of the form. One-third of forms contained a simple prompt, "Comments," for the narrative section. Fifteen percent of forms included a box to check if the information on the form had been discussed with the resident. Fifty percent of the assessments were unclear if they were meant to be formative or summative in nature. CONCLUSIONS Our document analysis of assessment forms revealed they do not always follow best practices in survey design for narrative sections, nor do they universally address elements deemed important for promotion of self-regulation and competency-based assessment.
Collapse
Affiliation(s)
- Judith C. French
- Judith C. French, PhD, is Surgical Educator, General Surgery Residency Program, Department of General Surgery, Cleveland Clinic, and Assistant Professor of Surgery, Cleveland Clinic Lerner College of Medicine of Case Western Reserve University
| | - Lily C. Pien
- Lily C. Pien, MD, MHPE, is Core Faculty, Allergy and Immunology Fellowship Program, Cleveland Clinic, and Associate Professor of Medicine, Cleveland Clinic Lerner College of Medicine of Case Western Reserve University
| |
Collapse
|
6
|
Kelly PD, Yengo-Kahn AM, Roth SG, Zuckerman SL, Chitale RV, Wellons JC, Chambless LB. Data-Driven Residency Training: A Scoping Review of Educational Interventions for Neurosurgery Residency Programs. Neurosurgery 2021; 89:750-759. [PMID: 34423828 DOI: 10.1093/neuros/nyab322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Accepted: 07/06/2021] [Indexed: 11/14/2022] Open
Abstract
BACKGROUND Neurosurgery residency programs are tasked with imparting large volumes of both clinical knowledge and technical skill to trainees in limited time. Many investigators have described local practices, which may offer evidence-based interventions in neurosurgical residency education, but this literature has not been systematically reviewed. OBJECTIVE To perform a scoping review of educational practices in neurosurgery, which are supported by quantitative, peer-reviewed research. METHODS A scoping review of the literature was performed. PubMed, Embase, and Web of Science databases were queried for articles describing educational interventions for neurosurgery residents, which included a quantitative assessment of the effect on resident performance. RESULTS From an initial set of 1785 unique articles, 29 studies were ultimately screened and included. Studies were into the following 6 topics: (1) didactics and curricula (n = 13), (2) nontechnical skills (n = 6), (3) wellness and burnout (n = 4), (4) assessment and feedback (n = 2), (5) mentorship and career development (n = 2), and (6) research (n = 2). Individual study results were described. CONCLUSION Several educational interventions in neurosurgical training are supported by quantitative evidence. Methodological shortcomings are prevalent among studies of education, particularly in the selection of meaningful outcome measures. A summary of evidence-based considerations is provided for current and future program directors.
Collapse
Affiliation(s)
- Patrick D Kelly
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Aaron M Yengo-Kahn
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Steven G Roth
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Scott L Zuckerman
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Rohan V Chitale
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - John C Wellons
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Lola B Chambless
- Department of Neurological Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|