Quality of dictated feedback associated with SIMPL operative assessments of pediatric surgical trainees

How Many Operative Performance Ratings Does a Pediatric Surgery Fellow Need to Be Deemed Practice Ready?

PURPOSE

Identifying the number of cases required for a fellow to achieve competence has been challenging. Workplace-based assessment (WBA) systems make collecting performance data practical and create the opportunity to translate WBA ratings into probabilistic statements about a fellow's likelihood of performing to a given standard on a subsequent assessment opportunity.

METHODS

We compared data from two pediatric surgery training programs that used the performance rating scale from the Society for Improving Medical Professional Learning (SIMPL). We used a Bayesian generalized linear mixed effects model to examine the relationship past and future performance for three procedures: Laparoscopic Inguinal Hernia Repair, Laparoscopic Gastrostomy Tube Placement, and Pyloromyotomy.

RESULTS

For site one, 26 faculty assessed 9 fellows on 16 procedures yielding 1094 ratings, of which 778 (71%) earned practice-ready ratings. For site two, 25 faculty rated 3 fellows on 4 unique procedures yielding 234 ratings of which 151 (65%) were deemed practice-ready. We identified similar model-based future performance expectations, with prior practice-ready ratings having a similar average effect across both sites (Site one, B = 0.25; Site two, B = 0.25). Similar prior practice-ready ratings were needed for Laparoscopic G-Tube Placement (Site one = 13; Site two = 14), while greater differences were observed for Laparoscopic Inguinal Hernia Repair (Site one = 10; Site two = 15) and Pyloromyotomy (Site one = 10; Site two = 15).

CONCLUSION

Our approach to modeling operative performance data is effective at determining future practice readiness of pediatric surgery fellows across multiple faculty and fellow groups. This method could be used to establish minimum case number requirements.

TYPE OF STUDY

Original manuscript, Study of Diagnostic Test.

LEVEL OF EVIDENCE

II.

Implementation of the SIMPL (Society for Improving Medical Professional Learning) performance assessment tool in the emergency department: A pilot study

Background

Feedback and assessment are difficult to provide in the emergency department (ED) setting despite their critical importance for competency-based education, and traditional end-of-shift evaluations (ESEs) alone may be inadequate. The SIMPL (Society for Improving Medical Professional Learning) mobile application has been successfully implemented and studied in the operative setting for surgical training programs as a point-of-care tool that incorporates three assessment scales in addition to dictated feedback. SIMPL may represent a viable tool for enhancing workplace-based feedback and assessment in emergency medicine (EM).

Methods

We implemented SIMPL at a 4-year EM residency program during a pilot study from March to June 2021 for observable activities such as medical resuscitations and related procedures. Faculty and residents underwent formal rater training prior to launch and were asked to complete surveys regarding the SIMPL app's content, usability, and future directions at the end of the pilot.

Results

A total of 36/58 (62%) of faculty completed at least one evaluation, for a total of 190 evaluations and an average of three evaluations per faculty. Faculty initiated 130/190 (68%) and residents initiated 60/190 (32%) evaluations. Ninety-one percent included dictated feedback. A total of 45/54 (83%) residents received at least one evaluation, with an average of 3.5 evaluations per resident. Residents generally agreed that SIMPL increased the quality of feedback received and that they valued dictated feedback. Residents generally did not value the numerical feedback provided from SIMPL. Relative to the residents, faculty overall responded more positively toward SIMPL. The pilot generated several suggestions to inform the optimization of the next version of SIMPL for EM training programs.

Conclusions

The SIMPL app, originally developed for use in surgical training programs, can be implemented for use in EM residency programs, has positive support from faculty, and may provide important adjunct information beyond current ESEs.

Advancements in Surgical Performance Evaluation and Feedback

PURPOSE OF REVIEW

How today's urology trainees acquire surgical skills has changed dramatically due to multiple forces placing strain on the graduate medical education mission. The development of workplace-based assessments that deliver feedback while capturing performance data has led to a paradigm shift toward individualized learning.

RECENT FINDINGS

Delivering feedback that drives surgical skill development requires the educator to provide a meaningful assessment of the learner after an operative experience. Workplace-based assessment involves direct observation of routine clinical practice and has become a central component of competency-based medical education. Urology has the chance to fully embrace competency-based medical education, employing robust feedback mechanisms and workplace-based assessments. We must first define what it means to be a proficient urologist and design an assessment system that captures this collective sentiment. This can only be done through effective engagement and collaboration with stakeholders across our specialty.

A Document Analysis of Nationally Available Faculty Assessment Forms of Resident Performance

BACKGROUND

Written feedback by faculty of resident performance is valuable when it includes components based on assessment for learning. However, it is not clear how often assessment forms include these components for summative and formative feedback.

OBJECTIVE

To analyze prompts used in forms for faculty assessment of resident performance, guided by best practices in survey research methodology, self-regulation theory, and competency-based assessment.

METHODS

A document analysis, which is a qualitative approach used to analyze content and structure of texts, was completed on assessment forms nationally available in MedHub. Due to the number of forms available, only internal medicine and surgery specialties were included. A document summary form was created to analyze the assessments. The summary form guided researchers through the analysis.

RESULTS

Forty-eight forms were reviewed, each from a unique residency program. All forms provided a textbox for comments, and 54% made this textbox required for assessment completion. Eighty-three percent of assessments placed the open textbox at the end of the form. One-third of forms contained a simple prompt, "Comments," for the narrative section. Fifteen percent of forms included a box to check if the information on the form had been discussed with the resident. Fifty percent of the assessments were unclear if they were meant to be formative or summative in nature.

CONCLUSIONS

Our document analysis of assessment forms revealed they do not always follow best practices in survey design for narrative sections, nor do they universally address elements deemed important for promotion of self-regulation and competency-based assessment.

Data-Driven Residency Training: A Scoping Review of Educational Interventions for Neurosurgery Residency Programs

BACKGROUND

Neurosurgery residency programs are tasked with imparting large volumes of both clinical knowledge and technical skill to trainees in limited time. Many investigators have described local practices, which may offer evidence-based interventions in neurosurgical residency education, but this literature has not been systematically reviewed.

OBJECTIVE

To perform a scoping review of educational practices in neurosurgery, which are supported by quantitative, peer-reviewed research.

METHODS

A scoping review of the literature was performed. PubMed, Embase, and Web of Science databases were queried for articles describing educational interventions for neurosurgery residents, which included a quantitative assessment of the effect on resident performance.

RESULTS

From an initial set of 1785 unique articles, 29 studies were ultimately screened and included. Studies were into the following 6 topics: (1) didactics and curricula (n = 13), (2) nontechnical skills (n = 6), (3) wellness and burnout (n = 4), (4) assessment and feedback (n = 2), (5) mentorship and career development (n = 2), and (6) research (n = 2). Individual study results were described.

CONCLUSION

Several educational interventions in neurosurgical training are supported by quantitative evidence. Methodological shortcomings are prevalent among studies of education, particularly in the selection of meaningful outcome measures. A summary of evidence-based considerations is provided for current and future program directors.