The reliability of resident self-evaluation of operative performance

Effectiveness of donning and doffing personal protective equipment education using video debriefing among Korean undergraduate nursing students

BACKGROUND

The use of appropriate personal protective equipment (PPE) in healthcare settings is a very important factor of infection control activities to prevent transmission of infection. In this study, we aimed to develop a non-face-to-face method that can reduce transmission of infection and an educational method that allows self-learning and self-training, and verify the effectiveness of a donning and doffing PPE education using video debriefing on third-year nursing students in clinical train.

METHODS

The researcher assigned groups through random sampling using a random number table from the list of research subjects who agreed to participate. Therefore, 35 subjects in the experimental group with an education on donning and doffing PPE using video debriefing and 34 subjects in the control group with video-based training (VBT) were participated in April 2023 at a university in Incheon, South Korea. The evaluation phase involved verifying effects of the education on observer evaluation donningand doffing PPE performance and self-evaluation donningand doffing PPE performance. Satisfaction with the education also assessed immediately after intervention by the experimental group.

RESULTS

The experimental group had significantly higher scores for observer evaluationdonning (t = 7.84, p < .001) and observer evaluation doffing (t = 6.52, p < .001) PPE performances than the control group. However, there was no significant difference in self-evaluation donning (t = 0.98, p < .330) or doffing PPE performance (t = 0.17, p < .869) between the two groups. In the experimental group that applied video debriefing, learning satisfaction was 4.33 ± 0.82 points out of 5 points.

CONCLUSIONS

The education on donning and doffing PPE applying video debriefing was effective. If the education developed in this study is used as a self-learning method to improve the infection control capacity necessary to respond to a disaster crisis caused by infectious diseases, it is considered to be useful for preventing the transmission of infectious diseases in the future.

Reliability of a workplace-based assessment for the United States general surgical trainees’ intraoperative performance using multivariate generalizability theory: a psychometric study

PURPOSE

The System for Improving and Measuring Procedure Learning (SIMPL), a smartphone-based operative assessment application, was developed to assess the intraoperative performance of surgical residents. This study aims to examine the reliability of the SIMPL assessment and determine the optimal number of procedures for a reliable assessment.

METHODS

In this retrospective observational study, we analyzed data collected between 2015 and 2023 from 4,616 residents across 94 General Surgery Residency programs in the United States that utilized the SIMPL smartphone application. We employed multivariate generalizability theory and initially conducted generalizability studies to estimate the variance components associated with procedures. We then performed decision studies to estimate the reliability coefficient and the minimum number of procedures required for a reproducible assessment.

RESULTS

We estimated that the reliability of the assessment of surgical trainees’ intraoperative autonomy and performance using SIMPL exceeded 0.70. Additionally, the optimal number of procedures required for a reproducible assessment was 10, 17, 15, and 17 for postgraduate year (PGY) 2, PGY 3, PGY 4, and PGY 5, respectively. Notably, the study highlighted that the assessment of residents in their senior years necessitated a larger number of procedures compared to those in their junior years.

CONCLUSION

The study demonstrated that the SIMPL assessment is reliably effective for evaluating the intraoperative performance of surgical trainees. Adjusting the number of procedures based on the trainees’ training stage enhances the assessment process’s accuracy and effectiveness.

Evaluating Performance and Autonomy Levels of Previous Preliminary Surgery Interns

INTRODUCTION

Surgery residents who complete a nondesignated preliminary year have an additional year of training compared to those who begin as categorical residents. While this additional year is sometimes perceived negatively, these more experienced residents may outperform traditional categorical (TC) residents in their first year of training.

METHODS

Operative assessment ratings were recorded for first year categorical general surgery residents in the United States between 2015 and 2023 using the Society for Improving Medical and Professional Learning assessment platform. Residents were categorized based on the completion of a nondesignated preliminary year ("Previous Prelim" [PP]) or not ("Traditional Categorical"). Ratings were analyzed using generalized mixed effects models. Performance and autonomy outcomes were dichotomized: "less experience" or "more experience" and "no autonomy" or "some autonomy", respectively. Fixed effects included academic month and case complexity, while random effects included resident, faculty, program, and procedure.

RESULTS

A total of 34,353 evaluations from 86 general surgery programs were collected. Of these, 829 were evaluations from PP residents. Faculty ratings of PP versus TC revealed no differences in adjusted probabilities of achieving a "more experience" rating (82% versus 76%, P = 0.26) but a higher adjusted probability of achieving a "some autonomy" rating (88% versus 80%, P = 0.04) for PP compared to TC. Analysis of resident self-reported ratings revealed higher adjusted probabilities of a "more experience" rating (77% versus 50%, P = 0.01) and "some autonomy" rating (87% versus 73%; P = 0.02) for PP compared to TC.

CONCLUSIONS

First year general surgery residents who previously completed a preliminary year have similar operative performance faculty ratings when compared to their peers.

Study of surgical performance during clavicle plate placements using 2 learning methods: naive practice versus deliberate practice

BACKGROUND

Companionship and simple experience or naive practice (NP) rarely lead to expert level surgery, in contrast to deliberate practice (DP) where an expert analyzes the learner's errors and sets goals to improve performance. The main hypothesis was that using DP for learning would result in faster and/or greater progress than using NP.

OBJECTIVES

The objective of this work was to compare the evolution of the learning curve for clavicle locking plate placement on a sawbone model of a clavicle fracture, by surgical trainees learning via two different methods; NP and DP.

PATIENTS AND METHODS

Ten surgical residents, divided into 2 groups of 5, each placed 6 plates. The 6 trials were filmed. The NP group saw an expert video before each placement. The DP group saw this video once and then received personalized advice from the expert for improvement, by analyzing their own video after each subsequent trial. Objective performance (OP) was measured by a standardized evaluation grid (OSATS, with a score ranging from 10 to 50 points per trial), self-evaluation of performance by a numerical scale (from 0 to 10) and stress by an analgesia-nociception index (ANI, calculated by heart rate recording, from 0 to 100).

RESULTS

The mean OP at the last trial of clavicle plate placement was 41.8 (NP group) and 48.2 (DP group), with a mean progression from the first to last trials of 0.8 in the NP group, and 5.1 in the DP group. The mean progression in self-evaluation between the first and last trials was 3.4 (NP group) and 4.6 (DP group). The mean progression of the ANI between the first and last trials was -4.5 (NP group) and +5 (DP group).

DISCUSSION

The results of learning a clavicle plate osteosynthesis technique measured by OSATS were better with deliberate practice than with naive practice. The progression in self-evaluated performance was better with deliberate practice, but with a higher stress level.

CONCLUSION

Deliberate practice is a technique for learning the surgical procedure which complements companionship and experience. It shortens the learning curve and improves the level of performance of surgical trainees.

LEVEL OF EVIDENCE

IV; non-interventional research.

Through the Looking Glass: Comparing Resident and Attending Views on Competency and Autonomy

BACKGROUND

Surgical residencies act as an apprenticeship to train new surgeons, ideally with increasing trainee autonomy and competency. Objective methods of measuring these two qualities are needed.

METHODS

The objective of this pilot study is to use the Surgical Autonomy Program to compare resident and attending perception of competency, via Zones of Proximal Development (ZPDs), and autonomy, via Teach, Advise, Guide, Solo (TAGS) scale for a common pediatric neurosurgical procedure. The most common pediatric neurosurgical procedure, a ventriculoperitoneal shunt, was broken down into four ZPDs. After each procedure, the resident and the attending rated the competency of the resident by ZPD and their autonomy using TAGS. These ratings were then compared to see if the attending and the resident agreed on their levels of competency and autonomy.

RESULTS

There were no significant differences in the rating of competency or autonomy between the residents and the attendings. Postgraduate year was significantly associated with differences in TAGS ratings with those having less experience showing a positive perception gap. Other covariates (age, gender) were not associated with differences in rating. The residents found that the ZPDs checklist was a useful tool for communicating with the attending and for getting appropriate autonomy in the operating room.

CONCLUSIONS

Pediatric neurosurgical trainees and attendings had concordance on their perceptions of resident autonomy and competency, suggesting that this tool could be a useful way of objectively grading the progress of neurosurgical residents in surgery.

The Importance of Multiorgan Procurement in the Improvement of Residents' Open Surgical Skills

INTRODUCTION

The multiorgan procurement (MOP) represents a chance for the general surgery resident to learn the fundamental steps of open abdominal surgery. The objective of this study was to evaluate the impact of MOP on the residents' open surgical skills.

METHODS

Residents' surgical skills were assessed during a 6-month transplant rotation (October 2020-March 2021) using a modified Objective Structured Assessment of Technical Skills with the global rating scale. The surgeries were self-assessed by residents and tutors based on 9 specific steps (SS) and 4 general skills (GS). Each item was rated from 1 (poor) to 5 (excellent) with a maximum score of 45 points for SS and 20 for GS. A crossed-effects linear regression analysis was performed both to evaluate any associations between GS/SS scores and some prespecified covariates, and to study differences in the assessments performed by residents and tutors.

RESULTS

Residents actively participated in a total of 59 procurements. In general, there were no significant differences in SS/GS mean scorings between residents (n = 15) and tutors (n = 5). There was a significantly positive association between mean GS/SS scorings and the number of donor surgeries performed (at least 5). Comparing the evaluations of the tutors with the residents, this significance was retained only when scorings were assigned by the tutors.

CONCLUSIONS

MOP was shown to improve basic open surgical skills among residents. Awareness of the utility of a clinical rotation in transplant surgery should be raised also on an institutional level.

A scoping review of self-monitoring in graduate medical education

BACKGROUND

Physicians and physicians-in-training have repeatedly demonstrated poor accuracy of global self-assessments, which are assessments removed from the context of a specific task, regardless of any intervention. Self-monitoring, an in-the-moment self-awareness of one's performance, offers a promising alternative to global self-assessment. The purpose of this scoping review is to better understand the state of self-monitoring in graduate medical education.

METHODS

We performed a scoping review following Arksey and O'Malley's six steps: identifying a research question, identifying relevant studies, selecting included studies, charting the data, collating and summarising the results and consulting experts. Our search queried Ovid Medline, Web of Science, PsychINFO, Eric and EMBASE databases from 1 January 1999 to 12 October 2022.

RESULTS

The literature search yielded 5363 unique articles. The authors identified 77 articles for inclusion. The search process helped create a framework to identify self-monitoring based on time and context dependence. More than 20 different terms were used to describe self-monitoring, and only 13 studies (17%) provided a definition for the equivalent term. Most research focused on post-performance self-judgements of a procedural skill (n = 31, 42%). Regardless of task, studies focused on self-judgement (n = 66, 86%) and measured the accuracy or impact on performance of self-monitoring (n = 41, 71%). Most self-monitoring was conducted post-task (n = 65, 84%).

CONCLUSION

Self-monitoring is a time- and context-dependent phenomenon that seems promising as a research focus to improve clinical performance of trainees in graduate medical education and beyond. The landscape of current literature on self-monitoring is sparse and heterogeneous, suffering from a lack of theoretical underpinning, inconsistent terminology and insufficiently clear definitions.

Validation of a simplified evaluation grid applied to the microsurgery on inert material via smartphone

OBJECTIVES

Microsurgery is usually performed in experimental research models and clinical surgery. It requires meticulous technical skills and continuous training. Inert materials such as Shirataki noodles are readily available and low-cost consumables regularly used for practice. The objective of this study was to evaluate the repeatability and reproducibility of a simplified evaluation grid of suture on inert material (Shirataki Konnyaku noodle) under smartphone magnification.

MATERIAL AND METHODS

Ten students performed end-to-end suture on inert material with magnification via their smartphone. Each suture was filmed, and the videos were randomized. Each student was evaluated on each video three times over three consecutive days, using a simplified evaluation grid. Intra- and inter-observer agreement was evaluated on Concordance Correlation Coefficients. Values were assessed on Pearson's correlation coefficient.

RESULTS

Intra-observer correlation was weak for 2 items (0.288 and 0.246) and moderate for the other 2 (0.419 and 0.529). Inter-observer correlation was weak for 3 items (0.344, 0.358, and 0.276) and close to zero for the other (0.034).

CONCLUSION

This simplified evaluation grid for microsurgery training on inert material via smartphone was poorly repeatable and reproducible. Loss of certain items in the grid due to the use of inert material probably impaired relevance.

The Program Director Perspective: Perceptions of PGY5 Residents' Operative Self-Efficacy and Entrustment

INTRODUCTION

A 2020 survey of post-graduate year 5 (PGY5) general surgery residents linked to the American Board of Surgery In-Training Examination (ABSITE) revealed significant deficits in self-efficacy (SE), or personal judgment of one's ability to complete a task, for 10 commonly performed operations. Identifying whether this deficit is similarly perceived by program directors (PDs) has not been well established. We hypothesized that PDs would perceive higher levels of operative SE compared to PGY5s.

METHODS

A survey was distributed through the Association of Program Directors in Surgery listserv; PDs were queried about their PGY5 residents' ability to perform the same 10 operations independently and their accuracy of patient assessments and operative plans for components of several core entrustable professional activities (EPAs). Results of this survey were compared to PGY5 residents' perception of their SE and entrustment based on the 2020 post-ABSITE survey. Chi-squared tests were used for statistical analysis.

RESULTS

108 responses were received, representing ∼32% (108/342) of general surgery programs. Perceptions from PDs of PGY5 residents' operative SE were highly concordant with resident perceptions; no significant differences were observed for 9 of 10 procedures. Both PGY5 residents and PDs perceived adequate levels of entrustment; no significant differences were observed for 6 of 8 EPA components.

CONCLUSIONS

These findings show concordance between PDs and PGY5 residents in their perceptions of operative SE and entrustment. Though both groups perceive adequate levels of entrustment, PDs corroborate the previously described operative SE deficit, illustrating the importance of improved preparation for independent practice.

Anterior plating technique for distal radius: comparing performance after learning through naive versus deliberate practice

BACKGROUND

Surgical teaching is most often carried out in the operating theatre through mentorship, and the performance of surgical procedures is rarely measured. The objective of this article is to compare the progression in learning curves of junior surgeons trained in the anterior plating technique for the distal radius on a nonbiological model according to three different methods.

METHODS

The materials comprised 12 junior surgeons of level 1 or 2 (as per Tang and Giddins) divided into three groups: control (G1), naive practice (G2), and deliberate practice (G3). The three groups watched a demonstration video of a level 5 expert. The four G1 surgeons (two level 1 and two level 2) saw the video only once, and each inserted five plates. The four G2 surgeons (two level 1 and two level 2) inserted five plates and watched the video before each time. The four G3 surgeons (two level 1 and two level 2) saw the video before the first plate insertion. Before posing the subsequent four plates, the four G3 surgeons watched their own video, and the expert indicated their errors and how to avoid them next time. A 12-criteria OSATS defined on the basis of the 60 videos, each graded from one (min.) to five (max.), was used to measure the objective surgical performance per plating (min. 12; max. 60) and per series of five plate fixations (min. 60, max. 300).

RESULTS

The total average objective performance of G1 was 44.73, of G2 was 50.57 and of G3 was 54.35. Change in objective performance was better for G3 (13.25) than G2 (5) or G1 (3.75). For all groups, the progression in objective performance was better amongst level 1 surgeons (9) than level 2 surgeons (5.6).

CONCLUSION

Surgical teaching is based on mentorship and experience. However, since "see one, practice many, do one" has started to replace "see one, do one, teach one", learning techniques have increasingly relied on procedure simulators. Against this background, few studies have looked at measuring the performance of surgical procedures and improved learning curves. Our results appear to suggest that deliberate practice, when used in addition to mentorship, is the best option for shortening the growth phase of the learning curve and improving performance. Deliberate practice is a learning technique for surgical procedures that is complementary to mentorship and experience, which allows the growth phase of the learning curve to be shortened and the objective performance of junior surgeons to be improved.

Multi-institutional Study of Otolaryngology Resident Intraoperative Experiences for Key Indicator Procedures

OBJECTIVE

There is concern that current otolaryngology residents may not receive adequate surgical training. We aimed to characterize residents' surgical experiences at 5 academic centers performing the 14 key indicator procedures (KIPs) outlined by the Accreditation Council for Graduate Medical Education.

STUDY DESIGN

Prospective study.

SETTING

Five otolaryngology training programs.

METHODS

Data were gathered from December 2019 to December 2020 with a smartphone application from the Society for Improving Medical Professional Learning. After each operation, residents and faculty rated trainee autonomy on a 4-level Zwisch scale and performance on a 5-level modified Dreyfus scale.

RESULTS

Residents and attendings (n = 92 and 78, respectively) logged 2984 evaluations. Attending ratings of resident autonomy and performance increased with training level (P < .001). Resident self-assessments of autonomy and performance were lower than paired attending assessments (P < .001). Among attending evaluations of KIPs performed by senior residents (postgraduate year 4 or 5), 55% of cases were performed with meaningful autonomy (passive help or supervision only). Similarly, attendings rated 55% of these cases as a practice-ready or exceptional performance. Senior residents had meaningful autonomy for ≥50% of cases for most KIPs, with the exception of flaps and grafts (40%), pediatric/adult airway (39%), and stapedectomy/ossiculoplasty (33%). Similarly, senior residents received practice-ready or exceptional performance ratings for ≥50% of cases across all KIPs other than pediatric/adult airway (42%) and stapedectomy/ossiculoplasty (33%).

CONCLUSION

In this multicenter study, resident surgical autonomy and performance varied across otolaryngology KIPs. The development of nationwide benchmarks will help programs and residents set educational goals.

LEVEL OF EVIDENCE

2.