Resident Surgeons Underrate Their Laparoscopic Skills and Comfort Level When Compared With the Rating by Attending Surgeons

Through the Looking Glass: Comparing Resident and Attending Views on Competency and Autonomy

BACKGROUND

Surgical residencies act as an apprenticeship to train new surgeons, ideally with increasing trainee autonomy and competency. Objective methods of measuring these two qualities are needed.

METHODS

The objective of this pilot study is to use the Surgical Autonomy Program to compare resident and attending perception of competency, via Zones of Proximal Development (ZPDs), and autonomy, via Teach, Advise, Guide, Solo (TAGS) scale for a common pediatric neurosurgical procedure. The most common pediatric neurosurgical procedure, a ventriculoperitoneal shunt, was broken down into four ZPDs. After each procedure, the resident and the attending rated the competency of the resident by ZPD and their autonomy using TAGS. These ratings were then compared to see if the attending and the resident agreed on their levels of competency and autonomy.

RESULTS

There were no significant differences in the rating of competency or autonomy between the residents and the attendings. Postgraduate year was significantly associated with differences in TAGS ratings with those having less experience showing a positive perception gap. Other covariates (age, gender) were not associated with differences in rating. The residents found that the ZPDs checklist was a useful tool for communicating with the attending and for getting appropriate autonomy in the operating room.

CONCLUSIONS

Pediatric neurosurgical trainees and attendings had concordance on their perceptions of resident autonomy and competency, suggesting that this tool could be a useful way of objectively grading the progress of neurosurgical residents in surgery.

Implementation and Validation of an Automated, Longitudinal Robotic Surgical Evaluation and Feedback Program at a High-volume Center and Impact on Training

Background

Surgical education lacks a standardized, proficiency-based approach to evaluation and feedback.

Objective

To assess the implementation and reception (ie, feasibility) of an automated, standardized, longitudinal surgical skill assessment and feedback system, and identify baseline trainee (resident and fellow) characteristics associated with achieving proficiency in robotic surgery while learning robotic-assisted laparoscopic prostatectomy.

Design setting and participants

A quality improvement study assessing a pilot of a surgical experience tracking program was conducted over 1 yr. Participants were six fellows, eight residents, and nine attending surgeons at a tertiary cancer center.

Intervention

Trainees underwent baseline self-assessment. After each surgery, an evaluation was completed independently by the trainee and attending surgeons. Performance was rated on a five-point anchored Likert scale (trainees were considered "proficient" when attending surgeons' rating was ≥4). Technical skills were assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) and Prostatectomy Assessment and Competency Evaluation (PACE).

Outcome measurements and statistical analysis

Program success and utility were assessed by evaluating completion rates, evaluation completion times, and concordance rates between attending and trainee surgeons, and exit surveys. Baseline characteristics were assessed to determine associations with achieving proficiency.

Results and limitations

Completion rates for trainees and attending surgeons were 72% and 77%, respectively. Fellows performed more steps/cases than residents (median [interquartile range]: 5 [3-7] and 3 [2-4], respectively; p < 0.01). Prior completion of robotics or laparoscopic skill courses and surgical experience measures were associated with achieving proficiency in multiple surgical steps and GEARS domains. Interclass correlation coefficients on individual components were 0.27-0.47 on GEARS domains.

Conclusions

An automated surgical experience tracker with structured, longitudinal evaluation and feedback can be implemented with good participation and minimal participant time commitment, and can guide curricular development in a proficiency-based education program by identifying modifiable factors associated with proficiency, individualizing education, and identifying improvement areas within the education program.

Patient summary

An automated, standardized, longitudinal surgical skill assessment and feedback system can be implemented successfully in surgical education settings and used to inform education plans and predict trainee proficiency.

A Study of Surgical Residents' Self-Assessment of Open Surgery Skills Using Gap Analysis

BACKGROUND

Surgical residents need structured and objective feedback to develop their skills and become capable of performing surgical procedures autonomously. A shortage of experienced surgical staff has prompted residents to seek feedback from self-assessment and peer assessments.

OBJECTIVES

We investigated whether surgical residents can reliably rate their own and their peers' basic surgical skills using the Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills.

METHODS

The study was a prospective and descriptive study conducted using gap analysis at the Copenhagen Academy for Medical Education and Simulation (Copenhagen, Denmark) from 2016 to 2017. Surgical residents were recruited during a course in basic open surgical skills. Among 102 course participants, 53 met the inclusion criteria and 22 participated in the study.

RESULTS

We recruited surgical residents based in the Capital Region and Zealand Region of Demark, and 42% of eligible residents participated in the study. Surgical residents underestimated their own surgical performance (median, 17 [range, 15-18] vs. 20 [range, 19.75-22]; P < 0.001). They also rated their peers lower than an experienced rater did (median 10 [range, 8.75-14] vs. 15 and median 20.5 [range, 19-22] vs. 23; both P < 0.001). Gap analysis revealed residents had unrecognized strengths (ie, self-underappraisal) in most GRS domains.

CONCLUSIONS

Ratings are unreliable when surgical residents assess their own and their peers' performances using GRS. A gap analysis revealed unrecognized strengths in time and motion, instrument handling, knowledge of instruments and sutures, and knowledge of specific procedure as well as unrecognized weaknesses in flow of operation and forward planning.

A scoping review of self-monitoring in graduate medical education

BACKGROUND

Physicians and physicians-in-training have repeatedly demonstrated poor accuracy of global self-assessments, which are assessments removed from the context of a specific task, regardless of any intervention. Self-monitoring, an in-the-moment self-awareness of one's performance, offers a promising alternative to global self-assessment. The purpose of this scoping review is to better understand the state of self-monitoring in graduate medical education.

METHODS

We performed a scoping review following Arksey and O'Malley's six steps: identifying a research question, identifying relevant studies, selecting included studies, charting the data, collating and summarising the results and consulting experts. Our search queried Ovid Medline, Web of Science, PsychINFO, Eric and EMBASE databases from 1 January 1999 to 12 October 2022.

RESULTS

The literature search yielded 5363 unique articles. The authors identified 77 articles for inclusion. The search process helped create a framework to identify self-monitoring based on time and context dependence. More than 20 different terms were used to describe self-monitoring, and only 13 studies (17%) provided a definition for the equivalent term. Most research focused on post-performance self-judgements of a procedural skill (n = 31, 42%). Regardless of task, studies focused on self-judgement (n = 66, 86%) and measured the accuracy or impact on performance of self-monitoring (n = 41, 71%). Most self-monitoring was conducted post-task (n = 65, 84%).

CONCLUSION

Self-monitoring is a time- and context-dependent phenomenon that seems promising as a research focus to improve clinical performance of trainees in graduate medical education and beyond. The landscape of current literature on self-monitoring is sparse and heterogeneous, suffering from a lack of theoretical underpinning, inconsistent terminology and insufficiently clear definitions.

Video-based learning of coping strategies for common errors improves laparoscopy training-a randomized study

AIMS

The aim of this study was to investigate whether shifting the focus to solution orientation and developing coping strategies for common errors could increase the efficiency of laparoscopic training and influence learning motivation. The concept of coping has been particularly defined by the psychologist Richard Lazarus [Lazarus and Folkman in Stress, appraisal, and coping, Springer publishing company, New York, 1984]. Based on this model, we examined the use of observational learning with a coping model for its effectiveness as a basic teaching model in laparoscopic training.

METHODS

55 laparoscopically naive medical students learned a standardized laparoscopic knot tying technique with video-based instructions. The control group was only offered a mastery video that showed the ideal technique and was free from mistakes. The intervention group was instructed on active error analysis and watched freely selectable videos of common errors including solution strategies (coping model) in addition to the mastery videos.

RESULTS

There was no statistically significant difference between the intervention and control groups for number of knot tying attempts until proficiency was reached (18.8 ± 5.5 vs. 21.3 ± 6.5, p = 0.142). However, there was a significantly higher fraction of knots achieving technical proficiency in the intervention group after first use of the coping model (0.7 ± 0.1 vs. 0.6 ± 0.2, p = 0.026). Additionally, the proportion of blinded attempts that met the criteria for technical proficiency was significantly higher for the intervention group at 60.9% vs. 38.0% in control group (p = 0.021). The motivational subscore "interest" of the validated score on current motivation (QCM) was significantly higher for the intervention group (p = 0.032), as well as subjective learning benefit (p = 0.002) and error awareness (p < 0.001).

CONCLUSION

Using video-based learning of coping strategies for common errors improves learning motivation and understanding of the technique with a significant difference in its qualitative implementation in laparoscopy training. The ability to think in a solution-oriented, independent way is necessary in surgery in order to recognize and adequately deal with technical difficulties and complications.

Differences in perceptions of capability, autonomy, and expectations between residents and surgical team members in executing EPAs in Ethiopian medical education

Background

Entrustable Professional Activities (EPAs) are units of professional practice that are defined as tasks or responsibilities that are entrusted to an unsupervised execution by a trainee. In 2021, a framework of 29 EPAs was developed for surgical residency training programs in Ethiopia, with the goal of residents being able to perform independently by the time they graduate. However, studies show that surgical residents lack confidence and are unable to execute EPAs autonomously upon graduation, and concerns have been raised about graduate competencies in EPA execution. The goal of this research is to assess how surgical team members judge/perceive residents' performance in executing these EPAs autonomously at the time of graduation and how residents rate their own capability and autonomy in executing EPAs in order to systematically introduce and implement EPAs in Ethiopian medical education.

Methods

A survey was conducted in the Departments of Surgery at four residency training institutions in Ethiopia. All eligible surgical team members and final-year general surgery residents were invited to participate. Surgical team members were asked to rate the observed performance of a group of graduating surgical residents in each of the 29 EPAs, and residents were asked to rate their own capability in executing EPAs. The analysis focused on variations in performance ratings between surgical team members and residents, as well as across surgical team members.

Results

A total of 125 surgical team members and 49 residents participated in this study. Residents rate their competence in performing these EPAs higher than surgical team members, mean 4.2 (SD = 0.63) vs. 3.7 (SD = 0.9). A statistically significant difference in perceptions of capability, autonomy, and expectations in executing EPAs was observed between the two groups of study (p = 0.03, CI: 0.51-0.95), as well as within surgical team members (p < 0.001).

Conclusions

Differences in perceptions of capability, autonomy, and expectations between residents and surgical team members, as well as within faculty members, were seen in executing EPAs. There were concerns about graduate surgical residents' competence to execute EPAs autonomously at the time of graduation. Surgical team members perceived that a set of graduating surgical residents are not yet safe to perform these EPAs independently (without supervision) and still requires distant supervision.

Using Self-Rated Examinations to Ensure Competence in Laparoscopy

INTRODUCTION

Asking participants to rate their own performance during unsupervised training in laparoscopy is reliable and may be cost-effective. The objective of the study was to explore the reliability of self-rated examinations where participants rate their own performance and decide themselves when they have passed tasks in basic laparoscopic skills.

METHODS

This prospective observational study was conducted at the Copenhagen Academy for Medical Education and Simulation where simulation-based laparoscopic skill training is offered. Here, participants taking part in a basic laparoscopic skills course were asked to rate their own performance and decide when they had passed the Training and Assessment of Basic Laparoscopic Techniques test. To explore reliability, all examinations were video recorded and rated by a blinded rater after the end of the course.

RESULTS

Thirty-two surgical trainees participated in the course, and 28 completed the study. We found a high reliability when comparing self-rated scores and blinded ratings with an intraclass correlation coefficient of 0.89 ( P < 0.001); self-rated scores compared with blinded ratings were not significantly different (mean = 451 vs. 455, P = 0.28), and the participants did not underestimate nor overestimate their performance.

CONCLUSIONS

Ratings from self-rated examinations in a basic laparoscopic skills course are reliable, and participants neither underestimate nor overestimate their performance. Self-rated examinations may also be beneficial because they also can offer a cost-effective approach to assessment of surgical trainees.

How Do Resident Surgeons Identify Operative Case Complexity? An Analysis of Resident versus Attending Perceptions

OBJECTIVE

Accurate recognition of patient-related complexity of an operation is critical for appropriate surgical decision making. It is not yet understood whether general surgery residents are able to accurately assess the relative complexity of a given operative case. This study investigates the agreement of case complexity ratings between residents and attending surgeons and explores whether resident-related factors correlate with any discordance in perception of patient-related operative complexity.

DESIGN

Residents and attending surgeons rated the relative complexity of completed cases on a 3 point scale via the SIMPL (Society for Improving Medical Professional Learning) operative assessment smartphone app. Additional trainee demographic data, autonomy ratings, and performance ratings were also obtained from the SIMPL registry for each rated case. Complexity agreement was defined as an equal rating between the resident and attending and assigned a value of zero. Over-estimate ratings were assigned a positive value and under-estimate ratings were assigned a negative value. Trends in complexity agreement were analyzed using descriptive statistics and mixed-effects models.

RESULTS

A total of 43,179 general surgery cases were rated by 1946 categorical general surgery residents and 1520 attending surgeons between 2015 and 2020. Residents and attendings agreed on case complexity in 63.23% of cases, while the residents overestimated complexity in 13.37% of cases and underestimated complexity in 23.40% of cases. Every level of resident except post-graduate year 2 had similar rates of agreement about the complexity of a procedure, while residents who received a higher autonomy rating were more likely to be in agreement with the faculty raters (OR 1.12, 95% CI 1.06-1.19).

CONCLUSIONS

The results of this study suggest that general surgery residents inaccurately perceive the patient-related complexity of a given case approximately one third of the time. Greater experience and operative autonomy appear to be associated with higher complexity agreement. Future research into factors influencing perceived case complexity may provide insight into how to best implement new teaching for surgical residents regarding the concept of case complexity.

Simulation-based structured training for developing laparoscopy skills in general surgery and obstetrics & gynecology postgraduates

BACKGROUND

Simulation-based training is an important strategy for skill development in a competency-based curriculum, especially so for laparoscopic surgery given its unique learning curve, need for practice, and patient safety concerns. The study was conducted for postgraduates in two surgical disciplines in a medical college tertiary care center. The study evaluates the acceptability and utility of structured simulation-based training for laparoscopic skills. Simulations provide deliberate practice in the leaner supportive environment.

MATERIALSAND METHODS

The educational intervention was carried out among the 16 postgraduates from year 2 and year 3 of general surgery and obstetrics and gynecology. A structured training and assessment module was designed and validated, and a 12-week structured training on laparoscopic box trainers was given to postgraduates under faculty guidance. Feedback from postgraduates and faculty was obtained using separate validated questionnaires. The pre- and post-training assessment scores were compared applying the Wilcoxon matched-pairs signed-rank (2-tailed) test using the SPSS Windows Version 20.0 (IBM Corporation).

RESULTS

Sixteen postgraduates participated in the study. The median satisfaction score of the postgraduates was 4 (range 3-5), a scale of a maximum of 5. The group identified faculty feedback, stepwise skills learning, supportive learning environment, and trainers' motivation as enabling factors in the training. The faculty observed that the trainees were effectively engaged. The group mean improvement scores in percentage were 40.08 for task 1, 42.08 for task 2, 43.13 for task 3, and 45.63 for task 4.

CONCLUSIONS

It is feasible to incorporate faculty-guided sessions of simulation training in basic laparoscopy skills for postgraduates in our setup. It is well accepted by the key stakeholders, and we recommend it to be incorporated in the formal training program.

Temporal changes in medical student perceptions of their clinical skills and needs using a repeated self-assessment instrument

BACKGROUND

Medical student needs in clinical skill training may change over time, but data on this topic are limited. This study uses repeated self-assessments on clinical rotations during medical school to evaluate students' perceptions of their clinical skill growth.

METHODS

A self-assessment rating was completed by students during each clinical rotation as they progressed through their core clinical rotation year. The instrument consisted of questions on 5 clinical skill categories where students rated their performance as "below", "at" or "above" expected, and open-ended questions on strengths and challenges. We evaluated changes in self-ratings between the first (n=136) and third (n=118) quarters by matched-pair analysis of the shift in responses between time points using a Sign Test. We also identified the main themes from the students' responses to open-ended questions.

RESULTS

We found 22.4 % and 13.3 % of students increased their self-assessment ratings on "Oral Presentation Skills" and on "Differential Diagnosis", respectively. In contrast, perceived ability to communicate with patients saw the largest negative shifts. "Patient Interaction" was the most commonly identified area of strength and "Knowledge and Organization" was most frequently cited as a barrier.

CONCLUSIONS

Students demonstrated a positive shift in perceived competence in some core clinical skills that are not strongly emphasized in the preclinical curriculum, likely reflecting increased exposure over time. However, their perceived competence in communication skills declined. This may reflect initial over-estimation or true decline due to competing needs/interests. These patterns of change can inform the design of longitudinal curricula that anticipate and address students' needs during clinical rotations, such as placing increased emphasis on presentation skills and differential diagnosis earlier in the curriculum, and adding more emphasis to communication skills in later phases.

Educational Research Report Correlation between Student Self-Assessment and Proctor Evaluation in a Veterinary Surgical Laboratory

Self-assessment has been shown to facilitate learning, goal setting, and professional development. We sought to evaluate whether veterinary students in a surgical curriculum would have self-assessments that differed from proctor evaluations and whether high-performing students would differ from low-performing students in self-assessment characteristics. Student and proctor assessments were compared for 8 weeks of a spay/neuter surgical laboratory taking place in the second year of the curriculum. Eight students were classified as high-performing, and 10 students were classified as low-performing. A quantitative evaluation of the scores and a qualitative assessment of written comments were completed. Proctors assigned higher scores to high-performing students compared to low-performing students, but no difference was observed overall in self-assessment scores assigned by students. When only anesthesia students were evaluated, we found a difference in self-assessment scores for high- versus low-performers, but this was not true for surgery students. Differences between proctor and student assessment scores diminished over time for all students and for anesthesia students, but not for surgery students. High-performing student anesthetists self-assessed and received proctor assessments with higher scores in technical skills. Comments from high-performing students tended to be less reflective and more positive. Low-performing students were more defensive and more likely to use I-statements in their comments. Overall, quantitative analysis did not reveal a difference in self-assessment between high-performers and low-performers; however, specific differences existed in qualitative characteristics, surgery versus anesthesia students, and proctor assessments. The differences in self-assessment between high- and low-performing students suggest areas of further investigation for the use of reflection in education.

Performance of Residents During Laparoscopic Cholecystectomy: Is Self-assessment Reliable?

BACKGROUND

To compare the self-assessment of surgical residents and observers (faculty members and nurses) during laparoscopic cholecystectomy (LC).

MATERIALS AND METHODS

A prospective observational study was conducted between February 2020 and July 2020 at a medical school hospital. Seventy-four LC surgeries were performed by surgical residents in the presence of faculty members. A self-assessment of the technical and nontechnical performance of the residents was requested. The self-assessment of residents was compared with observer evaluations using the Kruskal-Wallis test. Gwet AC2 fit coefficient was used to determine the consistency between the observers' and residents' assessments. Bland-Altman plots were generated with 95% limits of agreement to describe the agreement between the total scores of the observers.

RESULTS

The self-assessment of residents had a statistically significant higher score when compared with observers (faculty and nurses) (P<0.001). However, no significant difference was observed between the total scores given by the observers (faculty members and nurses) (P>0.05). There was a moderate agreement between the resident versus faculty members [0.503; 95% confidence interval (CI), 0.430-0.576] and resident versus nurse (0.518; 95% CI, 0.432-0.605) when evaluating technical skills. However, there was substantial agreement between faculty members and nurses (0.736; 95% CI, 0.684-0.789). Postoperative pain was significantly correlated with resident self-assessment (P=0.022).

CONCLUSION

The self-assessment scores of surgical residents in LC operations were overestimated compared with observer assessments.

Impacto de la participación del residente y su rol en desenlaces clínicos de los pacientes llevados a colecistectomía en un hospital universitario

Introducción. La colecistectomía es uno de los procedimientos quirúrgicos más realizados a nivel mundial, por lo que su aprendizaje es cada vez más necesario para los médicos residentes en entrenamiento, pero sin comprometer la seguridad de los pacientes. El objetivo de este estudio fue determinar el impacto de la participación de los médicos residentes en los principales desenlaces clínicos de la colecistectomía. Métodos. Se realizó un estudio prospectivo de cohortes, donde se incluyeron los pacientes llevados a colecistectomía laparoscópica, desde junio de 2019 hasta julio de 2020. Se llevó a cabo el análisis estadístico para describir medidas de frecuencia, tendencia central, dispersión y análisis bivariados para los desenlaces de interés. Resultados. Se incluyeron 482 pacientes a quienes se les practicó colecistectomía, 475 de ellas por vía laparoscópica. El 62,5 % fueron mujeres y el 76,2 % se realizaron de carácter urgente. En el 96 % de los procedimientos se contó con la participación de un residente. En el análisis bivariado no se encontró una diferencia estadísticamente significativa entre la participación del residente y un impacto negativo en los desenlaces clínicos de las variables relevantes. Discusión. No hay evidencia de que la participación de médicos residentes en la colecistectomía laparoscópica se asocie con desenlaces adversos en los pacientes, lo que sugiere estar en relación con una introducción temprana y responsable a este procedimiento por parte de los docentes, permitiendo que la colecistectomía sea un procedimiento seguro.

Impact of fellow compared to resident assistance on outcomes of minimally invasive surgery

INTRODUCTION

As fellowship training after general surgery residency has become increasingly common, the impact on resident education must be considered. Patient safety and procedure outcomes are often used as justification by attendings who favor fellows over residents in certain minimally invasive surgery (MIS) operations. The aim of the present study was to compare the impact of trainee level on the outcomes of selected MIS operations to determine if giving preference to fellows on grounds of outcomes is warranted.

METHODS

Patients who underwent elective laparoscopic hiatal hernia repair (LHHR), laparoscopic sleeve gastrectomy (LSG), laparoscopic Roux-en-Y gastric bypass (LRYGB), laparoscopic splenectomy (LS), laparoscopic cholecystectomy (LC), or laparoscopic ventral hernia repair (LVHR) with assistance of a general surgery chief resident or fellow were identified from the American College of Surgeon's National Surgical Quality Improvement Program database (2007-2012). Patients were matched 1:1 based on propensity score for the odds of undergoing operations assisted by a fellow.

RESULTS

5145 patients underwent LHHR, 1396 LSG, 9656 LRYGB, 863 LS, 13,434 LC, and 3069 LVHR. Fellows assisted in 41.7% of LHHR, 49.2% of LSG, 56.4% of LRYGB, 25.7% of LS, 17.1% of LC, and 27.0% of LVHR cases. After matching, overall and severe complication rates were comparable between cases performed with assistance of a fellow or chief resident. Median operative time was longer for LSG, LRYGB, and LC when a fellow assisted.

CONCLUSIONS

Surgical outcomes were similar between fellow and chief resident assistance in MIS operations, arguing that increased resident participation in basic and complex laparoscopic operations is appropriate without compromising patient safety.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

The reliability of resident self-evaluation of operative performance

BACKGROUND

Self-assessment is critical to professional self-regulation yet many trainees may not reliably self-evaluate. We examine the gap between resident and faculty perceptions of trainee operative performance and contributing factors.

METHODS

Surgery resident and faculty evaluations of trainee performance were collected from 14 academic institutions using smartphone-based performance assessments. Differences in resident/faculty ratings evaluating the same procedure were analyzed using descriptive statistics and Bayesian mixed models.

RESULTS

Of 7382 evaluations, 46% trainees and faculty performance ratings were discrepant (r = 0.47), with 80% residents rating themselves lower than faculty in those cases. This gap existed regardless of case complexity and widened as trainees gained experience. Trainees who overrated themselves had the lowest mean performance scores from faculty.

CONCLUSION

Half of residents perceived their performance differently from faculty, and this difference widened for senior residents. Future focus should be to provide opportunity for trainees to improve skills to reliably assess themselves before graduation.

Entrustable Professional Activities in General Surgery: Trends in Resident Self-Assessment

OBJECTIVE

Effective self-assessment is a cornerstone of lifelong professional development; however, evidence suggests physicians have a limited ability to self-assess. Novel strategies to improve the accuracy of learner self-assessment are needed. Our institution's surgical entrustable professional activity (EPA) implementation strategy incorporates resident self-assessment to address this issue. This study evaluates the accuracy of resident self-assessment versus faculty assessment across 5 EPAs in general surgery.

DESIGN, SETTING, PARTICIPANTS

Within a single academic general surgery residency program, assessment data for 5 surgery EPAs was prospectively collected using a mobile application. Matched assessments (resident and faculty assessments for the same clinical encounter) were identified and the remainder excluded. Assessment scores were compared using Welch's t test. Agreement was analyzed using Cohen's kappa with squared weights.

RESULTS

One thousand eight hundred and fifty-seven EPA assessments were collected in 17 months following implementation. One thousand one hundred and fifty-five (62.2%) were matched pairs. Residents under-rated their own performance relative to faculty assessments (2.36 vs 2.65, p < 0.01). This pattern held true for all subsets except for Postgraduate Year (PGY)2 residents and Inguinal Hernia EPAs. There was at least moderate agreement between matched resident and faculty EPA assessment scores (κ = 0.57). This was consistent for all EPAs except Trauma evaluations, which were completed by faculty from 2 different departments. Surgery resident self-assessments more strongly agreed with Surgery faculty assessments than Emergency Medicine faculty assessments (κ = 0.58 vs 0.36).

CONCLUSIONS

Resident EPA self-assessments are equivalent or slightly lower than faculty assessments across a wide breadth of clinical scenarios. Resident and faculty matched assessments demonstrate moderate agreement.

Mind the Gap: The Autonomy Perception Gap in the Operating Room by Surgical Residents and Faculty

OBJECTIVE

Examine the concordance of perceived operative autonomy between attendings and resident trainees.

DESIGN

Faculty and trainees rated trainee operative autonomy using the 4-level Zwisch scale over a variety of cases and training years. The respective ratings were then compared to explore the effects of experience, gender, case complexity, trainee, trainer, and other covariates to perceived autonomy.

SETTING

This study was conducted over 14 general surgery programs in the United States, members of the Procedural Learning and Safety Collaborative.

PARTICIPANTS

Participants included faculty and categorical trainees from 14 general surgery programs.

RESULTS

A total of 8681 observations was obtained. The sample included 619 unique residents and 457 different attendings. A total of 598 distinct procedures was performed. In 60% of the cases, the autonomy ratings between trainees and attendings were concordant, with only 3.5% of cases discrepant by more than 1 level. An autonomy perception gap was modeled based on the discrepancy between the trainee and attending Zwisch ratings for the same case. The mean Zwisch score expected for a trainee was lower than the attending across all scenarios. Trainees were more likely to perceive relatively more autonomy in the second half of the year. The autonomy perception gap decreased with increasing case complexity. As trainees gained experience, the perception gap increased with trainees underestimating autonomy.

CONCLUSIONS

Trainees and attendings generally demonstrated concordance on autonomy perception scores. However, in 40% of cases, a perception gap exists between trainee and attending with the trainee generally underestimating autonomy. The gap worsens as the trainee progresses through residency. This perception gap suggests that attendings and trainees could be better aligned on teaching goals and expectations.

Do Graduating Oral-Maxillofacial Surgery Residents Feel Confident in Practicing the Full Scope of the Specialty?

PURPOSE

Laskin (2008) suggested that the core of oral and maxillofacial surgery could be divided into 3 levels: 1) areas of expertise, such as oral pathology, oral medicine, dentoalveolar surgery, preprosthetic surgery, and maxillofacial traumatology; 2) areas of competence, including orthognathic surgery, temporomandibular joint surgery, and local reconstructive surgery; and 3) areas of familiarity that include cleft lip and palate surgery, craniofacial surgery, regional reconstructive surgery, oncologic surgery, and cosmetic surgery. The purpose of this study is to assess residents' confidence in practicing the full scope of oral-maxillofacial surgery and their perception of gaps existing in their training. The investigators hypothesized that graduating residents' confidence in practicing the full scope of oral-maxillofacial surgery falls within the suggested levels by Laskin.

METHODS

The investigators implemented a cross-sectional study. A questionnaire was designed, and a link to a Partners REDCap (Nashville, TN) survey was electronically mailed to 252 chief residents in the final month (July 2018) of their program. The questionnaire included resident demographic characteristics, program demographic characteristics, and residents' perceived preparedness, level of proficiency, comfort, and experience for each area.

RESULTS

The sample was composed of 78 graduating residents, with a response rate of 31%. Analyses included univariate statistics and a Wilcoxon signed rank test. Significance was set at P < .05. There was no significant difference between orthognathic surgery and each area in the expertise category, and in some cases, a significantly higher perception of ability in orthognathic surgery (P < .05). There was a significantly lower perception of training in oral medicine than each area in the expertise category (P < .05), suggesting a deficit in oral medicine training.

CONCLUSIONS

Residents perceived that their training in orthognathic surgery was at the level of expertise, higher than expected. They also perceived a training deficit in oral medicine. Training proficiency in dentoalveolar surgery was found to be at a level higher than any other area.

Simulation Training Incorporating a Pulsatile Carotid Endarterectomy Model Results in Increased Procedure-Specific Knowledge, Confidence, and Comfort in Post-graduate Trainees

OBJECTIVES

Surgical simulation has been used to facilitate the acquisition of vascular surgery skills. However, high cost and limited availability may restrict the use of this educational resource. We report how instruction using a low-cost, pulsatile, carotid endarterectomy (CEA) benchtop surgical simulation model can be used to enhance learners' procedure-specific knowledge, comfort, and confidence in performing the steps of a CEA procedure DESIGN: A single instructor engaged each participant in a one-on-one instructional session during which the instructor demonstrated, and then the participants performed, the steps of a CEA. Participants completed a pre- and postintervention assessment of knowledge and attitudes about preforming a CEA and use of simulation as a learning tool. Postintervention, participants rated the impact of the simulation model on their learning. A Related T-test and Wilcoxin signed Rank Test were used to compare pre- and postintervention results.

SETTINGS

University of Virginia Health System, Charlottesville, Virginia.

PARTICIPANTS

Seventeen postgraduate trainees.

RESULTS

A significant difference was observed in pre- and postknowledge scores (48% vs 91% correct, p < 0.01). Trainee confidence (1.65 vs 2.88, p < 0.01) and comfort (1.59 vs 2.82, p < 0.01) with doing the procedure also increased significantly. Sixteen (94%) responded that use of the simulator was extremely or very important as a tool for learning. All 17 trainees (100%) reported that the simulation experience was either essential or very useful in helping them learn how to perform a CEA. Sixty-five percent responded that they were extremely likely to apply the skills learned during the intervention the next time they performed a CEA.

CONCLUSIONS

A low-cost, pulsatile CEA simulation model used as an educational tool increased procedure-specific knowledge, comfort, and confidence among trainees. Learner's increased confidence and affirmation that they are likely to apply the learned skills in a clinical setting support the use of this educational approach to impact trainee behaviors.

Balancing Operative Efficiency and Surgical Education: A Functional Neurosurgery Model

BACKGROUND

Attending surgeons have dual obligations to deliver high-quality health care and train residents. In modern healthcare, lean principles are increasingly applied to processes preceding and following surgery. However, surgeons have limited data regarding variability and waste during any given operation.

OBJECTIVE

To measure variability and waste during the following key functional neurosurgery procedures: retrosigmoid craniectomy (microvascular decompression [MVD] and internal neurolysis) and deep brain stimulation (DBS). Additionally, we correlate variability with residents' self-reported readiness for the surgical steps. The aim is to guide surgeons as they balance operative safety and efficiency with training obligations.

METHODS

For each operation (retrosigmoid craniectomy and DBS), a standard workflow, segmenting the operation into components, was defined. We observed a representative sample of operations, timing the components, with a focus on variability. To assess perceptions of safety and risk among surgeons of various training levels, a survey was administered. Survey results were correlated with operative variability, attempting to identify areas for increasing value without compromising trainee experience.

RESULTS

A sampling of each operation (n = 36) was observed during the study period. For MVD, craniectomy had the highest mean duration and standard deviation, whereas the MVD itself had the lowest mean duration and standard deviation. For DBS, the segments with largest standard deviation in duration were registration and electrode placement. For many steps of both procedures, there was a statistically significant relationship between increasing level of training and increasing perception of safety.

CONCLUSION

This proof-of-concept study introduces an educational and process-improvement tool that can be used to aid surgeons in increasing the efficiency of patient care.

Resident involvement in minimally-invasive vs. open procedures

BACKGROUND

The objective of this study was to evaluate the impact of resident involvement on surgical outcomes in laparoscopic compared to open procedures.

METHODS

The American College of Surgeons National Surgical Quality Improvement Program 2007-2012 was queried for open and laparoscopic ventral hernia repair (VHR), inguinal hernia repair (IHR), splenectomy, colectomy, or cholecystectomy (CCY). Multivariable regression analyses were performed to assess the impact of resident involvement on surgical outcomes.

RESULTS

In total, 88,337 VHR, 20,586 IHR, 59,254 colectomies, 3301 splenectomies, and 95,900 CCY were identified. Resident involvement was predictive for major complication during open VHR (AOR, 1.29; p < 0.001), but not during any other procedure. Resident participation significantly prolonged operative time for open, as well as laparoscopic VHR, IHR, colectomy, splenectomy, and CCY (all p < 0.01).

CONCLUSIONS

The results of this study suggest that resident participation has a similar impact on surgical outcomes during laparoscopic and open surgery, and is generally safe.

Targeted surgical coaching can improve operative self-assessment ability: A single-blinded nonrandomized trial

BACKGROUND

The aim of the study was to investigate the effect of targeted surgical coaching on self-assessment of laparoscopic operative skill. Accurate self-assessment is vital for autonomous professional development. Surgical coaching can be used for performance improvement, but its role in this domain has been insufficiently investigated.

METHODS

This was a single site, nonrandomized, interrupted time series design trial. Participants were residents, fellows, and attending surgeons regularly performing laparoscopic general surgery operations. Each participant was enrolled in an individualized coaching program using review of personal and peer laparoscopic videos. The program involved 3 to 5 sessions over a period of 6 to 19 weeks. Coaching used case debriefing to target self-assessment proficiency, with a focus on objective interpretation of observations and facilitative capacity building. The primary outcome measure was self-assessment accuracy and correlation to expert ratings. The Objective Structured Assessment of Technical Skill global rating scale was utilized for evaluation.

RESULTS

Twelve participants were recruited and completed the coaching program. At baseline, there was no correlation between self-assessment and expert ratings. After completion of the coaching program there was correlation between self-assessment and expert ratings (P = .003) and improved self-assessment accuracy compared to baseline (P = .041).

CONCLUSION

This study has demonstrated that targeted coaching using video review of laparoscopic cases can improve operative self-assessment accuracy using the Objective Structured Assessment of Technical Skill.

Use of Entrustable Professional Activities in the Assessment of Surgical Resident Competency

Importance

Competency-based assessments of surgical resident performance require metrics of entrustable autonomy.

Objectives

To designate entrustable professional activities (EPAs) in global performance and in specific operations, and to identify differences in perceived capability, autonomy, and expectations between surgical faculty and residents.

Design, Setting, and Participants

This survey study was conducted from August 9, 2016, through August 24, 2016, in the Department of Surgery at the UCLA David Geffen School of Medicine. The survey instrument consisted of 5-point Likert scales for assessing perceptions of entrustability for 5 global and 5 operative EPAs. Faculty members were surveyed regarding resident capabilities and expected capabilities by postgraduate year. Residents were surveyed regarding their own capabilities, actual autonomy entrusted in the last EPA performed, and expected capabilities.

Main Outcomes and Measures

Differences in mean ratings were assessed across 7 comparison domains.

Results

Among 78 total faculty members, 31 (40%) participated in the survey. Among 49 residents, 39 (80%) participated in the survey. Residents generally rated their global EPA performance higher than the faculty did (mean, 3.7 vs 2.8; P < .01), but operative EPA performance ratings were equivalent (mean, 2.7 vs 2.4; P < .12). Faculty members perceived senior residents as underperforming expectations in operative EPAs. Most faculty members (80%) expected residents not to be independently capable of performing complex operations by graduation. Faculty members perceived residents in postgraduate years 4 and 5 to have greater operative capability than the level of autonomy entrusted to those residents (95% CI, 3.3-3.5 vs 1.9-2.2).

Conclusions and Relevance

Global and operative EPAs are practical for developing competency-based curricula. Graduated autonomy should be granted to improve the operative experience for residents.

Self-Reported Training Adequacy, Experience, and Comfort Level in Performing Schizophrenia-Related Clinical Skills among Psychiatry Residents and Fellows

OBJECTIVE

In the context of an educational program on schizophrenia for psychiatry trainees, this survey study analyzed associations between self-reported training adequacy, experience in providing patient care, and comfort level in performing schizophrenia-related clinical skills. The influence of the education on comfort level was also assessed for each skill.

METHODS

Survey respondents were psychiatry residents and fellows who participated in a schizophrenia education program at an in-person workshop or through online videos recorded at the workshop. In a pre-program survey, participants reported their experience in providing schizophrenia patient care and rated their training adequacy and comfort level for performing seven clinical skills involved in diagnosing and treating schizophrenia. The post-program survey included items for reassessing comfort level in performing the skills.

RESULTS

Across the seven clinical skills, the proportion of respondents (n = 79) who agreed or strongly agreed that their training was adequate ranged from 29 to 88 %. The proportion of high ratings for comfort level in skill performance ranged from 45 to 83 %. Comfort level was significantly associated with training adequacy for all seven clinical skills and with experience in providing patient care for four skills. For all skills, comfort level ratings were significantly higher after versus before the educational workshop. Commonly indicated needs for further training included education on new therapies, exposure to a broader range of patients, and opportunities for longitudinal patient management.

CONCLUSIONS

Psychiatry trainees' self-reported, disease-specific training adequacy, experiences, and comfort level have unique applications for developing and evaluating graduate medical curriculum.