Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment

Laparoscopic vs robotic inguinal hernia repair: a comparison of learning curves and skill transference in general surgery residents

BACKGROUND

There is no consensus on whether laparoscopic experience should be a prerequisite for robotic training. Further, there is limited information on skill transference between laparoscopic and robotic techniques. This study focused on the general surgery residents' learning curve and skill transference within the two minimally invasive platforms.

METHODS

General surgery residents were observed during the performance of laparoscopic and robotic inguinal hernia repairs. The recorded data included objective measures (operative time, resident participation indicated by percent active time on console or laparoscopy relative to total case time, number of handoffs between the resident and attending), and subjective evaluations (preceptor and trainee assessments of operative performance) while controlling for case complexity, patient comorbidities, and residents' prior operative experience. Wilcoxon two-sample tests and Pearson Correlation coefficients were used for analysis.

RESULTS

Twenty laparoscopic and forty-four robotic cases were observed. Mean operative times were 90 min for robotic and 95 min for laparoscopic cases (P = 0.4590). Residents' active participation time was 66% on the robotic platform and 37% for laparoscopic (P =  < 0.0001). On average, hand-offs occurred 9.7 times during robotic cases and 6.3 times during laparoscopic cases (P = 0.0131). The mean number of cases per resident was 5.86 robotic and 1.67 laparoscopic (P = 0.0312). For robotic cases, there was a strong correlation between percent active resident participation and their prior robotic experience (r = 0.78) while there was a weaker correlation with prior laparoscopic experience (r = 0.47). On the other hand, prior robotic experience had minimal correlation with the percent active resident participation in laparoscopic cases (r = 0.12) and a weak correlation with prior laparoscopic experience (r = 0.37).

CONCLUSION

The robotic platform may be a more effective teaching tool with a higher degree of entrustability indicated by the higher mean resident participation. We observed a greater degree of skill transference from laparoscopy to the robot, indicated by a higher degree of correlation between the resident's prior laparoscopic experience and the percent console time in robotic cases. There was minimal correlation between residents' prior robotic experience and their participation in laparoscopic cases. Our findings suggest that the learning curve for the robot may be shorter as prior robotic experience had a much stronger association with future robotic performance compared to the association observed in laparoscopy.

Veress needle insertion simulation model: A simple new module for advanced surgical skill training

Introduction

Training on Veress needle (VN) insertion cannot be done by observation without practicing tactile feedback. In this study, a simple and reproducible VN insertion training model was created. The aim of this study was to evaluate the validity of using the proposed model in simulating actual real-life surgical experiences.

Methods

The proposed VN insertion training model is made of three layers of synthetic rubber and plastic materials, simulating the tensile strength and texture of the three abdominal wall muscle layers. Surgeons and senior residents with experience in minimally invasive procedures were asked to practice VN insertion on this model, each completing the procedure three times. Participants were then asked to record their comments and answer six questions regarding their experience practicing on the model.

Results

Ten surgeons and four senior residents participated in this study. All participants agreed or strongly agreed that the model simulates the surgery experience regarding the shape and overall structure, tactile feedback and confirmation of complete/successful insertion. Twelve participants (86%) agreed or strongly agreed that the pressure/force needed for VN insertion was like real surgery experience and that the overall experience with using this model is similar to the real surgical experience. Almost all participants (93%) agreed or strongly agreed that the model is a valuable resource for training before practicing the procedure on real patients.

Conclusions

The VN insertion training model provides a valuable training opportunity on a demanding surgical skill. It is simple, reproducible and closely simulates surgery.

A quantitative cross-sectional study assessing the surgical trainee perception of the operating room educational environment

BACKGROUND

Limited hours and service provision are diminishing training opportunities, whilst increasing standards of surgical proficiency is being sought. It is imperative to maximise the value of each educational event. An objective measure of higher surgical trainee perception of the operating room environment in England has not been performed before and this can steer future change in optimising educational events in theatre. The Operating Room Educational Environment Measure (OREEM) evaluates each component of the learning environment to enable optimisation of these educational events. However, the OREEM has not yet been assessed for reliability in higher surgical trainees in England. The aim of the current study was to explore areas of strength and weakness in the educational environment in the operating room as perceived by surgical trainees' in one English region. The secondary aim was to assess the reliability of the OREEM.

METHODS

Using a quantitative approach, data was collected over one month from surgical trainees in England using the OREEM.

RESULTS

Fifty-four surgical trainees completed the questionnaire. The OREEM had good internal consistency (α = 0.906, variables = 40). The mean OREEM score was 79.16%. Areas for improvement included better learning opportunities (average subscale score = 72.9%) and conducting pre- and post-operative teaching (average score = 70.4%). Trainees were most satisfied with the level of supervision and workload (average subscale score = 82.87%). The learning environment favoured senior trainees (p = 0.017). There was a strong correlation between OREEM and the global satisfaction score (p < 0.001).

CONCLUSIONS

The OREEM was shown to be a reliable measure of the educational environment. It can be used to identify areas of improvement and as an audit tool. The current perception of the education environment is satisfactory, however, areas of improvement include reducing service provision, empowering trainees to plan lists, improving teamwork and using tools to optimise the educational value of each operation. There is a favourable attitude regarding the use of improvement tools, especially for dissatisfied trainees.

Implementation of the Operating Room Black Box Research Program at the Ottawa Hospital Through Patient, Clinical, and Organizational Engagement: Case Study

Background

A large proportion of surgical patient harm is preventable; yet, our ability to systematically learn from these incidents and improve clinical practice remains limited. The Operating Room Black Box was developed to address the need for comprehensive assessments of clinical performance in the operating room. It captures synchronized audio, video, patient, and environmental clinical data in real time, which are subsequently analyzed by a combination of expert raters and software-based algorithms. Despite its significant potential to facilitate research and practice improvement, there are many potential implementation challenges at the institutional, clinician, and patient level. This paper summarizes our approach to implementation of the Operating Room Black Box at a large academic Canadian center.

Objective

We aimed to contribute to the development of evidence-based best practices for implementing innovative technology in the operating room for direct observation of the clinical performance by using the case of the Operating Room Black Box. Specifically, we outline the systematic approach to the Operating Room Black Box implementation undertaken at our center.

Methods

Our implementation approach included seeking support from hospital leadership; building frontline support and a team of champions among patients, nurses, anesthesiologists, and surgeons; accounting for stakeholder perceptions using theory-informed qualitative interviews; engaging patients; and documenting the implementation process, including barriers and facilitators, using the consolidated framework for implementation research.

Results

During the 12-month implementation period, we conducted 23 stakeholder engagement activities with over 200 participants. We recruited 10 clinician champions representing nursing, anesthesia, and surgery. We formally interviewed 15 patients and 17 perioperative clinicians and identified key themes to include in an information campaign run as part of the implementation process. Two patient partners were engaged and advised on communications as well as grant and protocol development. Many anticipated and unanticipated challenges were encountered at all levels. Implementation was ultimately successful, with the Operating Room Black Box installed in August 2018, and data collection beginning shortly thereafter.

Conclusions

This paper represents the first step toward evidence-guided implementation of technologies for direct observation of performance for research and quality improvement in surgery. With technology increasingly being used in health care settings, the health care community should aim to optimize implementation processes in the best interest of health care professionals and patients.

Constructing Learning Curves to Benchmark Operative Performance of General Surgery Residents Against a National Cohort of Peers

OBJECTIVE

No method or data exist to allow surgical trainees or their programs to contextualize their technical progress. The objective of this study was to create peer benchmarks for Cumulative Sum (CUSUM) charts based upon operative evaluations from a national cohort of general surgery residents.

DESIGN, SETTING, PARTICIPANTS

In 2016-2018, faculty from 26 general surgery residency programs nationwide rated 328 residents' operative performance on a case-by-case basis using a validated 5-point Likert scale. An individual case was considered a "misstep" if scoring below the national median score for that procedure in that postgraduate year (PGY). We constructed 2-sided observed-expected CUSUM charts to capture each resident's cumulative performance over time relative to the national medians. Upper (failure) and lower (positive outlier) benchmarks were established based on the PGY-specific 75th percentile and median misstep rates; consistent/repeated missteps are reflected by crossing of the upper boundary. Procedures with ≤10 observations and residents who were evaluated <10 times for each PGY were excluded.

RESULTS

Around 8,161 evaluations on 76 procedure types were analyzed. The individual misstep rate was lowest among PGY-3s at 13.3% and highest among PGY-4s at 28.6%. No interns had curves that crossed the failure boundary. 8.7% of PGY-2s and 8.9% of PGY-3s finished the year past the failure boundary. PGY-2s had the most positive outliers, with 28.3% of them demonstrating an outlying success performance beyond the lower boundary for at least once. PGY-5s most frequently failed, with 16.7% ever crossing the upper boundary and 11.1% remaining above it at graduation.

CONCLUSIONS

CUSUM is a valid statistical approach for benchmarking individual residents' operative performance against national peers as they progress through the year in real-time. With further validation, CUSUM could be used to set progression and/or graduation standards and objectively identify residents who might benefit from remediation.

A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement

INTRODUCTION

Mobile apps that utilize the framework of entrustable professional activities (EPAs) to capture and deliver feedback are being implemented. If EPA apps are to be successfully incorporated into programmatic assessment, a better understanding of how they are experienced by the end-users will be necessary. The authors conducted a qualitative study using the Consolidated Framework for Implementation Research (CFIR) to identify enablers and barriers to engagement with an EPA app.

METHODS

Structured interviews of faculty and residents were conducted with an interview guide based on the CFIR. Transcripts were independently coded by two study authors using directed content analysis. Differences were resolved via consensus. The study team then organized codes into themes relevant to the domains of the CFIR.

RESULTS

Eight faculty and 10 residents chose to participate in the study. Both faculty and residents found the app easy to use and effective in facilitating feedback immediately after the observed patient encounter. Faculty appreciated how the EPA app forced brief, distilled feedback. Both faculty and residents expressed positive attitudes and perceived the app as aligned with the department's philosophy. Barriers to engagement included faculty not understanding the EPA framework and scale, competing clinical demands, residents preferring more detailed feedback and both faculty and residents noting that the app's feedback should be complemented by a tool that generates more systematic, nuanced, and comprehensive feedback. Residents rarely if ever returned to the feedback after initial receipt.

DISCUSSION

This study identified key enablers and barriers to engagement with the EPA app. The findings provide guidance for future research and implementation efforts focused on the use of mobile platforms to capture direct observation feedback.

Education Management Platform Enables Delivery and Comparison of Multiple Evaluation Types

OBJECTIVE

The purpose of this study was to determine whether an automated platform for evaluation selection and delivery would increase participation from surgical teaching faculty in submitting resident operative performance evaluations.

DESIGN

We built a HIPAA-compliant, web-based platform to track resident operative assignments and to link embedded evaluation instruments to procedure type. The platform matched appropriate evaluations to surgeons' scheduled procedures, and delivered multiple evaluation types, including Ottawa Surgical Competency Operating Room Evaluation (O-Score) evaluations and Operative Performance Rating System (OPRS) evaluations. Prompts to complete evaluations were made through a system of automatic electronic notifications. We compared the time spent in the platform to achieve evaluation completion. As a metric for the platform's effect on faculty participation, we considered a task that would typically be infeasible without workflow optimization: the evaluator could choose to complete multiple, complementary evaluations for the same resident in the same case. For those cases with multiple evaluations, correlation was analyzed by Spearman rank test. Evaluation data were compared between PGY levels using repeated measures ANOVA.

SETTING

The study took place at 4 general surgery residency programs: The University of Massachusetts Medical School-Baystate, the University of Connecticut School or Medicine, the University of Iowa Carver College of Medicine, and Maimonides Medical Center.

PARTICIPANTS

From March 2017 to February 2019, the study included 70 surgical teaching faculty and 101 general surgery residents.

RESULTS

Faculty completed 1230 O-Score evaluations and 106 OPRS evaluations. Evaluations were completed quickly, with a median time of 36 ± 18 seconds for O-Score evaluations, and 53 ± 51 seconds for OPRS evaluations. 89% of O-Score and 55% of OPRS evaluations were completed without optional comments within one minute, and 99% of O-Score and 82% of OPRS evaluations were completed within 2 minutes. For cases eligible for both evaluation types, attendings completed both evaluations on 74 of 221 (33%) of these cases. These paired evaluations strongly correlated on resident performance (Spearman coefficient = 0.84, p < 0.00001). Both evaluation types stratified operative skill level by program year (p < 0.00001).

CONCLUSIONS

Evaluation initiatives can be hampered by the challenge of making multiple surgical evaluation instruments available when needed for appropriate clinical situations, including specific case types. As a test of the optimized evaluation workflow, and to lay the groundwork for future data-driven design of evaluations, we tested the impact of simultaneously delivering 2 evaluation instruments via a secure web-based education platform. We measured the evaluation completion rates of faculty surgeon evaluators when rating resident operative performance, and how effectively the results of evaluation could be analyzed and compared, taking advantage of a highly integrated management of the evaluative information.

Does Perceived Resident Operative Autonomy Impact Patient Outcomes?

OBJECTIVE

We investigated the association of perceived trainee autonomy with patient clinical outcomes following colorectal surgery.

DESIGN

This was a prospective multi-institutional study that consisted of surgery trainees completing a survey tool immediately after participating in colorectal resections to rate their self-perceived autonomy and case characteristics. Self-perception of autonomy was classified as observer, assistant, surgeon, or teacher. The completed trainee surveys were linked with patient information available through each hospital's internal NSQIP directory. The primary outcome was death and serious morbidity (DSM) and secondary outcome was 30-day readmissions. Separate mixed effects regression models were used to examine the association between perceived trainee autonomy and DSM or 30-day readmissions. Fixed effects were used to control for the effects of the training environment. The models were constructed to adjust for patient and trainee characteristics associated with each outcome independently.

SETTING

This study was conducted at 7 general surgery training programs (5 academic medical centers and 2 independent training programs) with general surgery or colorectal surgery services.

PARTICIPANTS

This study included a total of 63 residents and fellows rotating on surgery services that performed colorectal resections at the included 7 general surgery training programs from January until March 2016.

RESULTS

The 63 trainees that participated in this study completed 417 surveys with over a 95% response rate. National Surgical Quality Improvement Program (NSQIP) patient records were available for 67% (n = 273) of completed surveys. The clinical year of the trainees were 6.1% PGY 1/2, 36% Post graduate year (PGY) 3, 40.9% PGY 4/5, and 17% fellows. Residents perceived their participation in the case to be that of an observer in 9.2% of surveys, an assistant in 51.6% of surveys, and the surgeon/teacher in 39.3% of surveys. About 50% of patients were male, 80% were White, the majority had an American Society of Anesthesiologists classification of 3, almost half had prior abdominal surgery, and over 80% of surgeries were elective. The primary operation types performed were laparoscopic (40.3%) and open (35.9%) partial colectomies. The rate of DSM in patients was approximately 24% when trainees perceived their role as observers, 23% when trainees perceived their role as assistants, and 18% when trainees perceived their role as surgeons/teachers. After adjustment for patient, trainee, and training environment, we found that the perceived level of trainee autonomy of a surgeon/teacher was associated with a 4-fold lower rate of DSM (odds ratio: 0.23, confidence of interval: 0.05-0.97, p = 0.045) compared to observers. The rate of readmissions was approximately 20% when trainees perceived their role as observers, 14% when trainees perceived their role as assistants and 9% when trainees perceived their role as surgeons/teachers. After adjustment for patient, trainee, and training environment, we found that the perceived level of trainee autonomy of a surgeon/teacher was significantly associated with a 10-fold lower rate of 30-day readmissions (odds ratio: 0.09, confidence of interval: 0.01-0.70, p = 0.022) compared to observers.

CONCLUSIONS

There was an association between increased perceived trainee autonomy and improved patient outcomes, suggesting that when trainees identify with an increased role in the operation, patients may have improved care. Further research is needed to understand this association further.

Resident Surgical Skills Web-Based Evaluation: A Comparison of 2 Assessment Tools

BACKGROUND

Evaluation of surgical skill competency is necessary as graduate medical education moves toward a competency-based curriculum. This study by the American Board of Orthopaedic Surgery (ABOS) and the Council of Orthopaedic Residency Directors (CORD) compares 2 web-based evaluation tools that assess the level of autonomy that is demonstrated by residents during surgical procedures in the operating room as measured by faculty.

METHODS

Two hundred and ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using 2 web-based evaluation tools in a crossover design in which residents requested faculty review of their surgical skills before starting a case. One thousand, one hundred and fifty Ottawa Surgical Competency Operating Room Evaluation (O-Score) assessments, which included a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, which included a single-question summative evaluation. Twenty-five different surgical procedures were evaluated.

RESULTS

There were no significant differences in rates of resident requests or faculty completion of the 2 scores. The most common surgical procedures that were assessed were total knee arthroplasty (n = 254, 11%), carpal tunnel release (n = 191, 8%), open reduction and internal fixation (ORIF) of stable hip fractures (n = 170, 7%), ORIF of simple ankle fractures (n = 169, 7%), and total hip arthroplasty (n = 166, 7%). Both instruments disclosed significant differences in competency among entry, intermediate, and advanced-level residents. The findings support the construct validity of the evaluation method. The survey results indicated that >70% of the faculty were confident that use of either the P-score or the O-score allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training.

CONCLUSIONS

This research has led to the modification of the O-score and the P-score into a combined OP-score instrument. The ABOS envisions that the OP-score instrument can be used with an expanded number of surgical procedures as a required element of residency training in the near future.

CLINICAL RELEVANCE

This study allows the profession of orthopaedic surgery education to take a leadership role in the measurement of competence for surgical skills for orthopaedic surgeons in residency training, an important clinically relevant topic to the practice of orthopaedic surgery.

Progressive Autonomy in the Era of Increased Supervision: AOA Critical Issues

The observation of decreased resident autonomy, ultimately influencing the readiness of a new graduate to practice, has been supported with a number of recent surveys. This perceived lack of autonomy is felt to be due, in part, to many reasons, including duty-hour regulations, increased supervision requirements, patient safety measures, concern for complication rates, and other performance measures. Pressure on faculty members to have increased clinical productivity may not allow for more resident autonomy.Increased clinical exposure to improve resident independence may come from several suggested areas. First, restructuring the residency program to allow for more clinical time may be one way to improve education. Second, increased use of surgical simulation will allow for more experience to develop technical skills within a controlled environment. Surgical simulators can be used to acquire new skills and also as a means of assessing competence. Third, competency-based education (CBE) has been offered as a way to improve resident education. At its core, CBE offers criterion-based assessments for residents and faculty that allow for more frequent feedback.