Reflections in a time of transition: orthopaedic faculty and resident understanding of accreditation schemes and opinions on surgical skills feedback

Early findings and strategies for successful implementation of SIMPL workplace-based assessments within vascular surgery residency and fellowship programs

OBJECTIVE

As medical education systems increasingly move toward competency-based training, it is important to understand the tools available to assess competency and how these tools are utilized. The Society for Improving Medical Professional Learning (SIMPL) offers a smart phone-based assessment system that supports workplace-based assessment of residents' and fellows' operative autonomy, performance, and case complexity. The purpose of this study was to characterize implementation of the SIMPL app within vascular surgery integrated residency (0+5) and fellowship (5+2) training programs.

METHODS

SIMPL operative ratings recorded between 2018 and 2022 were collected from all participating vascular surgery training institutions (n = 9 institutions with 5+2 and 0+5 programs; n = 4 institutions with 5+2 program only). The characteristics of programs, trainees, faculty, and SIMPL operative assessments were evaluated using descriptive statistics.

RESULTS

Operative assessments were completed for 2457 cases by 85 attendings and 86 trainees, totaling 4615 unique operative assessment ratings. Attendings included dictated feedback in 52% of assessments. Senior-level residents received more assessments than junior-level residents (postgraduate year [PGY]1-3, n = 439; PGY4-5, n = 551). Performance ratings demonstrated increases from junior to senior trainees for both resident and fellow cohorts with "performance-ready" or "exceptional performance" ratings increasing by nearly two-fold for PGY1 to PGY5 residents (28.1% vs 40.6%), and from first- to second-year fellows (PGY6, 46.7%; PGY7, 60.3%). Similar gains in autonomy were demonstrated as trainees progressed through training. Senior residents were more frequently granted autonomy with "supervision only" than junior residents (PGY1, 8.7%; PGY5, 21.6%). "Supervision only" autonomy ratings were granted to 21.8% of graduating fellows. Assessment data included a greater proportion of complex cases for senior compared with junior fellows (PGY6, 20.9% vs PGY7, 26.5%). Program Directors felt that faculty and trainee buy-in were the main barriers to implementation of the SIMPL assessment app.

CONCLUSIONS

This is the first description of the SIMPL app as an operative assessment tool within vascular surgery that has been successfully implemented in both residency and fellowship programs. The assessment data demonstrates expected progressive gains in trainees' autonomy and performance, as well as increasing case complexity, across PGY years. Given the selection of SIMPL as the assessment platform for required American Board of Surgery and Vascular Surgery Board Entrustable Professional Activities assessments, understanding facilitators and barriers to implementation of workplace-based assessments using this app is imperative, particularly as we move toward competency-based medical education.

Radiation Oncology Resident Evaluations: Current Methods and Resident Perceptions

Purpose

This study analyzes assessments within radiation oncology (RO) to determine characteristics of existing assessment methods and then report resident perceptions of these methods. We hypothesize familiarity with evaluation methods is predictive of the perceived utility of evaluations and behavioral changes.

Methods and materials

This study was conducted in 2 phases. Phase 1 involved requesting resident evaluation forms from RO residency programs to assess the Accreditation Council for Graduate Medical Education 6 Core Competencies. Analysis of variance was used to determine any significant differences between institutions or categories of questions. In phase 2, RO residents were surveyed about familiarity with the Accreditation Council for Graduate Medical Education Milestones and their perceptions about the current methods. Responses to questions were further analyzed using linear regression models.

Results

In phase 1, forms were obtained from 13 institutions, and they were based on the 6 Core Competencies with an average of 19 questions (standard deviation, 11; range, 5-47) in total. Analysis of variance did not show significant variation in the number of questions between the categories (F = 0.78, P = .6). A significant difference in the mean number of questions used for assessing each of the competencies was found between institutions (F = 6.6, P < .01). In phase 2, a majority of surveyed residents reported being "not" or only "slightly familiar" with the competencies (59.6%) and the factors used to assess them (73.1%). Resident-reported familiarity with the evaluation methods was not found to be a significant predictor of likelihood of changing postevaluation (coefficient = 0.41, P = .204), intimidation from receiving evaluations (coefficient = -0.06, P = .792), stress of receiving evaluations (coefficient = -0.11, P = .62), or usefulness of evaluations (coefficient = -0.02, P = .83).

Conclusions

Familiarity with evaluation methods is not correlated with perceptions or behavioral changes necessitating further investigation of alternative predictor variables. Despite the low familiarity with evaluation tools, most residents reported that evaluations were useful and likely to elicit changes in their behaviors and practice, highlighting the value of current evaluation methods.

Qualitative Study of Orthopedic Surgery Milestones 1.0: Burdens and Benefits

OBJECTIVE

The purpose of this study was to explore the context and mechanisms by which the first set of Milestones impacted the processes of the Clinical Competency Committee, how programs have incorporated the Milestones into their program, and to understand more about the resident perspective in order to improve Orthopedic Surgery Milestones.

DESIGN, SETTING, AND PARTICIPANTS

In 2020 all 201 accredited orthopedic surgery residency programs were invited to participate in qualitative telephone interviews to assess their experience with the Milestones and complete a supplemental online survey about their Clinical Competency Committee. Participants were comprised of a self-selected sample and complemented by a purposeful sample to ensure a breadth of perspectives. Interview data were analyzed using template analysis. Survey data were analyzed with descriptive statistics.

RESULTS

Interviews were completed with 101 individuals from 47 programs (23% of all programs). The two overarching themes were implementation and impact of Milestones. Subthemes within implementation were substantial variability in approaches to Milestone use in curriculum and assessment, faculty development, and methods to introduce residents to the Milestones assessment framework. The large number of subcompetencies created a significant burden for almost all programs. The structure of the Milestones was also viewed as poorly aligned with the variable design of rotation schedules across programs. Milestones have the potential to offer valuable feedback for trainees and programs overall.

CONCLUSIONS/APPLICATION

While some benefits were noted and most programs appreciated the intent, the structure and design of the initial set of orthopedic surgery Milestones created substantial challenges for these programs. The results of this study helped guide a revision of the Orthopedic Surgery Milestones. These results can also be used by program leadership to encourage reflection around past, current, and future utilization of the Milestones framework. Further research will be needed to determine the impact of the revision on programs.

Do End-of-Rotation Evaluations Adequately Assess Readiness to Operate?

PURPOSE

Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based microassessments and (2) which factors most influence EOR evaluations and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method.

METHOD

In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment.

RESULTS

Ratings of operative performance from EOR evaluations vs workplace-based microassessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: Model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74).

CONCLUSIONS

EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared with other trainees in the same PGY of training. Workplace-based microassessments may be better for assessing a trainee's ability to perform specific procedures autonomously, thus perhaps providing more insight into a trainee's true readiness for operative independence.

Value and Barriers to Use of the SIMPL Tool for Resident Feedback

OBJECTIVE

The System for Improving and Measuring Procedural Learning (SIMPL) is a smart-phone application used to provide residents with an evaluation of operative autonomy and feedback. This study investigated the perceived benefits and barriers to app use.

DESIGN

A database of previously performed SIMPL evaluations was analyzed to identify high, low, and never users. Potential predisposing factors to use were explored. A survey investigating key areas of value and barriers to use for the SIMPL application was sent to resident and faculty users. Respondents were asked to self-identify how often they used the app. The perceived benefits and barriers were correlated with the level of usage. Qualitative analysis of free text responses was used to determine strategies to increase usage.

SETTING

General surgery training programs who are members of the Procedural Learning and Safety Collaborative.

PARTICIPANTS

Surgical residents and faculty.

RESULTS

At least 1 SIMPL evaluation was created for 411 residents and 524 faculty. Thirty percent of both faculty and residents were high-frequency users. Thirty percent of faculty were never users. One hundred eighty-eight residents and 207 faculty (response rate 46%) completed the survey. High-frequency resident users were more likely to perceive a benefit for both numerical evaluations (76% vs 30%) and dictated feedback (92% vs 30%). Faculty and residents commonly blamed each other for not creating and completing evaluations regularly (87% of residents, 81% of faculty). Suggested strategies to increase usage included reminders and integration with existing data systems.

CONTRIBUTIONS

Frequent users perceive value from the application, particularly from dictated feedback and see a positive impact on feedback in their programs. Faculty engagement represents a major barrier to adoption. Mechanisms which automatically remind residents to initiate an evaluation will help improve utilization but programs must work to enhance faculty willingness to respond and dictate feedback.

Orthopaedic Surgery Residency Training: Consideration for a Surgical and Procedural Skills Competency

This perspectives report discusses the need to create a surgical and procedural skills competency for orthopedic surgery residency training programs.

Progressive Surgical Autonomy Observed in a Hand Surgery Resident Clinic Model

OBJECTIVE

Resident clinics (RCs) are intended to catalyze the achievement of educational milestones through progressively autonomous patient care. However, few studies quantify their effect on competency-based surgical education, and no previous publications focus on hand surgery RCs (HRCs). We demonstrate the achievement of progressive surgical autonomy in an HRC model.

DESIGN

A retrospective review of all patients seen in a weekly half-day HRC from October 2010 to October 2015 was conducted. Investigators compiled data on patient demographics, provider encounters, operational statistics, operative details, and dictated surgical autonomy on an ascending 5 point scoring system.

SETTING

A tertiary hand surgery referral center.

RESULTS

A total of 2295 HRC patients were evaluated during the study period in 5173 clinic visits. There was an average of 22.6 patients per clinic, including 9.0 new patients with 6.5 emergency room referrals. Totally, 825 operations were performed by 39 residents. Trainee autonomy averaged 2.1/5 (standard deviation [SD] = 1.2), 3.4/5 (SD = 1.3), 2.1/5 (SD = 1.3), 3.4/5 (SD = 1.2), 3.2/5 (SD = 1.5), 3.5/5 (SD = 1.5), 4.0/5 (SD = 1.2), 4.1/5 (SD = 1.2), in postgraduate years 1 to 8, respectively. Linear mixed model analysis demonstrated training level significantly effected operative autonomy (p = 0.0001). Continuity of care was maintained in 79.3% of cases, and patients were followed an average of 3.9 clinic encounters over 12.4 weeks.

CONCLUSIONS

Our HRC appears to enable surgical trainees to practice supervised autonomous surgical care and provide a forum in which to observe progressive operative competency achievement during hand surgery training. Future studies comparing HRC models to non-RC models will be required to further define quality-of-care delivery within RCs.

Crowdsourcing in Surgical Skills Acquisition: A Developing Technology in Surgical Education

BACKGROUND

The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined.

OBJECTIVE

A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017.

METHODS

Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed.

RESULTS

Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59-0.95, Cronbach's alpha 0.32-0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective.

CONCLUSIONS

Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.

Rapid Web-Based Platform for Assessment of Orthopedic Surgery Patient Care Milestones: A 2-Year Validation

OBJECTIVE

To determine the validity, feasibility, and responsiveness of a new web-based platform for rapid milestone-based evaluations of orthopedic surgery residents.

SETTING

Single academic medical center, including a trauma center and pediatrics tertiary hospital.

PARTICIPANTS

Forty residents (PG1-5) in an orthopedic residency program and their faculty evaluators.

METHODS

Residents and faculty were trained and supported in the use of a novel trainee-initiated web-based evaluation system. Residents were encouraged to use the system to track progress on patient care subcompetencies. Two years of prospectively collected data were reviewed from residents at an academic program. The primary outcome was Spearman's rank correlation between postgraduate year (PGY) and competency level achieved as a measure of validity. Secondary outcomes assessed feasibility, resident self-evaluation versus faculty evaluation, the distributions among subcompetencies, and responsiveness over time.

RESULTS

Between February 2014 and February 2016, 856 orthopedic surgery patient care subcompetency evaluations were completed (1.2 evaluations per day). Residents promptly requested feedback after a procedure (median = 0 days, interquartile range: 0-2), and faculty responded within 2 days in 51% (median = 2 days, interquartile range: 0-13). Primary outcome showed a correlation between PGY and competency level (r = 0.78, p < 0.001), with significant differences in competency among PGYs (p < 0.001 by Kruskal-Wallis rank sum test). Self-evaluations by residents substantially agreed with faculty-assigned competency level (weighted Cohen's κ = 0.72, p < 0.001). Resident classes beginning the study as PGY1, 2, and 3 separately demonstrated gains in competency over time (Spearman's rank correlation 0.39, 0.60, 0.59, respectively, each p < 0.001). There was significant variance in the number of evaluations submitted per subcompetency (median = 43, range: 6-113) and competency level assigned (p < 0.01).

CONCLUSIONS

Rapid tracking of trainee competency with milestone-based evaluations in a learner-centered mobile platform demonstrated validity, feasibility, and responsiveness. Next Accreditation System-mandated data may be efficiently collected and used for trainee and program self-study.

New Paradigms in Post-Graduate Surgical Education

With a growing focus on patient safety and trainee education, the Accreditation Council for Graduate Medical Education implemented changes including work hour restrictions, focused clinical competencies, and the Next Accreditation System (NAS). The NAS poses initial challenges on residencies with the implementation of surgical simulation programs and defining resident competency. It is the hope that innovative training methods will allow for improved advancement of knowledge and surgical skills given the current direction of post-graduate surgical training.