A National Survey of Integrated Vascular Surgery Residents' Experiences With and Attitudes About Quality Improvement During Residency

The Use of Learning Analytics to Enable Detection of Underperforming Trainees: An Analysis of National Vascular Surgery Trainee ACGME Milestones Assessment Data

OBJECTIVE

This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets.

SUMMARY BACKGROUND DATA

ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation.

METHODS

National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 subcompetencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings.

RESULTS

A total of 122 vascular surgery fellows (VSFs) (95.3%) and 52 integrated vascular surgery residents (IVSRs) (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6% to 25.4% across subcompetencies, which was not significantly different between the 2 groups for any of the subcompetencies ( P = 0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of 2.5, with 1 year remaining before graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% to 77.9% for VSFs and 33.3% to 75.0% for IVSRs.

CONCLUSION

The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

Pediatric emergency medicine fellows' milestone evaluations: Do they all meet the targets for graduation?

BACKGROUND

The ACGME Milestone Project created a competency-based trainee assessment tool. Subcompetencies (SCs) are scored on a 5-point scale; level 4 is recommended for graduation. The 2018 Milestones Report found that across subspecialties, not all graduates attain level 4 for every SC.

OBJECTIVE

The objective was to describe the number of pediatric emergency medicine (PEM) fellows who achieve ≥ level 4 in all 23 SCs at graduation and identify SCs where level 4 is not achieved and factors predictive of not achieving a level 4.

METHODS

This is a multicenter, retrospective cohort study of PEM fellows from 2014 to 2018. Program directors provided milestone reports. Descriptive analysis of SC scores was performed. Subanalyses assessed differences in residency graduation scores, first-year fellowship scores, and the rate of milestone attainment between fellows who did and did not attain ≥ level 4 at graduation.

RESULTS

Data from 392 fellows were obtained. There were no SCs in which all fellows attained ≥ level 4 at graduation; the range of fellows scoring < level 4 per SC was 7% to 39%. A total of 67% of fellows did not attain ≥ level 4 on one or more SC. While some fellows failed to attain ≥ level 4 on up to all 23 SCs, 26% failed to meet level 4 on only one or two. In 19 SCs, residency graduation and/or first year fellow scores were lower for fellows who did not attain ≥ level 4 at graduation compared to those who did (mean difference = 0.74 points). Among 10 SCs, fellows who did not attain ≥ level 4 at graduation had a faster rate of improvement compared to those who did attain ≥ level 4.

CONCLUSION

In our sample, 67% of PEM fellows did not attain level 4 for one or more of the SCs at graduation. Low scores during residency or early in fellowship may predict difficulty in meeting level 4 by fellowship completion.

Using Learning Analytics to Examine Achievement of Graduation Targets for Systems-Based Practice and Practice-Based Learning and Improvement: A National Cohort of Vascular Surgery Fellows

BACKGROUND

Surgeons provide patient care in complex health care systems and must be able to participate in improving both personal performance and the performance of the system. The Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery Milestones are utilized to assess vascular surgery fellows' (VSF) achievement of graduation targets in the competencies of Systems Based Practice (SBP) and Practice Based Learning and Improvement (PBLI). We investigate the predictive value of semiannual milestones ratings for final achievement within these competencies at the time of graduation.

METHODS

National ACGME milestones data were utilized for analysis. All trainees entering the 2-year vascular surgery fellowship programs in July 2016 were included in the analysis (n = 122). Predictive probability values (PPVs) were obtained for each SBP and PBLI sub-competencies by biannual review periods, to estimate the probability of VSFs not reaching the recommended graduation target based on their previous milestones ratings.

RESULTS

The rate of nonachievement of the graduation target level 4.0 on the SBP and PBLI sub-competencies at the time of graduation for VSFs was 13.1-25.4%. At the first time point of assessment, 6 months into the fellowship program, the PPV of the SBP and PBLI milestones for nonachievement of level 4.0 upon graduation ranged from 16.3-60.2%. Six months prior to graduation, the PPVs across the 6 sub-competencies ranged from 14.6-82.9%.

CONCLUSIONS

A significant percentage of VSFs do not achieve the ACGME Vascular Surgery Milestone targets for graduation in the competencies of SBP and PBLI, suggesting a need to improve curricula and assessment strategies in these domains across vascular surgery fellowship programs. Reported milestones levels across all time point are predictive of ultimate achievement upon graduation and should be utilized to provide targeted feedback and individualized learning plans to ensure graduates are prepared to engage in personal and health care system improvement once in unsupervised practice.

Quality improvement education in surgical specialty training: A comparison of Vascular Surgery resident and Urology Resident experiences and attitudes

BACKGROUND

Academic institutions have increasingly focused on educating physicians and surgeons in concepts of value-based care, including quality improvement (QI). The extent to which QI curricular competencies are addressed in specialty surgical residency training is unclear.

METHODS

A survey instrument was developed by content experts and sent to Vascular Surgery and Urology residents electronically. Descriptive statistics and bivariate associations were calculated using StataMP 13.1.

RESULTS

Vascular Surgery and Urology residents reported exposure to similar types of QI curriculum. Fewer than half of residents reported achieving targets for graduation (Vascular 31%, Urology 42%) related to QI, and few residents in either group felt very well-prepared to lead a QI initiative (Vascular 13%, Urology 8%).

CONCLUSION

QI education in surgical specialty training amongst Vascular Surgery and Urology residencies is similar and insufficient. Surgical specialties may benefit from collaborative efforts to improve the quality of QI education.

High value care education in general surgery residency programs: A multi-institutional needs assessment

BACKGROUND

The ACGME mandates that residency programs provide training related to high value care (HVC). The purpose of this study was to explore HVC education in general surgery residency programs.

METHODS

An electronic survey was distributed to general surgery residents in geographically diverse programs.

RESULTS

The response rate was 29% (181/619). Residents reported various HVC components in their curricula. Less than half felt HVC is very important for their future practice (44%) and only 15% felt confident they could lead a QI initiative in practice. Only 20% of residents reported participating in a root cause analysis and less than one-third of residents (30%) were frequently exposed to cost considerations.

CONCLUSION

Few residents feel prepared to lead quality improvement initiatives, have participated in patient safety processes, or are aware of patients' costs of care. This underscores the need for improved scope and quality of HVC education and establishment of formal curricula.