Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons' Early Outcomes

Importance

National data on the development of competence during training have been reported using the Accreditation Council for Graduate Medical Education (ACGME) Milestones system. It is now possible to consider longitudinal analyses that link Milestone ratings during training to patient outcomes data of recent graduates.

Objective

To evaluate the association of in-training ACGME Milestone ratings in a surgical specialty with subsequent complication rates following a commonly performed operation, endovascular aortic aneurysm repair (EVAR).

Design, Setting, and Participants

This study of patient outcomes followed EVAR in the Vascular Quality Initiative (VQI) registry (4213 admissions from 208 hospitals treated by 327 surgeons). All surgeons included in this study graduated from ACGME-accredited training programs from 2015 through 2019 and had Milestone ratings 6 months prior to graduation. Data were analyzed from December 1, 2021, through September 15, 2023. Because Milestone ratings can vary with program, they were corrected for program effect using a deviation score from the program mean.

Exposure

Milestone ratings assigned to individual trainees 6 months prior to graduation, based on judgments of surgical competence.

Main Outcomes and Measures

Surgical complications following EVAR for patients treated by recent graduates during the index hospitalization, obtained using the nationwide Society for Vascular Surgery Patient Safety Organization's VQI registry, which includes 929 participating centers in 49 US states.

Results

The study included outcomes for 4213 patients (mean [SD] age, 73.25 [8.74] years; 3379 male participants [80.2%]). Postoperative complications included 9.5% major (400 of 4213 cases) and 30.2% minor (1274 of 4213 cases) complications. After adjusting for patient risk factors and site of training, a significant association was identified between individual Milestone ratings of surgical trainees and major complications in early surgical practice in programs with lower mean Milestone ratings (odds ratio, 0.50; 95% CI; 0.27-0.95).

Conclusions and Relevance

In this study, Milestone assessments of surgical trainees were associated with subsequent clinical outcomes in their early career. Although these findings represent one surgical specialty, they suggest Milestone ratings can be used in any specialty to identify trainees at risk for future adverse patient outcomes when applying the same theory and methodology. Milestones data should inform data-driven educational interventions and trainee remediation to optimize future patient outcomes.

Medical School Rankings: Time to End or Time to Amend?

ABSTRACT

The recent decisions of several medical schools to no longer participate in the rankings published annually by U.S. News & World Report have added greater visibility to the issues surrounding medical school rankings than ever before. While these announcements garnered significant attention in academic medicine and from the lay public, the authors believe these events are unlikely to eliminate rankings in medical education. In this article, the authors outline the potential harms of rankings in higher education, provide a rationale for why they will likely endure despite recent scrutiny, and offer suggestions for prospective students and the academic medical community to engage with rankings in a more productive way. Criticisms of medical school rankings have been noted for decades, including their use of subjective information and data with questionable relevance to the quality of students' education or the outcomes of their educational experience, methodological concerns, and the potential harms of ranking systems (such as schools diverting their focus and resources away from initiatives that benefit students to improve their ranking). At the same time, rankings are ubiquitous in today's culture and are powerful in human decision-making, so there is reason to believe that medical school rankings may weather the current storm. Given these concerns, the authors suggest students continue to use a variety of resources to obtain information about medical schools and consider how each school fits with their own educational needs. The authors also encourage medical schools, medical education organizations, and governing medical bodies to suggest different metrics that reflect quality in medical education and that are of importance to applicants.

The Association of ACGME Milestones With Performance on American Board of Surgery Assessments: A National Investigation of Surgical Trainees

OBJECTIVE

To determine the relationship between, and predictive utility of, milestone ratings and subsequent American Board of Surgery (ABS) vascular surgery in-training examination (VSITE), vascular qualifying examination (VQE), and vascular certifying examination (VCE) performance in a national cohort of vascular surgery trainees.

BACKGROUND

Specialty board certification is an important indicator of physician competence. However, predicting future board certification examination performance during training continues to be challenging.

METHODS

This is a national longitudinal cohort study examining relational and predictive associations between Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings and performance on VSITE, VQE, and VCE for all vascular surgery trainees from 2015 to 2021. Predictive associations between milestone ratings and VSITE were conducted using cross-classified random-effects regression. Cross-classified random-effects logistic regression was used to identify predictive associations between milestone ratings and VQE and VCE.

RESULTS

Milestone ratings were obtained for all residents and fellows(n=1,118) from 164 programs during the study period (from July 2015 to June 2021), including 145,959 total trainee assessments. Medical knowledge (MK) and patient care (PC) milestone ratings were strongly predictive of VSITE performance across all postgraduate years (PGYs) of training, with MK ratings demonstrating a slightly stronger predictive association overall (MK coefficient 17.26 to 35.76, β = 0.15 to 0.23). All core competency ratings were predictive of VSITE performance in PGYs 4 and 5. PGY 5 MK was highly predictive of VQE performance [OR 4.73, (95% CI, 3.87-5.78), P <0.001]. PC subcompetencies were also highly predictive of VQE performance in the final year of training [OR 4.14, (95% CI, 3.17-5.41), P <0.001]. All other competencies were also significantly predictive of first-attempt VQE pass with ORs of 1.53 and higher. PGY 4 ICS ratings [OR 4.0, (95% CI, 3.06-5.21), P <0.001] emerged as the strongest predictor of VCE first-attempt pass. Again, all subcompetency ratings remained significant predictors of first-attempt pass on CE with ORs of 1.48 and higher.

CONCLUSIONS

ACGME Milestone ratings are highly predictive of future VSITE performance, and first-attempt pass achievement on VQE and VCE in a national cohort of surgical trainees.

Using eHealth Data to Inform CPD for Medical Practitioners: A Scoping Review with a Consultation Exercise with International Experts

INTRODUCTION

eHealth data analytics is widely used in health care research. However, there is limited knowledge on the role of eHealth data analysis to inform continuing professional development (CPD). The aim of this study was to collate available research evidence on the use of eHealth data for the development of CPD programs and plans for medical practitioners.

METHODS

A scoping review was conducted using the six-stage Arksey and O'Malley Framework. A consultation exercise (stage 6) was performed with 15 international experts in the fields of learning and practice analytics to deepen the insights.

RESULTS

Scoping review. The literature searches identified 9876 articles published from January 2010 to May 2022. After screening and full-text review, a total of nine articles were deemed relevant for inclusion. The results provide varied-and at times partial or diverging-answers to the scoping review research questions. Consultation exercise. Research rigor, field of investigation, and developing the field were the three themes emerged from analysis. Participants validated the scoping review methodology and confirmed its results. Moreover, they provided a meta-analysis of the literature, a description of the current CPD ecosystem, and clear indications of what is and should be next for the field.

DISCUSSION

This study shows that there is no formal or well-established correlation between eHealth data and CPD planning and programming. Overall findings fill a gap in the literature and provide a basis for further investigation. More foundational work, multidisciplinary collaborations, and stakeholders' engagement are necessary to advance the use of eHealth data analysis for CPD purposes.