Faculty evaluations of resident medical knowledge: can they be used to predict American Board of Surgery In-Training Examination performance?

Gender and graduating results in the Anesthesiology and Intensive Care Abomey-Calavi (Cotonou, Benin) program

Background: For unknown reasons female participation in anesthesiology is very low in Sub-Saharan Africa (SSA), especially in West Africa, and few women reach top academic or clinical positions. Objective: Women reduced professional perspectives. Design: Male and female residents’ performances were compared when they presented for their first try the graduating exams of the specialty. Settings: The Cotonou anesthesiology and intensive care training center, the second largest in French-speaking SSA, where 146 anesthesiologists from 14 African countries graduated since its creation in 1996. Method: All results at their final exams (consisting in 3 written questions and 2 clinical evaluations) were retrieved for the 125 men and 21 women who graduated. Scores obtained by women and males were compared using Student’s t tests. Their total of points was used to divide graduates into deciles. The proportion of women was counted in each decile. Results: Women performed better at both anesthesia and intensive care clinical evaluations taken separately and together (total 68.2% vs. 64.2% p=0.004) and were even with men for the three written exams (anesthesia, intensive care and basic sciences - total 66.2 % vs. 66.1% p=0.99). When clinical and written scores are added in each sector, women scored better than males for anesthesiology (69.2% vs. 65.2% p=0.01) and were even for intensive care (65.0% vs. 64.1% p=0.51). Globally women and men results were similar (67% vs. 65%, p=0.1) The proportion of women in each decile increased from the lowest to the best deciles, but the absolute low number of women gives this trend borderline significance (slope +1,56 % women per decile, p=0.046)**. Conclusion: Women performance at end-specialty exams is unlikely to explain their subsequent underrepresentation at the academic level in anesthesia and intensive care in SSA**.

Prospective Factors that Predict American Board of Surgery In-Training Examination Performance: A Systematic Review

INTRODUCTION

American Board of Surgery In-Training Examination (ABSITE) performance has become an important factor when monitoring resident progress. Understanding which prospective factors predict performance can help identify residents at risk.

METHODS

A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Prospective factors such as prior examination performance, clinical evaluations, and demographics were evaluated.

RESULTS

A final 35 studies were included. The prospective factor most consistently found to predict ABSITE performance is performance on prior knowledge-based examinations such as the USMLE step exams. The ACGME Medical Knowledge 1 milestone evaluation also appears to correlate to ABSITE performance, although clinical evaluations, in general, do not. Demographics have no significant correlation to ABSITE performance.

DISCUSSION

Using performance on prior knowledge-based examinations programs may be able to identify residents at risk for failing ABSITE. It may be possible to initiate early intervention before rather than only remediation after poor performance.

Early Detection and Remediation of Problem Learners

There are myriad types of problem learners in surgical residency and most have difficulty in more than 1 competency. Programs that use a standard curriculum of study and assessment are most successful in identifying struggling learners early. Many problem learners lack appropriate systems for study; a multidisciplinary educational team that is separate from the team that evaluates the success of remediation is critical. Struggling residents who require formal remediation benefit from performance improvement plans that clearly outline the issues of concern, describe the steps required for remediation, define success of remediation, and outline consequences for failure to remediate appropriately.

USMLE Step 1 Is Pass/Fail - Should the ABSITE Follow Suit?

The United States Medical Licensing Examination (USMLE) Step 1 was recently made pass/fail. This decision was controversial largely because of the reliance on USMLE Step 1 scores in resident selection. However, these scores do not correlate with resident ability. In this manuscript, we consider if the American Board of Surgery In-Training Examination (ABSITE) should be pass/fail as well. The ABSITE has been used for "high-stakes" purposes, such as preliminary resident advancement and prospective fellow evaluation, for which it was not intended. Moreover, similar to the USMLE Step 1 exam, ABSITE scores have demonstrated no correlation with clinical ability. A pass/fail ABSITE would return the exam to its original purpose and minimize an over-reliance on scores. Moving forward, new objective measures will need to be developed to assess surgical trainees in a more holistic manner.

A Novel Method of Evaluating Key Factors for Success in a Multifaceted Critical Care Fellowship Using Data Envelopment Analysis

BACKGROUND

The current system of summative multi-rater evaluations and standardized tests to determine readiness to graduate from critical care fellowships has limitations. We sought to pilot the use of data envelopment analysis (DEA) to assess what aspects of the fellowship program contribute the most to an individual fellow's success. DEA is a nonparametric, operations research technique that uses linear programming to determine the technical efficiency of an entity based on its relative usage of resources in producing the outcome.

DESIGN

Retrospective cohort study.

SUBJECTS AND SETTING

Critical care fellows (n = 15) in an Accreditation Council for Graduate Medical Education (ACGME) accredited fellowship at a major academic medical center in the United States.

METHODS

After obtaining institutional review board approval for this retrospective study, we analyzed the data of 15 anesthesiology critical care fellows from academic years 2013-2015. The input-oriented DEA model develops a composite score for each fellow based on multiple inputs and outputs. The inputs included the didactic sessions attended, the ratio of clinical duty works hours to the procedures performed (work intensity index), and the outputs were the Multidisciplinary Critical Care Knowledge Assessment Program (MCCKAP) score and summative evaluations of fellows.

RESULTS

A DEA efficiency score that ranged from 0 to 1 was generated for each of the fellows. Five fellows were rated as DEA efficient, and 10 fellows were characterized in the DEA inefficient group. The model was able to forecast the level of effort needed for each inefficient fellow, to achieve similar outputs as their best performing peers. The model also identified the work intensity index as the key element that characterized the best performers in our fellowship.

CONCLUSIONS

DEA is a feasible method of objectively evaluating peer performance in a critical care fellowship beyond summative evaluations alone and can potentially be a powerful tool to guide individual performance during the fellowship.

Are Surgical Milestone Assessments Predictive of In-Training Examination Scores?

OBJECTIVES

With the recent utilization of Accreditation Council for Graduate Medical Education developmental milestones as part of resident evaluation, we sought to explore whether milestone-based ratings were associated with American Board of Surgery In-Training Examination (ABSITE) scores.

METHODS

Mid-year milestone ratings were obtained from the Accreditation Council for Graduate Medical Education Accreditation Database System for years 2014, 2015, and 2016 for all postgraduate years 1-5 general surgery residents in our program and paired with ABSITE scores (n = 69) from January of the following year. Linear regression was used to assess predictor importance of milestones on both ABSITE percentage correct scores and ABSITE percentile scores.

RESULTS

Minimal to small positive correlations were observed between milestones and ABSITE percentile scores (r = 0.09-0.25), while moderately large correlations were observed between milestones and percentage correct scores (r = 0.65-0.76). The Medical Knowledge 1 (MK1) milestone significantly predicted ABSITE percentage correct scores, and explained 60% of the variance (adjusted R² = 0.603). MK1 also significantly predicted ABSITE percentile scores, although weaker in magnitude, with MK1 explaining 20% of the variance (adjusted R² = 0.197). Postgraduate year level and other milestones were not influential predictors of ABSITE scores.

CONCLUSIONS

The mid-year MK1 milestone rating was predictive of ABSITE scores and may serve as a useful marker for Clinical Competency Committees to identify residents who could benefit from additional support to prepare for the ABSITE, although given the small exploratory nature of this study, additional research is still needed.

Development of a Tool to Assess Basic Competency in the Performance of Rigid Bronchoscopy

RATIONALE

Rigid bronchoscopy is increasingly used by pulmonologists for the management of central airway disorders. However, an assessment tool to evaluate the competency of operators in the performance of this technique has not been developed. We created the Rigid Bronchoscopy Tool for Assessment of Skills and Competence (RIGID-TASC) to serve as an objective, competency-oriented assessment tool of basic rigid bronchoscopic skills, including rigid bronchoscopic intubation and central airway navigation.

OBJECTIVES

To assess whether RIGID-TASC scores accurately distinguish the basic rigid bronchoscopy skills of novice, intermediate, and expert operators, and to determine whether RIGID-TASC has adequate interrater reliability when used by different independent testers.

METHODS

At two academic medical centers in the United States, 30 physician volunteers were selected in three categories: 10 novices at rigid bronchoscopy (performed at least 50 flexible, but no rigid, bronchoscopies), 10 operators with intermediate experience (performed 5-20 rigid bronchoscopies), and 10 experts (performed ≥100 rigid bronchoscopies). Participants included pulmonary and critical care fellows, interventional pulmonology fellows, and faculty interventional pulmonologists. Each subject then performed rigid bronchoscopic intubation and navigation on a manikin, while being scored independently by two testers, using RIGID-TASC.

MEASUREMENTS AND MAIN RESULTS

Mean scores for three categories (novice, intermediate, and expert) were 58.10 (±4.6 [SE]), 78.15 (±3.8), and 94.40 (±1.1), respectively. There was significant difference between novice and intermediate (20.05, 95% confidence interval [CI] = 7.77-32.33, P = 0.001), and intermediate and expert (16.25, 95% CI = 3.97-28.53, P = 0.008) operators. The interrater reliability (intraclass correlation coefficient) between the two testers was high (r = 0.95, 95% CI = 0.90-0.98).

CONCLUSIONS

RIGID-TASC showed evidence of construct validity and interrater reliability in this setting and group of subjects. It can be used to reliably and objectively score and classify operators from novice to expert in basic rigid bronchoscopic intubation and navigation.