Does the NBME Surgery Shelf exam constitute a “double jeopardy” of USMLE Step 1 performance?

Multivariate Modeling of Student Performance on NBME Subject Exams

Aim This study sought to determine whether it was possible to develop statistical models which could be used to accurately correlate student performance on clinical subject exams based on their National Board of Medical Examiner (NBME) self-assessment performance and other variables, described below, as such tools are not currently available.  Methods Students at a large public medical school were provided fee vouchers for NBME self-assessments before clinical subject exams. Multivariate regression models were then developed based on how self-assessment performance correlated to student success on the subsequent subject exam (Medicine, Surgery, Family Medicine, Obstetrics-Gynecology, Pediatrics, and Psychiatry) while controlling for the proximity of the self-assessment to the exam, USMLE Step 1 score, and the academic quarter. Results The variables analyzed satisfied the requirements of linear regression. The correlation strength of individual variables and overall models varied by discipline and outcome (equated percent correct or percentile, Model R² Range: 0.1799-0.4915). All models showed statistical significance on the Omnibus F-test (p<0.001). Conclusion The correlation coefficients demonstrate that these models have weak to moderate predictive value, dependent on the clinical subject, in predicting student performance; however, this varies widely based on the subject exam in question. The next step is to utilize these models to identify struggling students to determine if their use reduces failure rates and to further improve model accuracy by controlling for additional variables.

Sustained Clinical Performance During Surgical Rotations Predicts NBME Shelf Exam Outcomes

OBJECTIVE

To determine if longitudinal, excellent clinical performance reflected in subjective evaluations during a surgery clerkship would be associated with a greater likelihood of National Board of Medical Examiners Surgery Shelf Exam ("shelf exam") success.

DESIGN

We retrospectively reviewed medical students' surgical clerkship performance from 2014 to 2019. Clinical evaluations for each rotation were abstracted and students were stratified by performance: excellent performers and non-excellent performers. The rotation performance grades were then combined to classify overall clerkship performance: sustained excellent performers, improved performers, worsened performers, and sustained non-excellent performers. We compared the shelf exam scores between performer class for each clinical rotation and the overall clerkship. Using logistic regression, we also sought to determine if clinical performance predicted passing the shelf exam.

SETTING

Emory University School of Medicine in Atlanta, Georgia.

PARTICIPANTS

Third-year medical students (N = 674) who completed a surgery clerkship.

RESULTS

Excellent performers scored higher than non-excellent performers on the shelf exam during both clinical rotations (all p < 0.01). Sustained excellent performers had the highest exam scores out of all the clerkship performance groups (p < 0.0001). Excellent performers for both rotations were associated with increased odds of passing the shelf exam. Sustained excellent performers had the greatest odds (OR 3, 95% confidence interval 1.5-6.3, p = 0.003) of passing the exam.

CONCLUSIONS

Clinical performance during the surgical clerkship and individual rotations correlates with shelf exam scores. Students should be encouraged to excel on the wards to maximize the educational experience and improve their odds of passing the exam.

Using the Clerkship Shelf Exam Score as a Qualification for an Overall Clerkship Grade of Honors: A Valid Practice or Unfair to Students?

Most clerkships require medical students to pass the National Board of Medical Examiners (NBME) subject (shelf) exam to pass the clerkship. Many use the NBME's recommended honors cut score on the shelf exam to determine medical student eligibility for an overall clerkship grade of honors. This use of a conjunctive scoring model for determining honors is inconsistent with the logic behind the intended use of this model for making pass-fail determinations. Further, many clerkships use grading systems that employ both this conjunctive model for honors eligibility and a compensatory scoring model for determining the overall clerkship grade. For students who fall short of the shelf exam honors cut score, such a grading system effectively increases the weighting of shelf exam performance and decreases the clerkship's transparency about the weighting of performance on other assessments toward the clerkship composite score and overall grade. It may also lead to contradictory grading results in which student B has a higher composite score than student A, yet student B receives a lower overall grade. The author illustrates how to calculate a weight for shelf exam performance that would be fairer to students and would help create a more transparent weighting scheme for the grading system. The author recommends that clerkships restructure their grading systems so that shelf exam honors-eligibility cut scores are not used as conjunctive criteria for determining overall clerkship grades of honors. A reexamination of the NBME's practice of suggesting honors-eligibility cut scores for shelf exams is also recommended.

All work and no play: Addressing medical students' concerns about duty hours on the surgical clerkship

BACKGROUND

Given the option of preferencing rotations for a 3rd year core surgery clerkship, we observed students often requested services perceived as less time-intensive. We compare self-reported duty hours with academic outcomes.

METHODS

We examined duty hours from 165 third-year medical students on a surgery clerkship at a single institution for academic year 2016-2017. Partial correlations and logistic regression modeling were used to assess the number of hours medical students worked on academic outcomes.

RESULTS

Medical student duty hours did not significantly correlate with the NBME Surgery Subject examination score (r = 0.08; p = 0.34), CPE score (r = 0.14; p = 0.09) or a clerkship grade of Honors (OR 0.993; CI 0.925-1.065). Prior completion of an internal medicine clerkship was correlated with a higher NBME Surgery Subject examination score (r = 0.27; p < 0.001).

CONCLUSION

This analysis demonstrates duty hours on a surgical clerkship do not correlate with academic performance. These data can be used to counsel students on career planning and choosing surgical rotations based on interest and not perceived workload.