Surgery Clerkship Evaluations Are Insufficient for Clinical Skills Appraisal: The Value of a Medical Student Surgical Objective Structured Clinical Examination

Medical student perceptions of assessments of clinical reasoning in a general surgery clerkship

BACKGROUND

Components factoring into general surgery clerkship grades vary by institution, and while evaluators attempt to remain unbiased when evaluating medical student performance, subjectivity and implicit bias remain an issue. Our institution recently implemented a case-based structured oral examination to provide the general surgery clerkship director objective insight into students' clinical reasoning skills. We hypothesized that medical students believe this exam, along with graded clinical documentation and the Observed Standardized Clinical Encounter (OSCE), are fair assessments and increase students' awareness of their clinical reasoning skills.

METHODS

A survey was sent to third-year medical students in the classes of 2023 and 2024 at our institution who had completed their general surgery clerkship. Students rated five grading assessments (i.e., preceptor evaluations, the oral examination, clinical documentation, the OSCE, and the shelf exam) on fairness and the ability of the assessment to give them insight into their clinical reasoning on a five-point Likert scale 1-5 (with 1 = Strongly Agree, 5 = Strongly Disagree).

RESULTS

One hundred and ten of 162 (67.9%) students responded to the survey. The shelf examination was the most highly regarded assessment tool followed by the oral examination. Seventy-three percent agreed or strongly agreed that the oral exam was a fair assessment, and 80% agreed or strongly agreed that it gave them insight into their clinical reasoning skills. Alternatively, only 41.8% of students agreed or strongly agreed that preceptor evaluations were fair assessments and 42.7% agreed or strongly agreed that it gave them insight into their clinical reasoning.

CONCLUSIONS

Third-year medical students on a general surgery clerkship favor the shelf examination and a case-based oral examination over other assessment tools regarding fairness and perception of their clinical reasoning. This type of examination can provide general surgery clerkship directors with additional objective data to assess medical students more fairly and improve students' clinical reasoning.

Comparability of Objective Structured Clinical Examinations (OSCEs) and Written Tests for Assessing Medical School Students' Competencies: A Scoping Review

Objective Structured Clinical Examinations (OSCEs) and written tests are commonly used to assess health professional students, but it remains unclear whether the additional human resources and expenses required for OSCEs, both in-person and online, are worthwhile for assessing competencies. This scoping review summarized literature identified by searching MEDLINE and EMBASE comparing 1) OSCEs and written tests and 2) in-person and online OSCEs, for assessing health professional trainees' competencies. For Q1, 21 studies satisfied inclusion criteria. The most examined health profession was medical trainees (19, 90.5%), the comparison was most frequently OSCEs versus multiple-choice questions (MCQs) (18, 85.7%), and 18 (87.5%) examined the same competency domain. Most (77.5%) total score correlation coefficients between testing methods were weak (r < 0.40). For Q2, 13 articles were included. In-person and online OSCEs were most used for medical trainees (9, 69.2%), checklists were the most prevalent evaluation scheme (7, 63.6%), and 14/17 overall score comparisons were not statistically significantly different. Generally low correlations exist between MCQ and OSCE scores, providing insufficient evidence as to whether OSCEs provide sufficient value to be worth their additional cost. Online OSCEs may be a viable alternative to in-person OSCEs for certain competencies where technical challenges can be met.

Examination Scores but not Clinical Performance Correlate With Duration of Preclinical Didactic Time: A Synchronous Comparison of Second- Versus Third-Year Medical Students on the Surgery Clerkship

OBJECTIVE

Numerous institutions have reduced preclinical didactic time to facilitate earlier clinical exposure during the second year of medical education. However, the effects that shortened preclinical education may have on performance in the surgery clerkship are unclear. This study aims to compare the clinical and examination performance of second- (MS2) and third-year (MS3) students synchronously completing an identical surgery clerkship.

DESIGN

All students completing the surgery clerkship (identical didactics, examinations, clinical rotations, etc.) were included. MS3s received 24 months of preclinical education, whereas MS2s received 14 months. Performance outcomes included weekly quizzes based on lectures, NBME Surgery Shelf Exam, numeric clinical evaluations, objective structured clinical examination (OSCE) scores, and overall clerkship grades.

SETTING

University of Miami Miller School of Medicine.

PARTICIPANTS

All second- (MS2) and third-year (MS3) medical students completing the Surgery Clerkship over 1 year (n = 395).

RESULTS

There were 199 MS3 (50%) and 196 MS2 (50%) students. MS3s demonstrated higher median shelf exams (77% vs 72% MS2s), weekly quiz score averages (87% vs 80% MS2s), clinical evaluations (96% vs 95% MS2s), and overall clerkship grades (89% vs 87% MS2s), all p < 0.020. There was no difference in median OSCE performance (both 92%; p = 0.499). A greater proportion of MS3 students performed in the highest 50% of weekly quiz scores (57% vs 43% MS2), NBME shelf exams (59% vs 39% MS2), and overall clerkship grades (45% vs 37% MS2), all p < 0.010. No significant difference in the proportion of students placing in the top 50% of clinical parameters including the OSCE (48% MS3 vs 46% MS2; p = 0.106) and clinical evaluations (45% MS3 vs 38%; p = 0.185) was observed.

CONCLUSIONS

Although the duration of preclerkship education may correspond to examination scores, MS2s and MS3s perform similarly on clinical metrics. Future strategies to enhance available preclinical didactic time and preparation for examinations are needed.

Effects of Recorded versus Live Teleconference Didactic Lectures on Medical Student Performance in the Surgery Clerkship

OBJECTIVE

Due to the COVID-19 pandemic, numerous institutions converted medical education didactics to electronic formats including both live teleconference didactics and recorded faculty lectures. This study aims to compare the effect of recorded versus live teleconference didactic lectures on medical student examination scores during the surgery clerkship.

DESIGN

Medical students completing the Surgery Clerkship received a weekly series of didactic lectures taught by faculty via a teleconference (2020-2021 academic year) or recorded format (2021-2022 academic year). Performance outcomes included weekly quizzes, National Board of Medical Examiners (NBME) Surgery Shelf Exam, and clerkship Objective Structured Clinical Examination (OSCE) scores.

SETTING

University of Miami Miller School of Medicine.

PARTICIPANTS

All second- (MS2) and third-year (MS3) medical students completing the Surgery Clerkship over two academic years (n = 312).

RESULTS

Students who received live teleconference lectures (n = 156) demonstrated higher average scores on weekly quizzes (89%) and the NBME shelf exam (76%) compared to those receiving recorded lectures (n = 156; 71% quiz, 70% shelf exam), both p < 0.001. There was a significant association with performance in the highest quartile (Q1) of weekly quiz scores and receiving live lectures (40% vs. recorded lectures 1%, p < 0.001). Comparing only MS3 students, mean weekly quiz scores and Q1 achievement were significantly higher (both p < 0.001) in the teleconference cohort with no significant difference in NBME shelf exam performance (p = 0.971). No difference in OSCE performance was observed between groups.

CONCLUSION

These results suggest that synchronous teleconferences may be more effective than recorded lectures for achieving institutional learning objectives on the surgery clerkship without any negative impact on NBME shelf exam or clinical evaluation parameters. This information should be used to inform future institutional clerkship design and educational initiatives.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Surgical clerkship: Do examination scores correlate with clinical performance?

BACKGROUND

This study aims to determine if there are correlations between clinical performance and objective grading parameters for medical students in the third-year surgery clerkship.

METHODS

Clerkship grades were compiled from 2016 to 2020. Performance on clinical rotations, NBME shelf exam, oral exam, and weekly quizzes were reviewed. Students were divided into quartiles (Q1-Q4) based on clinical performance. Standard statistical analysis was performed.

RESULTS

There were 625 students included in the study. Students in Q1+Q2 were more likely than those in Q3+Q4 to score in the top quartile on the shelf exam (29% vs. 19%, p = 0.002), oral exam (24% vs. 17%, p = 0.032), and quizzes (22% vs. 15%, p = 0.024). However, there was negligible correlation between clinical performance and performance on objective measures: shelf exam (R² = 0.027, p < 0.001), oral exam (R² = 0.021, p < 0.001), and weekly quizzes (R² = 0.053, p = 0.092).

CONCLUSIONS

Clinical performance does not correlate with objective grading parameters for medical students in the third-year surgery clerkship.

Does surgical observed structured clinical exam (OSCE) predict clerkship grade, shelf exam scores, and preceptor clinical evaluation?

BACKGROUND

Clinical evaluation of medical student performance has been criticized as variable and subjective. The aim of this study was to assess the correlation of a summative surgical OSCE exam to clinical faculty evaluations as well as surgery shelf exam score and final grades.

METHODS

The performance of 392 students who completed the surgical clerkship between 2017 and 2019 was assessed via Pearson Coefficients comparing OSCE grades, clinical evaluations of Medical Knowledge and Patient Care, Communication and Professionalism, the National Board of Medical Examiners (NBME) shelf surgical subject exam, and final clerkship grade.

RESULTS

Results demonstrate a statistically significant positive relationship between the OSCE, Shelf score and grade, final clerkship grade, and all clinical evaluations except Communication skills. The greatest correlation occurred between OSCE and shelf scores and grades. Although significant, the degree of correlation with clinical observation was significantly less.

CONCLUSION

This study demonstrates that a surgical OSCE has a small positive correlation with clinical knowledge as measured by the NBME shelf exam. There is also an equal correlation with medical knowledge standards, with the OSCE better predicting NBME shelf outcome. This lower correlation to clinical assessment suggests that either the clinical grades contain elements not detected on an OSCE exam but could also support the hypothesis that variability in clinical grades do contain a significant degree of subjectivity.

Predicting success: A comparative analysis of student performance on the surgical clerkship and the NBME surgery subject exam

Background

The National Board of Medical Examiners surgery shelf is a well-established terminal measure of student medical knowledge. No study has explored the correlation between intraclerkship quizzes and shelf exam performance.

Methods

Weekly quiz and National Board of Medical Examiners scores were collected from 156 third-year students who participated in a 12-week surgical clerkship from 2015 to 2017. Kruskal-Wallis, Wilcoxon rank sum, and linear regression analysis was completed.

Results

Trauma/Burns, Esophagus/Anorectal, and Wound/Intensive Care Unit quiz content corresponded with increased National Board of Medical Examiners performance with β-coefficients of 1.57 (P < .001), 1.42 (P < .001), 1.38 (P < .001), respectively. Wound/Intensive Care Unit and Cardio/Vascular content corresponded with decreased likelihood of scoring < 70 points on the National Board of Medical Examiners (OR: 0.75 (P = .03), and 0.68 (P = .02)). Aggregate quiz scores stratified by academic block were 67 (IQR 64–69.5), 77 (IQR 74.5–80), 76.5 (IQR of 67–89.5), 83 (IQR of 76–85) corresponding to academic blocks 1, 2, 3, and 4, respectively (P < .001).

Conclusion

Modeling National Board of Medical Examiners outcomes as a function of weekly quizzes taken during a 12-week surgery clerkship is a viable concept.

•

Intra-clerkship quiz scores correlate with NBME surgical shelf performance.

•

Wound/ICU, esophagus/anorectal, trauma/burns content was “high-yield.”

•

Wound/ICU and Cardiac/Vascular quizzes were predictive of NBME scores.

•

Clerkship participation later in the third year correlates with increased performance.

Development of an Objective Structured Clinical Examination as a Component of Assessment for Initial Board Certification in Anesthesiology

With its first administration of an Objective Structured Clinical Examination (OSCE) in 2018, the American Board of Anesthesiology (ABA) became the first US medical specialty certifying board to incorporate this type of assessment into its high-stakes certification examination system. The fundamental rationale for the ABA's introduction of the OSCE is to include an assessment that allows candidates for board certification to demonstrate what they actually "do" in domains relevant to clinical practice. Inherent in this rationale is that the OSCE will capture competencies not well assessed in the current written and oral examinations-competencies that will allow the ABA to judge whether a candidate meets the standards expected for board certification more properly. This special article describes the ABA's journey from initial conceptualization through first administration of the OSCE, including the format of the OSCE, the process for scenario development, the standardized patient program that supports OSCE administration, examiner training, scoring, and future assessment of reliability, validity, and impact of the OSCE. This information will be beneficial to both those involved in the initial certification process, such as residency graduate candidates and program directors, and others contemplating the use of high-stakes summative OSCE assessments.

Using the Clerkship Shelf Exam Score as a Qualification for an Overall Clerkship Grade of Honors: A Valid Practice or Unfair to Students?

Most clerkships require medical students to pass the National Board of Medical Examiners (NBME) subject (shelf) exam to pass the clerkship. Many use the NBME's recommended honors cut score on the shelf exam to determine medical student eligibility for an overall clerkship grade of honors. This use of a conjunctive scoring model for determining honors is inconsistent with the logic behind the intended use of this model for making pass-fail determinations. Further, many clerkships use grading systems that employ both this conjunctive model for honors eligibility and a compensatory scoring model for determining the overall clerkship grade. For students who fall short of the shelf exam honors cut score, such a grading system effectively increases the weighting of shelf exam performance and decreases the clerkship's transparency about the weighting of performance on other assessments toward the clerkship composite score and overall grade. It may also lead to contradictory grading results in which student B has a higher composite score than student A, yet student B receives a lower overall grade. The author illustrates how to calculate a weight for shelf exam performance that would be fairer to students and would help create a more transparent weighting scheme for the grading system. The author recommends that clerkships restructure their grading systems so that shelf exam honors-eligibility cut scores are not used as conjunctive criteria for determining overall clerkship grades of honors. A reexamination of the NBME's practice of suggesting honors-eligibility cut scores for shelf exams is also recommended.

A Comparison of Objective Assessment Data for the United States and International Medical Graduates in a General Surgery Residency

OBJECTIVE

To compare objective assessment scores between international medical graduates (IMGs) and United States Medical Graduates. Scores of residents who completed a preliminary year, who later matched into a categorical position, were compared to those who matched directly into a categorical position at the Mayo Clinic, Rochester.

DESIGN

Postgraduate year (PGY) 1 to 5 residents participate in a biannual multistation, OSCE-style assessment event as part of our surgical training program. Assessment data were, retrospectively, reviewed and analyzed from 2008 to 2016 for PGY-1 and from 2013 to 2016 for PGY 2 to 5 categorical residents.

SETTING

Academic medical center.

PARTICIPANTS

Categorical PGY 1 to 5 General Surgery (GS) residents at Mayo Clinic Rochester, MN.

RESULTS

A total of 86 GS residents were identified. Twenty-one residents (1 United States Medical Graduates [USMG] and 20 IMGs) completed a preliminary GS year, before matching into a categorical position and 68 (58 USMGs and 10 IMGs) residents, who matched directly into a categorical position, were compared. Mean scores (%) for the summer and winter multistation assessments were higher for PGY-1 trainees with a preliminary year than those without (summer: 59 vs. 37, p < 0.001; winter: 69 vs. 61, p = 0.05). Summer and winter PGY-2 scores followed the same pattern (74 vs. 64, p < 0.01; 85 vs. 71, p < 0.01). For the PGY 3 to 5 assessments, differences in scores between these groups were not observed. IMGs and USMGs scored equivalently on all assessments. Overall, junior residents showed greater score improvement between tests than their senior colleagues (mean score increase: PGY 1-2 = 18 vs. PGY 3-5 = 3, p < 0.001).

CONCLUSIONS

Residents with a previous preliminary GS year at our institution scored higher on initial assessments compared to trainees with no prior GS training at our institution. The scoring advantage of an added preliminary year decreased as trainees progressed through residency.