Low Correlation between Subjective and Objective Measures of Knowledge on Surgery Clerkships

Curricular representation of neurogastroenterology: A survey among medical students in Germany

BACKGROUND

Neurogastroenterological disorders (NGDs) are highly prevalent and substantially impact patients' quality of life. Effective treatment of NGDs depends on the competence and training of medical caregivers. Students' perceived competence in neurogastroenterology and its place in medical school curricula are assessed in this study.

METHODS

A multi-center digital survey among medical students was conducted at five universities. Self-ratings of competence regarding basic mechanisms, diagnosis, and treatment of six chronic medical conditions were assessed. These included irritable bowel syndrome (IBS), gastroesophageal reflux disease, and achalasia. Ulcerative colitis, hypertension, and migraine were included as references.

KEY RESULTS

Of 231 participants, 38% remembered that neurogastroenterology was covered in their curriculum. Highest competence ratings were stated for hypertension and the lowest for IBS. These findings were identical for all institutions irrespective of their curricular model and demographic parameters. Students who remembered neurogastroenterology as a part of their curriculum reported higher competence ratings. According to 72% of students, NGDs should be highlighted more prominently in the curriculum.

CONCLUSIONS & INFERENCES

Despite its epidemiological relevance, neurogastroenterology is only weakly represented in medical curricula. Students report low levels of subjective competence in handling NGDs. In general, assessing the learners' perspective on an empirical basis may enrichen the process of national standardization of medical school curricula.

Measuring what matters: identifying assessments that reflect learning on the core surgical clerkship

Purpose

There are various assessments used during the core surgical clerkship (CSC), each of which may be influenced by factors external to the CSC or have inherent biases from an equity lens. In particular, the National Board of Medical Examiners' Clinical Subject Exams ("Shelf") is used heavily and may not reflect clerkship curriculum or clinical learning.

Methods

This is a retrospective review of medical student characteristics and assessments during the CSC from July 2017-June 2021. Assessment methods included: subjective Clinical Performance Assessments (CPA), Shelf, Objective Structured Clinical Examinations, and a short-answer in-house examination (IHE) culminating in a Final Grade (FG) of Honors/Pass/Fail. A Shelf score threshold for Honors was added in academic years 2020-2021. Descriptive, univariate, and multivariable logistic and linear regression statistics were utilized.

Results

We reviewed records of 192 students. Of these, 107 (55.7%) were female, median age was 24 [IQR: 23-26] years, and most were White/Caucasian (N = 106, 55.2%). Univariate analysis showed the number of Exceeds Expectations obtained on CPA to be influenced by surgical subspecialty taken (p = 0.013) and academic year (p < 0.001). Shelf was influenced by students' race (p = 0.009), timing of CSC before or after Internal Medicine (67.9 ± 7.3 vs 72.9 ± 7.1, p < 0.001), and Term taken (increasing from 66.0 ± 8.7 to 73.4 ± 7.5, p < 0.001). IHE scores did not have any external associations. After adjustment with multivariable logistic and linear regressions, CPA and IHE did not have external associations, but higher scores were obtained on Shelf exam in Terms 3, 5, and 6 (by 4.62 [95% CI 0.86-8.37], 4.92 [95% CI 0.53-9.31], and 7.56 [95% CI 2.81-12.31] points, respectively. Odds of FG honors were lower when Shelf threshold was implemented (OR 0.17 [95% CI 0.06-0.50]), and increased as students got older (OR 1.14 [95% CI 1.01-1.30]) or on specific subspecialties, such as vascular surgery (OR 7.06 [95% CI 1.21-41.26]).

Conclusions

The Shelf is substantially influenced by temporal associations across Terms and timing in relation to other clerkships, such as Internal Medicine. An IHE reflective of a clerkship's specified curriculum may be a more equitable summative assessment of the learning that occurs from the CSC curriculum, with fewer biases or influences external to the CSC.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00047-8.

Surgical clerkship: Do examination scores correlate with clinical performance?

BACKGROUND

This study aims to determine if there are correlations between clinical performance and objective grading parameters for medical students in the third-year surgery clerkship.

METHODS

Clerkship grades were compiled from 2016 to 2020. Performance on clinical rotations, NBME shelf exam, oral exam, and weekly quizzes were reviewed. Students were divided into quartiles (Q1-Q4) based on clinical performance. Standard statistical analysis was performed.

RESULTS

There were 625 students included in the study. Students in Q1+Q2 were more likely than those in Q3+Q4 to score in the top quartile on the shelf exam (29% vs. 19%, p = 0.002), oral exam (24% vs. 17%, p = 0.032), and quizzes (22% vs. 15%, p = 0.024). However, there was negligible correlation between clinical performance and performance on objective measures: shelf exam (R² = 0.027, p < 0.001), oral exam (R² = 0.021, p < 0.001), and weekly quizzes (R² = 0.053, p = 0.092).

CONCLUSIONS

Clinical performance does not correlate with objective grading parameters for medical students in the third-year surgery clerkship.

Sustained Clinical Performance During Surgical Rotations Predicts NBME Shelf Exam Outcomes

OBJECTIVE

To determine if longitudinal, excellent clinical performance reflected in subjective evaluations during a surgery clerkship would be associated with a greater likelihood of National Board of Medical Examiners Surgery Shelf Exam ("shelf exam") success.

DESIGN

We retrospectively reviewed medical students' surgical clerkship performance from 2014 to 2019. Clinical evaluations for each rotation were abstracted and students were stratified by performance: excellent performers and non-excellent performers. The rotation performance grades were then combined to classify overall clerkship performance: sustained excellent performers, improved performers, worsened performers, and sustained non-excellent performers. We compared the shelf exam scores between performer class for each clinical rotation and the overall clerkship. Using logistic regression, we also sought to determine if clinical performance predicted passing the shelf exam.

SETTING

Emory University School of Medicine in Atlanta, Georgia.

PARTICIPANTS

Third-year medical students (N = 674) who completed a surgery clerkship.

RESULTS

Excellent performers scored higher than non-excellent performers on the shelf exam during both clinical rotations (all p < 0.01). Sustained excellent performers had the highest exam scores out of all the clerkship performance groups (p < 0.0001). Excellent performers for both rotations were associated with increased odds of passing the shelf exam. Sustained excellent performers had the greatest odds (OR 3, 95% confidence interval 1.5-6.3, p = 0.003) of passing the exam.

CONCLUSIONS

Clinical performance during the surgical clerkship and individual rotations correlates with shelf exam scores. Students should be encouraged to excel on the wards to maximize the educational experience and improve their odds of passing the exam.

Predicting success: A comparative analysis of student performance on the surgical clerkship and the NBME surgery subject exam

Background

The National Board of Medical Examiners surgery shelf is a well-established terminal measure of student medical knowledge. No study has explored the correlation between intraclerkship quizzes and shelf exam performance.

Methods

Weekly quiz and National Board of Medical Examiners scores were collected from 156 third-year students who participated in a 12-week surgical clerkship from 2015 to 2017. Kruskal-Wallis, Wilcoxon rank sum, and linear regression analysis was completed.

Results

Trauma/Burns, Esophagus/Anorectal, and Wound/Intensive Care Unit quiz content corresponded with increased National Board of Medical Examiners performance with β-coefficients of 1.57 (P < .001), 1.42 (P < .001), 1.38 (P < .001), respectively. Wound/Intensive Care Unit and Cardio/Vascular content corresponded with decreased likelihood of scoring < 70 points on the National Board of Medical Examiners (OR: 0.75 (P = .03), and 0.68 (P = .02)). Aggregate quiz scores stratified by academic block were 67 (IQR 64–69.5), 77 (IQR 74.5–80), 76.5 (IQR of 67–89.5), 83 (IQR of 76–85) corresponding to academic blocks 1, 2, 3, and 4, respectively (P < .001).

Conclusion

Modeling National Board of Medical Examiners outcomes as a function of weekly quizzes taken during a 12-week surgery clerkship is a viable concept.

•

Intra-clerkship quiz scores correlate with NBME surgical shelf performance.

•

Wound/ICU, esophagus/anorectal, trauma/burns content was “high-yield.”

•

Wound/ICU and Cardiac/Vascular quizzes were predictive of NBME scores.

•

Clerkship participation later in the third year correlates with increased performance.

Curricular Innovation in the Surgery Clerkship: Can Assessment Methods Influence Development of Critical Thinking and Clinical Skills?

OBJECTIVE

Although key clinical skills have been defined in the Core Entrustable Professional Activities, there is a need to improve medical school curricula with standardized training opportunities and assessments of these skills. Thus, we aimed to develop an innovative curriculum that emphasized critical thinking and clinical skills. We hypothesized that we would be able to observe measurable improvement on assessments of students' critical thinking and clinical skills after the implementation of the new curriculum.

DESIGN

Prospective, Quasi-Experimental study with the use of historical controls.

SETTING

This study took place through the third-year surgical clerkship at the University of Texas Medical Branch at the Galveston, Houston, and Austin, Texas, locations.

PARTICIPANTS

A total of 214 students taking the third-year surgical clerkship for the first time during the periods of interest were included.

RESULTS

Although the students with traditional curriculum improved 9.5% on a short answer exam from preclerkship to postclerkship completion, the students with new curriculum improved by 40%. Students under the new curriculum performed significantly better on the Objective Structured Clinical Exam; however, their shelf scores were lower.

CONCLUSIONS

Under this new curriculum and grading system, we demonstrated that students can be incentivized to improve critical thinking and clinical skills, but this needs to be balanced with knowledge-based incentives.

Objectivity in subjectivity: do students' self and peer assessments correlate with examiners' subjective and objective assessment in clinical skills? A prospective study

OBJECTIVES

The qualitative subjective assessment has been exercised either by self-reflection (self-assessment (SA)) or by an observer (peer assessment (PA)) and is considered to play an important role in students' development. The objectivity of PA and SA by students as well as those by faculty examiners has remained debated. This matters most when it comes to a high-stakes examination. We explored the degree of objectivity in PA, SA, as well as the global rating by examiners being Examiners' Subjective Assessment (ESA) compared with Objective Structured Clinical Examinations (OSCE).

DESIGN

Prospective cohort study.

SETTING

Undergraduate medical students at Alfaisal University, Riyadh.

PARTICIPANTS

All second-year medical students (n=164) of genders, taking a course to learn clinical history taking and general physical examination.

MAIN OUTCOME MEASURES

A Likert scale questionnaire was distributed among the participants during selected clinical skills sessions. Each student was evaluated randomly by peers (PA) as well as by himself/herself (SA). Two OSCEs were conducted where students were assessed by an examiner objectively as well as subjectively (ESA) for a global rating of confidence and well-preparedness. OSCE-1 had fewer topics and stations, whereas OSCE-2 was terminal and full scale.

RESULTS

OSCE-1 (B=0.10) and ESA (B=8.16) predicted OSCE-2 scores. 'No nervousness' in PA (r=0.185, p=0.018) and 'confidence' in SA (r=0.207, p=0.008) correlated with 'confidence' in ESA. In 'well-preparedness', SA correlated with ESA (r=0.234, p=0.003).

CONCLUSIONS

OSCE-1 and ESA predicted students' performance in the OSCE-2, a high-stakes evaluation, indicating practical 'objectivity' in ESA, whereas SA and PA had minimal predictive role. Certain components of SA and PA correlated with ESA, suggesting partial objectivity given the limited objectiveness of ESA. Such difference in 'qualitative' objectivity probably reflects experience. Thus, subjective assessment can be used with some degree of objectivity for continuous assessment.

Optimizing physician skill development for medical students: The four-part assessment

BACKGROUND

Medical student performance has been poorly correlated with residency performance and warrants further investigation. We propose a novel surgical assessment tool to determine correlations with clinical aptitude.

METHODS

Retrospective review of medical student assessments from 2013 to 2015. Faculty rating of student performance was evaluated by: 1) case presentation, 2) problem definition, 3) question response and 4) use of literature and correlated to final exam assessment. A Likert scale interrater reliability was evaluated.

RESULTS

Sixty student presentations were scored (4.8 assessors/presentation). A student's case presentation, problem definition, and question response was correlated with performance (r = 0.49 to 0.61, p ≤ 0.003). Moderate correlations for either question response or use of literature was demonstrated (0.3 and 0.26, p < 0.05).

CONCLUSION

Our four-part assessment tool identified correlations with course and examination grades for medical students. As surgical education evolves, validated performance and reliable testing measures are required.

Faculty Evaluations Correlate Poorly with Medical Student Examination Performance in a Fourth-Year Emergency Medicine Clerkship

BACKGROUND

Clerkship directors routinely evaluate medical students using multiple modalities, including faculty assessment of clinical performance and written examinations. Both forms of evaluation often play a prominent role in final clerkship grade. The degree to which these modalities correlate in an emergency medicine (EM) clerkship is unclear.

OBJECTIVE

We sought to correlate faculty clinical evaluations with medical student performance on a written, standardized EM examination of medical knowledge.

METHODS

This is a retrospective study of fourth-year medical students in a 4-week EM elective at one academic medical center. EM faculty performed end of shift evaluations of students via a blinded online system using a 5-point Likert scale for 8 domains: data acquisition, data interpretation, medical knowledge base, professionalism, patient care and communication, initiative/reliability/dependability, procedural skills, and overall evaluation. All students completed the National EM M4 Examination in EM. Means, medians, and standard deviations for end of shift evaluation scores were calculated, and correlations with examination scores were assessed using a Spearman's rank correlation coefficient.

RESULTS

Thirty-nine medical students with 224 discrete faculty evaluations were included. The median number of evaluations completed per student was 6. The mean score (±SD) on the examination was 78.6% ± 6.1%. The examination score correlated poorly with faculty evaluations across all 8 domains (ρ 0.074-0.316).

CONCLUSION

Faculty evaluations of medical students across multiple domains of competency correlate poorly with written examination performance during an EM clerkship. Educators need to consider the limitations of examination score in assessing students' ability to provide quality patient clinical care.

Surgery Clerkship Evaluations Are Insufficient for Clinical Skills Appraisal: The Value of a Medical Student Surgical Objective Structured Clinical Examination

OBJECTIVE

Optimal methods for medical student assessment in surgery remain elusive. Faculty- and housestaff-written evaluations constitute the chief means of student assessment in medical education. However, numerous studies show that this approach has poor specificity and a high degree of subjectivity. We hypothesized that an objective structured clinical examination (OSCE) in the surgery clerkship would provide additional data on student performance that would confirm or augment other measures of assessment.

DESIGN

We retrospectively reviewed data from OSCEs, National Board of Medical Examiners shelf examinations, oral presentations, and written evaluations for 51 third-year Harvard Medical School students rotating in surgery at Massachusetts General Hospital from 2014 to 2015. We expressed correlations between numeric variables in Pearson coefficients, stratified differences between rater groups by one-way analysis of variance, and compared percentages with 2-sample t-tests. We examined commentary from both OSCE and clinical written evaluations through textual analysis and summarized these results in percentages.

RESULTS

OSCE scores and clinical evaluation scores correlated poorly with each other, as well as with shelf examination scores and oral presentation grades. Textual analysis of clinical evaluation comments revealed a heavy emphasis on motivational factors and praise, whereas OSCE written comments focused on cognitive processes, patient management, and methods to improve performance.

CONCLUSIONS

In this single-center study, an OSCE provided clinical skills data that were not captured elsewhere in the surgery clerkship. Textual analysis of faculty evaluations reflected an emphasis on interpersonal skills, rather than appraisal of clinical acumen. These findings suggest complementary roles of faculty evaluations and OSCEs in medical student assessment.

Strategies for increasing the feasibility of performance assessments during competency-based education: Subjective and objective evaluations correlate in the operating room

BACKGROUND

Competency-based education necessitates assessments that determine whether trainees have acquired specific competencies. The evidence on the ability of internal raters (staff surgeons) to provide accurate assessments is mixed; however, this has not yet been directly explored in the operating room. This study's objective is to compare the ratings given by internal raters vs an expert external rater (independent to the training process) in the operating room.

METHODS

Raters assessed general surgery residents during a laparoscopic cholecystectomy for their technical and nontechnical performance.

RESULTS

Fifteen cases were observed. There was a moderately positive correlation (r_s = .618, P = .014) for technical performance and a strong positive correlation (r_s = .731, P = .002) for nontechnical performance. The internal raters were less stringent for technical (mean rank 3.33 vs 8.64, P = .007) and nontechnical (mean rank 3.83 vs 8.50, P = .01) performances.

CONCLUSIONS

This study provides evidence to help operationalize competency-based assessments.

Defining Honors in the Surgery Clerkship

BACKGROUND

Although highly influential, no published criteria exist that define who should receive the highest grade in the core surgery clerkship ("honors"). Therefore, significant variability exists in how this evaluation is assigned. Identifying the critical characteristics of the student receiving this grade can improve its usefulness in residency selection, class standing, and direct students' efforts. The purpose of this study was to attain expert consensus on the characteristics of an honors student in the core surgery clerkship.

STUDY DESIGN

A 3-round modified Delphi technique was used in 2 parallel cycles to obtain expert consensus from the major stakeholders-program directors and clerkship directors in surgery. Experts were recruited from across the United States, although not from the same institutions. The 2 consensus lists were evaluated for congruency.

RESULTS

All 15 of the invited clerkship directors and 14 of 15 invited program directors participated. A total of 65 unique characteristics were submitted by program directors and consensus was reached on 23. Clerkship directors submitted 62 characteristics and achieved agreement on 22. Ten of the final characteristics were identical between the 2 groups. These were communication skills, "shelf" exam score, synthetic ability (organizing data into meaningful care plans), absence of professionalism issues, outstanding work ethic, taking advantage of learning opportunities, accurate and complete history and physicals, enthusiasm, becoming an essential member of the care team, and outstanding clinical acumen.

CONCLUSIONS

Expert consensus on the characteristics of an honors student in the core surgery clerkship was achieved. By using these criteria, the honors grade becomes emblematic of these 10 characteristics. This might reduce grade inflation within and between institutions, provide program directors with a consistent and reliable assessment of excellence, and effectively direct student efforts.

Beyond the Ivory Tower: A Comparison of Grades Across Academic and Community OB/GYN Clerkship Sites

CONSTRUCT: Decentralized clinical education is the use of community facilities and community physicians to educate medical students. The theory behind decentralized clinical education is that academic and community sites will provide educational equivalency as determined by objective and subjective performance measures, while training more medical students and exposing students to rural or underserved communities. One of the major challenges of decentralized clinical education is ensuring site comparability in both learning opportunities and evaluation of students.

BACKGROUND

Previous research has examined objective measures of student performance, but less is known about subjective performance measures, particularly in the field of obstetrics and gynecology (OB/GYN). This study explores the implications of clinical site on the adequacy of subjective and objective performance measures.

APPROACH

This was a retrospective cohort study of 801 students in the University of Washington School of Medicine OB/GYN clerkship from 2008 to 2012. Academic sites included those with OB/GYN residency programs (n = 2) and community sites included those without residency programs (n = 29). The association between clerkship site and National Board of Medical Examiners (NBME) grade was assessed using linear regression and clinical and final grade using multinomial regression, estimating β coefficient and relative risks (RR), respectively, and 95% confidence intervals (CIs), adjusting for gender, academic quarter of clerkship, and year of clerkship.

RESULTS

There were no differences in NBME exam grades of students at academic sites (76.4 (7.3) versus 74.6 (8.0), β = -0.11, 95% CI [1.35, 1.12] compared to community sites. For clinical grade, students at community sites were 2.4 times more likely to receive honors relative to high pass (RR 2.45), 95% CI [1.72, 3.50], and for final grade, students at community sites were 1.9 times more likely to receive honors relative to pass (RR 1.98), 95% CI [1.27, 3.09], and 1.6 times more likely to receive honors relative to high pass (RR 1.62), 95% CI [1.05, 2.50], compared to those at academic sites.

CONCLUSIONS

Students at community sites receive higher clinical and final grades in the OB/GYN clerkship. This highlights a significant challenge in decentralized clinical education-ensuring site comparability in clinical grading, Further work should examine the differences in sites, as well as improve standardization of clinical grading. This also underscores an important consideration, as the final grade can influence medical school rank, nomination into honor societies, and ranking of residency applicants.

Correlating surgical clerkship evaluations with performance on the National Board of Medical Examiners examination

BACKGROUND

Evaluation of medical students during the surgical clerkship is controversial. Performance is often based on subjective scoring, whereas objective knowledge is based on written examinations. Whether these measures correspond or are relevant to assess student performance is unknown. We hypothesized that student evaluations correlate with performance on the National Board Of Medical Examiners (NBME) examination.

METHODS

Data were collected from the 2011-2012 academic year. Medical students underwent a ward evaluation using a seven-point Likert scale assessing six educational competencies. Students also undertook the NBME examination, where performance was recorded as a percentile score adjusted to national standards.

RESULTS

A total of 129 medical students were studied. Scores on the NBME ranged from the 52nd to the 96th percentile with an average in the 75th percentile (±9). Clerkship scores ranged from 3.2-7.0 with a mean of 5.7 (±0.8). There was a strong positive association between higher NBME scores and higher clerkship evaluations shown by a Pearson correlation coefficient of 0.47 (P<0.001). Students clustered with below average ward evaluations (3.0-4.0) were in the 69.5th percentile of NBME scores, whereas students clustered with above average ward evaluations (6.0-7.0) were in the 79.2th percentile (P<0.001).

CONCLUSIONS

A strong positive relationship exists between subjective ward evaluations and NBME performance. These data may afford some confidence to surgical faculty and surgical resident ability to accurately evaluate medical students during clinical clerkships. Understanding factors in student performance may help in improving the surgical clerkship experience.

The definition of polytrauma: variable interrater versus intrarater agreement--a prospective international study among trauma surgeons

BACKGROUND

The international trauma community has recognized the lack of a validated consensus definition of "polytrauma." We hypothesized that using a subjective definition, trauma surgeons will not have substantial agreement; thus, an objective definition is needed.

METHODS

A prospective observational study was conducted between December 2010 and June 2011 (John Hunter Hospital, Level I trauma center). Inclusion criteria were all trauma call patients with subsequent intensive care unit admission. The study was composed of four stages as follows: (1) four trauma surgeons assessed patients until 24 hours, then coded as either "yes" or "no" for polytrauma, and results compared for agreement; (2) eight trauma surgeons representing the United States, Germany, and the Netherlands graded the same prospectively assessed patients and coded as either "yes" or "no" for polytrauma; (3) 12 months later, the original four trauma surgeons repeated assessment via data sheets to test intrarater variability; and (4) individual subjective definitions were compared with three anatomic scores, namely, (a) Injury Severity Score (ISS) of greater than 15, (b) ISS of greater 17, and (c) Abbreviated Injury Scale (AIS) score of greater than 2 in at least two ISS body regions.

RESULTS

A total of 52 trauma patients were included. Results for each stage were as follows: (1) κ score of 0.50, moderate agreement; (2) κ score of 0.41, moderate agreement; (3) Rater 1 had moderate intrarater agreement (κ score, 0.59), while Raters 2, 3, 4 had substantial intrarater agreement (κ scores, 0.75, 0.66, and 0.71, respectively); and (4) none had most agreement with ISS of greater than 15 (κ score, 0.16), while both definitions ISS greater than 17 and Abbreviated Injury Scale (AIS) score of greater than 2 in at least two ISS body regions had on average fair agreement (κ scores, 0.27 and 0.39, respectively).

CONCLUSION

Based on subjective assessments, trauma surgeons do not agree on the definition of polytrauma, with the subjective definition differing both within and across institutions.

Validity of faculty and resident global assessment of medical students' clinical knowledge during their pediatrics clerkship

OBJECTIVE

Medical knowledge is one of six core competencies in medicine. Medical student assessments should be valid and reliable. We assessed the relationship between faculty and resident global assessment of pediatric medical student knowledge and performance on a standardized test in medical knowledge.

METHODS

Retrospective cross-sectional study of medical students on a pediatric clerkship in academic year 2008-2009 at one academic health center. Faculty and residents rated students' clinical knowledge on a 5-point Likert scale. The inter-rater reliability of clinical knowledge ratings was assessed by calculating the intra-class correlation coefficient (ICC) for residents' ratings, faculty ratings, and both rating types combined. Convergent validity between clinical knowledge ratings and scores on the National Board of Medical Examiners (NBME) clinical subject examination in pediatrics was assessed with Pearson product moment correlation correction and the coefficient of the determination.

RESULTS

There was moderate agreement for global clinical knowledge ratings by faculty and moderate agreement for ratings by residents. The agreement was also moderate when faculty and resident ratings were combined. Global ratings of clinical knowledge had high convergent validity with pediatric examination scores when students were rated by both residents and faculty.

CONCLUSIONS

Our findings provide evidence for convergent validity of global assessment of medical students' clinical knowledge with NBME subject examination scores in pediatrics.

Using elements from an acute abdominal pain Objective Structured Clinical Examination (OSCE) leads to more standardized grading in the surgical clerkship for third-year medical students

BACKGROUND

There is poor reliability in the Likert-based assessments of patient interaction and general knowledge base for medical students in the surgical clerkship. The Objective Structured Clinical Examination (OSCE) can be used to assess these competencies.

OBJECTIVE

We hypothesize that using OSCE performance to replace the current Likert-based patient interaction and general knowledge base assessments will not affect the pass/fail rate for third-year medical students in the surgical clerkship.

METHODS

In this retrospective study, third-year medical student clerkship data from a three-station acute abdominal pain OSCE were collected from the 2009-2010 academic year. New patient interaction and general knowledge base assessments were derived from the performance data and substituted for original assessments to generate new clerkship scores and ordinal grades. Two-sided nonparametric statistics were used for comparative analyses, using an α = 0.05.

RESULTS

Seventy third-year medical students (50.0% female) were evaluated. A sign test showed a difference in the original (4.45/5) and the new (4.20/5) median patient interaction scores (p < 0.01). A sign test did not show a difference in the original (4.00/5) and the new (4.11/5) median general knowledge base scores (p = 0.28). Nine clerkship grades changed between these different grading schemes (p = 0.045), with an overall agreement of 87.1% and a kappa statistic of 0.81. There were no differences in the pass/fail rate (p > 0.99).

CONCLUSIONS

We conclude that there are no differences in pass/fail rate, but there is a more standardized distribution of patient interaction assessments and utilization of the full spectrum of possible passing grades. We recommend that the current patient interaction assessment for third-year medical students in the surgical clerkship be replaced with that found through trained standardized patients in this three-station acute abdominal pain OSCE.