Are We Measuring What Matters? How Student and Clerkship Characteristics Influence Clinical Grading

ChatGPT Interactive Medical Simulations for Early Clinical Education: Case Study

BACKGROUND

The transition to clinical clerkships can be difficult for medical students, as it requires the synthesis and application of preclinical information into diagnostic and therapeutic decisions. ChatGPT-a generative language model with many medical applications due to its creativity, memory, and accuracy-can help students in this transition.

OBJECTIVE

This paper models ChatGPT 3.5's ability to perform interactive clinical simulations and shows this tool's benefit to medical education.

METHODS

Simulation starting prompts were refined using ChatGPT 3.5 in Google Chrome. Starting prompts were selected based on assessment format, stepwise progression of simulation events and questions, free-response question type, responsiveness to user inputs, postscenario feedback, and medical accuracy of the feedback. The chosen scenarios were advanced cardiac life support and medical intensive care (for sepsis and pneumonia).

RESULTS

Two starting prompts were chosen. Prompt 1 was developed through 3 test simulations and used successfully in 2 simulations. Prompt 2 was developed through 10 additional test simulations and used successfully in 1 simulation.

CONCLUSIONS

ChatGPT is capable of creating simulations for early clinical education. These simulations let students practice novel parts of the clinical curriculum, such as forming independent diagnostic and therapeutic impressions over an entire patient encounter. Furthermore, the simulations can adapt to user inputs in a way that replicates real life more accurately than premade question bank clinical vignettes. Finally, ChatGPT can create potentially unlimited free simulations with specific feedback, which increases access for medical students with lower socioeconomic status and underresourced medical schools. However, no tool is perfect, and ChatGPT is no exception; there are concerns about simulation accuracy and replicability that need to be addressed to further optimize ChatGPT's performance as an educational resource.

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail

Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.

Beyond Competency: A Student Perspective on Growth Through Clerkship Feedback

This article proposes a paradigm shift from the competency-based model of clerkship feedback using checklists to a coaching-based, action plan-oriented process that centers on individualized student-oriented goals. Using a student perspective, the authors examine the feedback literature and put forward a proposal to use an impact model whose emphasis is to improve the learning climate for students. Several techniques are reviewed which include goal generation and creation of dynamic action plans. By intentionally focusing on coaching relationships as a platform for feedback, the learners and mentors share goals and the result of feedback becomes action-based behaviors which may help negate personal attribution and bias in the feedback process.

Racial/Ethnic Disparities in Clerkship Grading: Perspectives of Students and Teachers

PURPOSE

Racial/ethnic disparities exist in clinical clerkship grading, yet little is known about medical student and faculty perspectives on why these disparities occur. This study explored what happens during clerkships that might explain grading disparities.

METHOD

Medical students and clerkship teachers at 3 U.S. medical schools completed a demographic survey and semistructured interview. The constant comparative method was used to analyze transcripts by inductively developing codes; grouping codes in categories; and refining codes, descriptions, and group assignments to identify themes. Interpretations of and relationships among themes were iteratively discussed to develop a grounded theory.

RESULTS

Fifty-nine participants (29 medical students, 30 teachers [28 clinical faculty, 2 residents]) were interviewed in 2020. The Social Milieu of Medical Education (relationships, fit, opportunities, and judgments in the clinical-learning setting) was the organizing theme, influenced by 5 additional themes: Societal Influence (experiences in society), Students' Characteristics and Background (personal characteristics and experiences outside medical school), Assessment Processes (collection of student performance data and how data inform grades), Learning Environment (resources available and messaging within the clinical setting), and Students' Interactions and Reactions (interactions with and reactions to peers and teachers). The grounded theory highlights complex, multilayered aspects of how the social milieu of medical education is shaped by and shapes students' experiences, relationships, and clerkship assessments and promotes clerkship-grading disparities.

CONCLUSIONS

Mitigating clerkship-grading disparities will require intervening on interrelated, contextual factors to provide equitable opportunities for students from diverse backgrounds and with varying styles of engagement in clinical-learning settings, along with attending to modifying assessment processes.

Amplifying the Student Voice: Medical Student Perceptions of AΩA

BACKGROUND

Recent literature has suggested racial disparities in Alpha Omega Alpha Honor Medical Society (AΩA) selection and raised concerns about its effects on the learning environment. Internal reviews at multiple institutions have led to changes in selection practices or suspension of student chapters; in October 2020, the national AΩA organization provided guidance to address these concerns.

OBJECTIVE

This study aimed to better understand student opinions of AΩA.

DESIGN

An anonymous survey using both multiple response option and free response questions.

PARTICIPANTS

Medical students at the Perelman School of Medicine at the University of Pennsylvania.

MAIN MEASURES

Descriptive statistics and logistic regressions were used to examine predictors of student opinion towards AΩA. Free responses were analyzed by two independent coders to identify key themes.

KEY RESULTS

In total, 70% of the student body (n = 547) completed the survey. Sixty-three percent had a negative opinion of AΩA, and 57% felt AΩA should not exist at the student level. Thirteen percent believed AΩA membership appropriately reflects the student body; 8% thought selection processes were fair. On multivariate analysis, negative predictors of a student's preference to continue AΩA at the student level included belief that AΩA membership does not currently mirror class composition (OR: 0.45, [95% CI: 0.23-0.89]) and that AΩA selection processes were unfair (OR: 0.20 [0.08-0.47]). Self-perception as not competitive for AΩA selection was also a negative predictor (OR: 0.44 [0.22-0.88]). Major qualitative themes included equity, impact on the learning environment, transparency, and positive aspects of AΩA.

CONCLUSIONS

This single-institution survey demonstrated significant student concerns regarding AΩA selection fairness and effects on the learning environment. Many critiques extended beyond AΩA itself, instead focusing on the perceived magnification of existing disparities in the learning environment. As the national conversation about AΩA continues, engaging student voices in the discussion is critical.

Role of faculty characteristics in failing to fail in clinical clerkships

INTRODUCTION

In the context of competency-based medical education, poor student performance must be accurately documented to allow learners to improve and to protect the public. However, faculty may be reluctant to provide evaluations that could be perceived as negative, and clerkship directors report that some students pass who should have failed. Student perception of faculty may be considered in faculty promotion, teaching awards, and leadership positions. Therefore, faculty of lower academic rank may perceive themselves to be more vulnerable and, therefore, be less likely to document poor student performance. This study investigated faculty characteristics associated with low performance evaluations (LPEs).

METHOD

The authors analysed individual faculty evaluations of medical students who completed the third-year clerkships over 15 years using a generalised mixed regression model to assess the association of evaluator academic rank with likelihood of an LPE. Other available factors related to experience or academic vulnerability were incorporated including faculty age, race, ethnicity, and gender.

RESULTS

The authors identified 50 120 evaluations by 585 faculty on 3447 students between January 2007 and April 2021. Faculty were more likely to give LPEs at the midpoint (4.9%), compared with the final (1.6%), evaluation (odds ratio [OR] = 4.004, 95% confidence interval [CI] [3.59, 4.53]; p < 0.001). The likelihood of LPE decreased significantly during the 15-year study period (OR = 0.94 [0.90, 0.97]; p < 0.01). Full professors were significantly more likely to give an LPE than assistant professors (OR = 1.62 [1.08, 2.43]; p = 0.02). Women were more likely to give LPEs than men (OR = 1.88 [1.37, 2.58]; p 0.01). Other faculty characteristics including race and experience were not associated with LPE.

CONCLUSIONS

The number of LPEs decreased over time, and senior faculty were more likely to document poor medical student performance compared with assistant professors.

New Heuristics to Stratify Applicants: Predictors of General Surgery Residency Applicant Step 1 Scores

OBJECTIVE

In 2022, United States Medical Licensing Examination (USMLE) Step 1 scores will become pass/fail. This may be problematic, as residency programs heavily rely on USMLE Step 1 scores as a metric when determining interview invitations. This study aimed to assess candidate application metrics associated with USMLE Step 1 scores to offer programs new cues for stratifying applicants.

DESIGN

Retrospective cohort study analyzing interviewed applicants to one general surgery residency program in 2019 and 2020. Applicant data analyzed included USMLE Step 1 scores, number of publications, clerkship scores, letter of recommendation scores (out of 2, scored by 0.25 interval), interview overall score (out of 5, scored by integer level), and standardized question score (out of 10). Each year, applicant's answers to one standardized behavioral question during their interview were scored by interviewers.

SETTING

Tertiary medical center, academic general surgery residency program.

PARTICIPANTS

Interviewed applicants at one general surgery residency program whose applications contained complete demographic information (203 out of 247).

RESULTS

Multiple Linear Regression revealed that higher surgical clerkship (β = 0.19, p = 0.006) and higher standardized interview question (β = 0.32, p < 0.001) scores were positively associated with applicant USMLE Step 1 score (F[7, 195] = 6.61, p < 0.001, R2 = 0.19). Letter of recommendation score, number of peer reviewed publications, gender, race, and applicant type (preliminary/categorical) were not associated with USMLE Step 1 scores.

CONCLUSIONS

With USMLE Step 1 scores transitioning to pass/fail, surgical residency programs need new selection heuristics. Surgery clerkship scores and standardized behavioral questions answered by applicants prior to the interview could provide a holistic view of applicants and help programs better stratify candidates without USMLE Step 1 scores.

Gender Disparity in Evaluation of Internal Medicine Clerkship Performance

IMPORTANCE

Women studying medicine currently equal men in number, but evidence suggests that men and women might not be evaluated equally throughout their education.

OBJECTIVE

To examine whether there are differences associated with gender in either objective or subjective evaluations of medical students in an internal medicine clerkship.

DESIGN, SETTING, AND PARTICIPANTS

This single-center retrospective cohort study evaluated data from 277 third-year medical students completing internal medicine clerkships in the 2017 to 2018 academic year at an academic hospital and its affiliates in Pennsylvania. Data were analyzed from September to November 2020.

EXPOSURE

Gender, presumed based on pronouns used in evaluations.

MAIN OUTCOMES AND MEASURES

Likert scale evaluations of clinical skills, standardized examination scores, and written evaluations were analyzed. Univariate and multivariate linear regression were used to observe trends in measures. Word embeddings were analyzed for narrative evaluations.

RESULTS

Analyses of 277 third-year medical students completing an internal medicine clerkship (140 women [51%] with a mean [SD] age of 25.5 [2.3] years and 137 [49%] presumed men with a mean [SD] age of 25.9 [2.7] years) detected no difference in final grade distribution. However, women outperformed men in 5 of 8 domains of clinical performance, including patient interaction (difference, 0.07 [95% CI, 0.04-0.13]), growth mindset (difference, 0.08 [95% CI, 0.01-0.11]), communication (difference, 0.05 [95% CI, 0-0.12]), compassion (difference, 0.125 [95% CI, 0.03-0.11]), and professionalism (difference, 0.07 [95% CI, 0-0.11]). With no difference in examination scores or subjective knowledge evaluation, there was a positive correlation between these variables for both genders (women: r = 0.35; men: r = 0.26) but different elevations for the line of best fit (P < .001). Multivariate regression analyses revealed associations between final grade and patient interaction (women: coefficient, 6.64 [95% CI, 2.16-11.12]; P = .004; men: coefficient, 7.11 [95% CI, 2.94-11.28]; P < .001), subjective knowledge evaluation (women: coefficient, 6.66 [95% CI, 3.87-9.45]; P < .001; men: coefficient, 5.45 [95% CI, 2.43-8.43]; P < .001), reported time spent with the student (women: coefficient, 5.35 [95% CI, 2.62-8.08]; P < .001; men: coefficient, 3.65 [95% CI, 0.83-6.47]; P = .01), and communication (women: coefficient, 6.32 [95% CI, 3.12-9.51]; P < .001; men: coefficient, 4.21 [95% CI, 0.92-7.49]; P = .01). The model based on the men's data also included growth mindset as a significant variable (coefficient, 4.09 [95% CI, 0.67-7.50]; P = .02). For narrative evaluations, words in context with "he or him" and "she or her" differed, with agentic terms used in descriptions of men and personality descriptors used more often for women.

CONCLUSIONS AND RELEVANCE

Despite no difference in final grade, women scored higher than men on various domains of clinical performance, and performance in these domains was associated with evaluators' suggested final grade. The content of narrative evaluations significantly differed by student gender. This work supports the hypothesis that how students are evaluated in clinical clerkships is associated with gender.