1
|
Richards SM, Nadeau NJ. The impact of surgery clerkship grades on applicant competitiveness for orthopaedic surgery residency match. Surg Open Sci 2024; 20:55-56. [PMID: 38911058 PMCID: PMC11190740 DOI: 10.1016/j.sopen.2024.05.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2024] [Accepted: 05/24/2024] [Indexed: 06/25/2024] Open
Abstract
Medical school clerkship grades are an important method for applicants to distinguish themselves when applying to residency programs. Given the lack of standardization among medical schools in the clerkship grading process, it has become more challenging for orthopaedic surgery residencies to ascertain the true value of surgery clerkship grades between applicants. This letter to the editor is a response to the article by Hoy et al., "Analysis of variability and trends in medical school clerkship grades," and offers further perspectives on the variability of surgery clerkship grading and its effect on applicants.
Collapse
Affiliation(s)
- Sean M. Richards
- Hackensack Meridian School of Medicine, Nutley, NJ, 07110, United States of America
| | - Nicolas J. Nadeau
- Hackensack Meridian School of Medicine, Nutley, NJ, 07110, United States of America
| |
Collapse
|
2
|
Gauer JL, Mustapha T, Violato C. Race and Gender Bias in Clerkship Grading. TEACHING AND LEARNING IN MEDICINE 2024; 36:304-311. [PMID: 37334710 DOI: 10.1080/10401334.2023.2224789] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Accepted: 05/11/2023] [Indexed: 06/20/2023]
Abstract
Phenomenon: Existing literature, as well as anecdotal evidence, suggests that tiered clinical grading systems may display systematic demographic biases. This study aimed to investigate these potential inequities in-depth. Specifically, this study attempted to address the following gaps in the literature: (1) studying grades actually assigned to students (as opposed to self-reported ones), (2) using longitudinal data over an 8-year period, providing stability of data, (3) analyzing three important, potentially confounding covariates, (4) using a comprehensive multivariate statistical design, and (5) investigating not just the main effects of gender and race, but also their potential interaction. Approach: Participants included 1,905 graduates (985 women, 51.7%) who received the Doctor of Medicine degree between 2014 and 2021. Most of the participants were white (n = 1,310, 68.8%) and about one-fifth were nonwhite (n = 397, 20.8%). There were no reported race data for 10.4% (n = 198). To explore potential differential grading, a two-way multivariate analysis of covariance was employed to examine the impact of race and gender on grades in eight required clerkships, adjusting for prior academic performance. Findings: There were two significant main effects, race and gender, but no interaction effect between gender and race. Women received higher grades on average on all eight clerkships, and white students received higher grades on average on four of the eight clerkships (Medicine, Pediatrics, Surgery, Obstetrics/Gynecology). These relationships held even when accounting for prior performance covariates. Insights: These findings provide additional evidence that tiered grading systems may be subject to systematic demographic biases. It is difficult to tease apart the contributions of various factors to the observed differences in gender and race on clerkship grades, and the interactions that produce these biases may be quite complex. The simplest solution to cut through the tangled web of grading biases may be to move away from a tiered grading system altogether.
Collapse
Affiliation(s)
- Jacqueline L Gauer
- Office of Medical Education, University of Minnesota, Minneapolis, Minnesota, USA
| | - Taj Mustapha
- Departments of Internal Medicine and Pediatrics, Office of Diversity, Equity, and Inclusion, University of Minnesota Medical School, Minneapolis, Minnesota, USA
| | - Claudio Violato
- Department of Medicine, Office of Assessment and Evaluation, University of Minnesota Medical School, Minneapolis, Minnesota, USA
| |
Collapse
|
3
|
Hoy JF, Shuman SL, Smith SR, Kogan M, Simcock XC. Analysis of variability and trends in medical school clerkship grades. Surg Open Sci 2024; 19:80-86. [PMID: 38595833 PMCID: PMC11002300 DOI: 10.1016/j.sopen.2024.03.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2024] [Accepted: 03/22/2024] [Indexed: 04/11/2024] Open
Abstract
Background Medical school clerkship grades are used to evaluate orthopedic surgery residency applicants, however, high interinstitutional variability in grade distribution calls into question the utility of clerkship grades when evaluating applicants from different medical schools. This study aims to evaluate the variability in grade distribution among medical schools and look for trends in grade distribution over recent years. Methods Applications submitted to Rush University's orthopedic surgery residency program from 2015, 2019, and 2022 were collected from the Electronic Residency Application Service. Applications from the top 100 schools according to the 2023-2024 U.S. News and World Report Research Rankings were reviewed. The percentage of "honors" grades awarded by medical schools for the surgery and internal medicine clerkships were extracted from applicants' Medical Student Performance Evaluation letters. Results The median percentage of honors given in 2022 was 36.0 % (range 10.0-82.0) for the surgery clerkship and 33.0 % (range 6.7-80.0) for the internal medicine clerkship. Honors were given 6.6 % more in the surgery clerkship in 2022 compared to 2015. There was a negative correlation between a higher (worse) U.S. News and World Report research ranking and the percentage of honors awarded in 2022 for the surgery and internal medicine clerkships. Conclusion There is substantial interinstitutional variability in the rate that medical schools award an "honors" grade with evidence of grade inflation in the surgery clerkship. Residency programs using clerkship grades to compare applicants should do so cautiously provided the variability demonstrated in this study.
Collapse
Affiliation(s)
- John F. Hoy
- Rush University Medical Center, Chicago, IL, USA
| | | | | | - Monica Kogan
- Rush University Medical Center, Chicago, IL, USA
| | | |
Collapse
|
4
|
Caretta-Weyer HA, Eva KW, Schumacher DJ, Yarris LM, Teunissen PW. Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:S98-S107. [PMID: 37983402 DOI: 10.1097/acm.0000000000005365] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/11/2023]
Abstract
PURPOSE The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda. METHOD Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion. RESULTS Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process. CONCLUSIONS This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.
Collapse
Affiliation(s)
- Holly A Caretta-Weyer
- H.A. Caretta-Weyer is associate professor and associate residency director, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| | - Kevin W Eva
- K.W. Eva is associate director and senior scientist, Centre for Health Education Scholarship, and professor and director, educational research and scholarship, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada; ORCID: http://orcid.org/0000-0002-8672-2500
| | - Daniel J Schumacher
- D.J. Schumacher is professor of pediatrics, Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: http://orcid.org/0000-0001-5507-8452
| | - Lalena M Yarris
- L.M. Yarris is professor and vice chair of faculty development, Department of Emergency Medicine, Oregon Health & Science University, Portland, Oregon
| | - Pim W Teunissen
- P.W. Teunissen is professor of workplace learning in health care, School of Health Professions Education, and gynecologist, Department of Obstetrics & Gynecology, Maastricht University and Maastricht University Medical Center, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-0930-0048
| |
Collapse
|
5
|
Huynh A, Nguyen A, Beyer RS, Harris MH, Hatter MJ, Brown NJ, de Virgilio C, Nahmias J. Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:769-774. [PMID: 36780667 DOI: 10.1097/acm.0000000000005168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.
Collapse
Affiliation(s)
- Ashley Huynh
- A. Huynh is a first-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-4413-6829
| | - Andrew Nguyen
- A. Nguyen is a first-year medical student, University of Florida College of Medicine, Gainesville, Florida; ORCID: https://orcid.org/0000-0002-8131-150X
| | - Ryan S Beyer
- R.S. Beyer is a second-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-0283-3749
| | - Mark H Harris
- M.H. Harris is a second-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-1598-225X
| | - Matthew J Hatter
- M.J. Hatter is a second-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0003-2922-6196
| | - Nolan J Brown
- N.J. Brown is a fourth-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-6025-346X
| | - Christian de Virgilio
- C. de Virgilio is professor of surgery, Harbor-UCLA Medical Center, Torrance, California
| | - Jeffry Nahmias
- J. Nahmias is professor of trauma, burns, surgical critical care, and acute care surgery, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0003-0094-571X
| |
Collapse
|
6
|
O'Sullivan L, Kagabo W, Prasad N, Laporte D, Aiyer A. Racial and Ethnic Bias in Medical School Clinical Grading: A Review. JOURNAL OF SURGICAL EDUCATION 2023; 80:806-816. [PMID: 37019709 DOI: 10.1016/j.jsurg.2023.03.004] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Revised: 02/26/2023] [Accepted: 03/17/2023] [Indexed: 05/23/2023]
Abstract
OBJECTIVE Improving diversity in healthcare is a widely recognized national goal. The diversity of medical student matriculants has increased, yet this trend is not seen in the composition of competitive residency programs. In this review, we examine racial and ethnic disparities in medical student grading during clinical years and explore the consequences of how this may exclude minority students from accessing competitive residency positions. DESIGN Following PRISMA guidelines, we searched PubMed, Embase, Scopus, and ERIC databases using variations of the terms "race," "ethnicity," "clerkship," "rotation," "grade," "evaluation", or "shelf exam." Of 391 references found using the criteria, 29 were related to clinical grading and race/ethnicity and included in the review. The GRADE criteria were used to determine the quality of evidence. SETTING Johns Hopkins School of Medicine, Baltimore MD. RESULTS Five studies examining a total of 107,687 students from up to 113 different schools found racial minority students receive significantly fewer Honors grades in core clerkships compared to White students. Three studies examining 94,814 medical student evaluations from up to 130 different schools found significant disparities in the wording of written clerkship evaluations based on race and/or ethnicity. CONCLUSIONS A large body of evidence suggests the presence of racial bias in subjective clinical grading and written clerkship evaluations of medical students. Grading disparities can disadvantage minority students when applying to competitive residency programs and may contribute to a lack of diversity in these fields. As low minority representation has a negative impact on patient care and research advancement, strategies to resolve this issue must be further explored.
Collapse
Affiliation(s)
- Lucy O'Sullivan
- Johns Hopkins Orthopedics, Johns Hopkins University School of Medicine, Baltimore, Maryland.
| | - Whitney Kagabo
- Department of Orthopedics, Johns Hopkins Orthopedics, Johns Hopkins Medicine, Baltimore, Maryland
| | - Niyathi Prasad
- Department of Orthopedics, Johns Hopkins Orthopedics, Johns Hopkins Medicine, Baltimore, Maryland
| | - Dawn Laporte
- Department of Orthopedics, Johns Hopkins Orthopedics, Johns Hopkins Medicine, Baltimore, Maryland
| | - Amiethab Aiyer
- Department of Orthopedics, Johns Hopkins Orthopedics, Johns Hopkins Medicine, Baltimore, Maryland
| |
Collapse
|
7
|
Variation in core clerkship grading reported on the Medical Student Performance Evaluation (MSPE) for orthopaedic surgery applicants: a retrospective review. CURRENT ORTHOPAEDIC PRACTICE 2022. [DOI: 10.1097/bco.0000000000001152] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
8
|
Swails JL, Gadgil MA, Goodrum H, Gupta R, Rahbar MH, Bernstam EV. Role of faculty characteristics in failing to fail in clinical clerkships. MEDICAL EDUCATION 2022; 56:634-640. [PMID: 34983083 DOI: 10.1111/medu.14725] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Revised: 12/21/2021] [Accepted: 12/27/2021] [Indexed: 06/14/2023]
Abstract
INTRODUCTION In the context of competency-based medical education, poor student performance must be accurately documented to allow learners to improve and to protect the public. However, faculty may be reluctant to provide evaluations that could be perceived as negative, and clerkship directors report that some students pass who should have failed. Student perception of faculty may be considered in faculty promotion, teaching awards, and leadership positions. Therefore, faculty of lower academic rank may perceive themselves to be more vulnerable and, therefore, be less likely to document poor student performance. This study investigated faculty characteristics associated with low performance evaluations (LPEs). METHOD The authors analysed individual faculty evaluations of medical students who completed the third-year clerkships over 15 years using a generalised mixed regression model to assess the association of evaluator academic rank with likelihood of an LPE. Other available factors related to experience or academic vulnerability were incorporated including faculty age, race, ethnicity, and gender. RESULTS The authors identified 50 120 evaluations by 585 faculty on 3447 students between January 2007 and April 2021. Faculty were more likely to give LPEs at the midpoint (4.9%), compared with the final (1.6%), evaluation (odds ratio [OR] = 4.004, 95% confidence interval [CI] [3.59, 4.53]; p < 0.001). The likelihood of LPE decreased significantly during the 15-year study period (OR = 0.94 [0.90, 0.97]; p < 0.01). Full professors were significantly more likely to give an LPE than assistant professors (OR = 1.62 [1.08, 2.43]; p = 0.02). Women were more likely to give LPEs than men (OR = 1.88 [1.37, 2.58]; p 0.01). Other faculty characteristics including race and experience were not associated with LPE. CONCLUSIONS The number of LPEs decreased over time, and senior faculty were more likely to document poor medical student performance compared with assistant professors.
Collapse
Affiliation(s)
- Jennifer L Swails
- Department of Internal Medicine, Mc Govern Medical School, University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Meghana A Gadgil
- Division of Hospital Medicine, San Francisco General Hospital, San Francisco, California, USA
- Division of Health Policy and Management, School of Public Health, University of California, Berkeley, Berkeley, California, USA
| | - Heath Goodrum
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Resmi Gupta
- Division of Clinical and Translational Sciences, Department of Internal Medicine, McGovern Medical School, Houston, Texas, USA
| | - Mohammad H Rahbar
- Division of Clinical and Translational Sciences, Department of Internal Medicine, McGovern Medical School, Houston, Texas, USA
- Department of Epidemiology, Human Genetics, and Environmental Sciences, School of Public Health, The University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Elmer V Bernstam
- Department of Internal Medicine, Mc Govern Medical School, University of Texas Health Science Center at Houston, Houston, Texas, USA
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| |
Collapse
|
9
|
Wu JC, Tang KP, Hsu YHE, Yang YT, Chu JS, Lin YK, Hou WH. Medical undergraduates' self-evaluation: before and after curriculum reform. BMC MEDICAL EDUCATION 2022; 22:296. [PMID: 35443681 PMCID: PMC9019532 DOI: 10.1186/s12909-022-03330-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/04/2022] [Accepted: 03/25/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND In 2013, Taiwan launched a curriculum reform-the 7-year undergraduate medical education program was shortened to 6 years. This study explored the evaluation results from students regarding the curriculum reform and investigated graduates' perceptions regarding the curriculum organization of the two academic training programs affected by this curricular reform. METHODS A cross-sectional survey was conducted from May 14 to June 12, 2019. The 315 graduates from both the 7-year and 6-year curriculum programs in the same medical school in Taipei were invited to participate in this study. In total, 197 completed questionnaires were received, representing a response rate of 62.5%. The results of the principal component analysis confirmed the validity of the constructs employed in this self-administered questionnaire. RESULTS The t-test results yielded two main findings. First, the graduates from the 6-year program had significantly lower scores for preparedness for the upcoming postgraduate-year residency training than did their 7-year program counterparts. Additionally, the male graduates had significantly higher scores in terms of perceptions regarding curriculum organization and preparedness for postgraduate-year residency training than the female graduates. The results of stepwise regression also indicated that the sex difference was significantly correlated with graduates' readiness for their postgraduate-year residency training. CONCLUSION To avoid sex disparities in career development, a further investigation of female medical students' learning environment and conditions is necessary. In addition to the cross-sectional study of students' perceptions, further repeated measurements of the objective academic or clinical performance of graduates in clinical settings are desirable.
Collapse
Affiliation(s)
- Jeng-Cheng Wu
- Department of Urology, Taipei Medical University Hospital, Taipei, Taiwan
- Department of Urology, School of Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan
- Department of Health Promotion and Health Education, College of Education, National Taiwan Normal University, Taipei, Taiwan
| | - Kung-Pei Tang
- National Taipei University of Education, Taipei, Taiwan
| | - Yi-Hsin Elsa Hsu
- International Ph.D. Program in Biotech and Healthcare Management, Taipei Medical University, Taipei, Taiwan
| | - Ya-Ting Yang
- Center for General Education, Taipei Medical University, Taipei, Taiwan
| | - Jan-Show Chu
- School of Medicine, College of Medicine, Taipei Medical University, 250 Wu-Hsing Street, Taipei, 11031, Taiwan.
| | - Yen-Kuang Lin
- Graduate Institute of Athletics and Coaching Science, National Taiwan Sport University, Taoyuan, Taiwan.
- Biostatistics Research Center, Taipei Medical University, Taipei, Taiwan.
| | - Wen-Hsuan Hou
- Department of Physical Medicine and Rehabilitation, Taipei Medical University Hospital, Taipei, Taiwan.
- Department of Geriatrics and Gerontology, Taipei Medical University Hospital, Taipei, Taiwan.
- School of Gerontology and Long-Term Care, College of Nursing, Taipei Medical University, Taipei, Taiwan.
- Graduate Institute of Clinical Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan.
| |
Collapse
|
10
|
Bullock JL, Seligman L, Lai CJ, O'Sullivan PS, Hauer KE. Moving toward Mastery: Changes in Student Perceptions of Clerkship Assessment with Pass/Fail Grading and Enhanced Feedback. TEACHING AND LEARNING IN MEDICINE 2022; 34:198-208. [PMID: 34014793 DOI: 10.1080/10401334.2021.1922285] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/17/2020] [Revised: 03/08/2021] [Accepted: 03/20/2021] [Indexed: 06/12/2023]
Abstract
ProblemClerkship grades contribute to a summative assessment culture in clerkships and can therefore interfere with students' learning. For example, by focusing on summative, tiered clerkship grades, students often discount accompanying feedback that could inform future learning. This case report seeks to explore whether an assessment system intervention which eliminated tiered grades and enhanced feedback was associated with changes in student perceptions of clerkship assessment and perceptions of the clinical learning environment. Intervention: In January 2019, our institution eliminated tiered clerkship grading (honors/pass/fail) for medical students during the core clerkship year and implemented pass/fail clerkship grading along with required twice weekly, work-based assessments for formative feedback. Context: In this single institution, cross-sectional survey study, we collected data from fourth-year medical students one year after an assessment system intervention. The intervention entailed changing from honors/pass/fail to pass/fail grading in all eight core clerkships and implementing an electronic system to record twice-weekly real-time formative work-based assessments. The survey queried student perceptions on the fairness and accuracy of grading and the clinical learning environment-including whether clerkships were mastery- or performance-oriented. We compared responses from students one year after the assessment intervention to those from the class one year before the intervention. Comparisons were made using unpaired, two-tailed t-tests or chi-squared tests as appropriate with Cohen's d for effect size estimation for score differences. Content analysis was used to analyze responses from two open-ended questions about feedback and grading. Impact: Survey response rates were similar before and after intervention (76% (127/168) vs. 72% (118/163), respectively) with no between-group differences in demographics. The after-intervention group showed statistically significant increases in the following factors: "grades are transparent and fair" (Cohen's d = 0.80), "students receive useful feedback" (d = 0.51), and "resident evaluation procedures are fair" (d = 0.40). After-intervention respondents perceived the clerkship learning environment to be more mastery-oriented (d = 0.52), less performance approach-oriented (d = 0.63), and less performance avoid-oriented (d = 0.49). There were no statistical differences in the factors "attending evaluation procedures are fair," "evaluations are accurate," "evaluations are biased," or "perception of stereotype threat." Open-ended questions revealed student recommendations to improve clerkship summary narratives, burden of work-based assessment, and in-person feedback. Lessons Learned: After an assessment system change to pass/fail grading with work-based assessments, we observed moderate to large improvements in student perceptions of clerkship grading and the mastery orientation of the learning environment. Our intervention did not improve perceptions around bias in assessment in clerkships. Other medical schools may consider similar interventions to begin to address student concerns with clerkship assessment and promote a more adaptive learning environment.
Collapse
Affiliation(s)
- Justin L Bullock
- Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California, USA
| | - Lee Seligman
- Department of Medicine, Columbia University Irving Medical Center, New York-Presbyterian Hospital, New York, New York, USA
| | - Cindy J Lai
- Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California, USA
| | - Patricia S O'Sullivan
- Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California, USA
- Department of Surgery, University of California, San Francisco School of Medicine, San Francisco, California, USA
| | - Karen E Hauer
- Department of Medicine, University of California, San Francisco School of Medicine, San Francisco, California, USA
| |
Collapse
|
11
|
Lund S, D'Angelo J, D'Angelo AL, Heller S, Stulak J, Rivera M. New Heuristics to Stratify Applicants: Predictors of General Surgery Residency Applicant Step 1 Scores. JOURNAL OF SURGICAL EDUCATION 2022; 79:349-354. [PMID: 34776371 DOI: 10.1016/j.jsurg.2021.10.007] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/19/2021] [Revised: 10/04/2021] [Accepted: 10/20/2021] [Indexed: 06/13/2023]
Abstract
OBJECTIVE In 2022, United States Medical Licensing Examination (USMLE) Step 1 scores will become pass/fail. This may be problematic, as residency programs heavily rely on USMLE Step 1 scores as a metric when determining interview invitations. This study aimed to assess candidate application metrics associated with USMLE Step 1 scores to offer programs new cues for stratifying applicants. DESIGN Retrospective cohort study analyzing interviewed applicants to one general surgery residency program in 2019 and 2020. Applicant data analyzed included USMLE Step 1 scores, number of publications, clerkship scores, letter of recommendation scores (out of 2, scored by 0.25 interval), interview overall score (out of 5, scored by integer level), and standardized question score (out of 10). Each year, applicant's answers to one standardized behavioral question during their interview were scored by interviewers. SETTING Tertiary medical center, academic general surgery residency program. PARTICIPANTS Interviewed applicants at one general surgery residency program whose applications contained complete demographic information (203 out of 247). RESULTS Multiple Linear Regression revealed that higher surgical clerkship (β = 0.19, p = 0.006) and higher standardized interview question (β = 0.32, p < 0.001) scores were positively associated with applicant USMLE Step 1 score (F[7, 195] = 6.61, p < 0.001, R2 = 0.19). Letter of recommendation score, number of peer reviewed publications, gender, race, and applicant type (preliminary/categorical) were not associated with USMLE Step 1 scores. CONCLUSIONS With USMLE Step 1 scores transitioning to pass/fail, surgical residency programs need new selection heuristics. Surgery clerkship scores and standardized behavioral questions answered by applicants prior to the interview could provide a holistic view of applicants and help programs better stratify candidates without USMLE Step 1 scores.
Collapse
Affiliation(s)
- Sarah Lund
- Mayo Clinic Department of Surgery, Rochester, Minnesota.
| | | | | | - Stephanie Heller
- Mayo Clinic Division of Trauma, Critical Care, and General Surgery, Rochester, Minnesota
| | - John Stulak
- Mayo Clinic Department of Cardiovascular Surgery, Rochester, Minnesota
| | - Mariela Rivera
- Mayo Clinic Division of Trauma, Critical Care, and General Surgery, Rochester, Minnesota
| |
Collapse
|
12
|
Maldjian PD, Trivedi UK. Does Objective Scoring of Applications for Radiology Residency Affect Diversity? Acad Radiol 2021; 29:1417-1424. [PMID: 34865953 DOI: 10.1016/j.acra.2021.11.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2021] [Revised: 11/03/2021] [Accepted: 11/05/2021] [Indexed: 11/01/2022]
Abstract
RATIONALE AND OBJECTIVES It is important to recognize if reliance on certain factors in applications affects selection of trainees from under-represented groups. Our purpose is to determine if objective scoring of radiology residency applications based on quantifiable data regarding academic performance, United States Medical Licensing Examination (USMLE) scores and research productivity affects selection of female and under-represented minority (URM) candidates. MATERIALS AND METHODS We reviewed 502 applications from three successive match cycles from United States allopathic medical students. Each application was scored for academic performance, USMLE results and research productivity determining an overall score. The scores of males were compared to females and URM were compared to non-URM candidates. USMLE cutoff scores were evaluated for disparate effects. RESULTS There were 348 male, 154 female, 73 URM and 429 non-URM candidates. For male versus female applicants, there was no significant difference in mean academic performance, USMLE Step 2 Clinical Knowledge and research productivity scores. Males had higher mean USMLE Step 1 (p = 0.005) and overall candidate scores (p = 0.02). Between URM and non-URM candidates there was no significant difference in academic performance. Non-URM applicants had higher mean USMLE Step 1 (p = 0.008), USMLE Step 2 (p = 0.002), research productivity (p = 0.001) and overall scores (p = 0.02). Use of USMLE cutoff scores demonstrated disparate effects on female and URM candidates. CONCLUSION Objective scoring of applications and use of USMLE cutoff scores can disadvantage candidates from underrepresented groups. Screening filters can affect the diversity of candidate pools for radiology residency.
Collapse
|
13
|
Bird JB, Friedman KA, Arayssi T, Olvet DM, Conigliaro RL, Brenner JM. Review of the Medical Student Performance Evaluation: analysis of the end-users' perspective across the specialties. MEDICAL EDUCATION ONLINE 2021; 26:1876315. [PMID: 33606615 PMCID: PMC7899642 DOI: 10.1080/10872981.2021.1876315] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/14/2020] [Revised: 12/23/2020] [Accepted: 01/11/2021] [Indexed: 06/12/2023]
Abstract
The Medical Student Performance Evaluation (MSPE) is an important tool of communication used by program directors to make decisions in the residency application process. To understand the perspective and usage of the MSPE across multiple medical specialties now and in anticipation of the planned changes in USMLE Step 1 score-reporting. A survey instrument including quantitative and qualitative measures was developed and piloted. The final survey was distributed to residency programs across 28 specialties in 2020 via the main contact on the ACGME listserv. Of the 28 specialties surveyed, at least one response was received from 26 (93%). Eight percent of all programs (364/4675) responded to the survey, with most respondents being program directors. Usage of the MSPE varied among specialties. Approximately 1/3 of end-users stated that the MSPE is very or extremely influential in their initial screening process. Slightly less than half agreed or strongly agreed that they trust the information to be an accurate representation of applicants, though slightly more than half agree that the MSPE will become more influential once USMLE Step 1 becomes pass/fail. Professionalism was rated as the most important component and noteworthy characteristics among the least important in the decision-making process. Performance in the internal medicine clerkship was rated as the most influential while neurology and psychiatry performances were rated as less influential. Overwhelmingly, respondents suggested that including comparative performance and/or class rank would make the MSPE more useful once USMLE Step 1 becomes pass/fail. MSPE end-users across a variety of specialties utilize this complex document in different ways and value it differentially in their decision-making processes. Despite this, continued mistrust of the MSPE persists. A better understanding of end-users' perceptions of the MSPE offers the UME community an opportunity to transform the MSPE into a highly valued, trusted document of communication.
Collapse
Affiliation(s)
- Jeffrey B. Bird
- Barbara Zucker School of Medicine, Hofstra/Northwell, Hempstead, New York, USA
| | - Karen A. Friedman
- Vice Chair for Education, Department of Medicine, Northwell Health, Manhasset, NY, A Professor of Medicine at the Donald and Barbara Zucker School of Medicine, Hofstra/Northwell, Hempstead, New York, USA
| | - Thurayya Arayssi
- Senior Associate Dean, Medical Education and CPD, Weill Cornell Medicine-Qatar and a Professor of Clinical Medicine, Weill Cornell Medicine, NY, NY
| | - Doreen M. Olvet
- Assistant Professor and Medical Education ProjectManager Department of Science Education, Donald, Barbara Zucker School of Medicine, Hofstra/Northwell, Hempstead, New York, USA
| | | | - Judith M. Brenner
- Associate Dean for Educational Data and Analytics Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hofstra University, Hempstead, NY, USA
| |
Collapse
|
14
|
Surgical clerkship: Do examination scores correlate with clinical performance? Am J Surg 2021; 222:1163-1166. [PMID: 34602278 DOI: 10.1016/j.amjsurg.2021.09.016] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2021] [Revised: 08/23/2021] [Accepted: 09/13/2021] [Indexed: 11/23/2022]
Abstract
BACKGROUND This study aims to determine if there are correlations between clinical performance and objective grading parameters for medical students in the third-year surgery clerkship. METHODS Clerkship grades were compiled from 2016 to 2020. Performance on clinical rotations, NBME shelf exam, oral exam, and weekly quizzes were reviewed. Students were divided into quartiles (Q1-Q4) based on clinical performance. Standard statistical analysis was performed. RESULTS There were 625 students included in the study. Students in Q1+Q2 were more likely than those in Q3+Q4 to score in the top quartile on the shelf exam (29% vs. 19%, p = 0.002), oral exam (24% vs. 17%, p = 0.032), and quizzes (22% vs. 15%, p = 0.024). However, there was negligible correlation between clinical performance and performance on objective measures: shelf exam (R2 = 0.027, p < 0.001), oral exam (R2 = 0.021, p < 0.001), and weekly quizzes (R2 = 0.053, p = 0.092). CONCLUSIONS Clinical performance does not correlate with objective grading parameters for medical students in the third-year surgery clerkship.
Collapse
|
15
|
Horan DP, Baldwin K, Purtill JJ, Namdari S. Predictors of Success in an Orthopaedic Residency. JBJS Rev 2021; 9:01874474-202106000-00009. [PMID: 34125721 DOI: 10.2106/jbjs.rvw.20.00180] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
» Orthopaedic surgery is one of the most competitive residencies to match into. Meanwhile, the average applicant's United States Medical Licensing Examination (USMLE) test scores, research involvement, and number of clinical honors increase every year. » Measures such as USMLE scores, productivity in research, Alpha Omega Alpha (AΩA) honor society status, number of clinical honors, and performance on away rotations have all been cited as factors contributing to program directors choosing applicants for interviews and ranking them for their program. However, questions remain as to whether these measures translate to success on board examinations, high faculty evaluations, and designation as chief resident during orthopaedic residency. » USMLE scores have been shown to correlate with Orthopaedic In-Training Examination (OITE) and American Board of Orthopaedic Surgery (ABOS) scores, while clinical grades and AΩA status correlate with faculty evaluations. Participating in research as a medical student was predictive of research productivity in residency but did not correlate with standardized testing scores or faculty evaluations. » The literature has suggested ways in which measures such as personality and grit may be used in the application process and how these factors may contribute to predictors of success. However, additional research is needed to measure and define personality and grit during the application evaluation process.
Collapse
Affiliation(s)
- Dylan P Horan
- Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, Pennsylvania
| | - Keith Baldwin
- The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - James J Purtill
- Rothman Orthopaedic Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania
| | - Surena Namdari
- Rothman Orthopaedic Institute, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania
| |
Collapse
|
16
|
Hernandez CA, Daroowalla F, LaRochelle JS, Ismail N, Tartaglia KM, Fagan MJ, Kisielewski M, Walsh K. Determining Grades in the Internal Medicine Clerkship: Results of a National Survey of Clerkship Directors. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:249-255. [PMID: 33149085 DOI: 10.1097/acm.0000000000003815] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
PURPOSE Trust in and comparability of assessments are essential in clerkships in undergraduate medical education for many reasons, including ensuring competency in clinical skills and application of knowledge important for the transition to residency and throughout students' careers. The authors examined how assessments are used to determine internal medicine (IM) core clerkship grades across U.S. medical schools. METHODS A multisection web-based survey of core IM clerkship directors at 134 U.S. medical schools with membership in the Clerkship Directors in Internal Medicine was conducted in October through November 2018. The survey included a section on assessment practices to characterize current grading scales used, who determines students' final clerkship grades, the nature/type of summative assessments, and how assessments are weighted. Respondents were asked about perceptions of the influence of the National Board of Medical Examiners (NBME) Medicine Subject Examination (MSE) on students' priorities during the clerkship. RESULTS The response rate was 82.1% (110/134). There was considerable variability in the summative assessments and their weighting in determining final grades. The NBME MSE (91.8%), clinical performance (90.9%), professionalism (70.9%), and written notes (60.0%) were the most commonly used assessments. Clinical performance assessments and the NBME MSE accounted for the largest percentage of the total grade (on average 52.8% and 23.5%, respectively). Eighty-seven percent of respondents were concerned that students' focus on the NBME MSE performance detracted from patient care learning. CONCLUSIONS There was considerable variability in what IM clerkships assessed and how those assessments were translated into grades. The NBME MSE was a major contributor to the final grade despite concerns about the impact on patient care learning. These findings underscore the difficulty in comparing learners across institutions and serve to advance discussions for how to improve accuracy and comparability of grading in the clinical environment.
Collapse
Affiliation(s)
- Caridad A Hernandez
- C.A. Hernandez is professor of medicine, Departments of Internal Medicine and Medical Education, University of Central Florida College of Medicine, Orlando, Florida
| | - Feroza Daroowalla
- F. Daroowalla is associate professor of medicine, Department of Medical Education, and Internal Medicine Clerkship Director, University of Central Florida College of Medicine, Orlando, Florida
| | - Jeffrey S LaRochelle
- J.S. LaRochelle is professor of medicine, Department of Medical Education, and assistant dean of medical education, University of Central Florida College of Medicine, Orlando, Florida
| | - Nadia Ismail
- N. Ismail is associate professor of medicine, Department of Medicine, and associate dean, curriculum, Baylor College of Medicine, Houston, Texas
| | - Kimberly M Tartaglia
- K.M. Tartaglia is associate professor of clinical medicine and pediatrics, Division of Hospital Medicine, The Ohio State University, Columbus, Ohio
| | - Mark J Fagan
- M.J. Fagan is professor of medicine emeritus, Department of Medicine, Alpert Medical School of Brown University, Providence, Rhode Island
| | - Michael Kisielewski
- M. Kisielewski is Surveys and Research Manager, Alliance for Academic Internal Medicine, Alexandria, Virginia
| | - Katherine Walsh
- K. Walsh is associate professor of clinical internal medicine, Division of Hematology and Internal Medicine Inpatient Clerkship Director, The Ohio State University, Columbus, Ohio
| |
Collapse
|