1
|
Srinivasan N, Samaan JS, Premkumar A, Rajeev ND, Yeo YH, Samakar K. Perceptions and Expectations Regarding the Step 1 Score Change Among Surgical and Non-Surgical Program Directors: A Systematic Review and Meta-Analysis. Am Surg 2024; 90:1666-1681. [PMID: 38305212 DOI: 10.1177/00031348241230093] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/03/2024]
Abstract
There are currently no studies examining differences in perceptions and expected impact of the Step 1 score change to pass/fail between surgical and non-surgical program directors (PDs). We conducted a systematic review in May 2023 of PubMed, Scopus, Web of Science, and PSYCInfo to evaluate studies examining PDs' perspectives regarding the Step 1 score change. We performed random-effects meta-analyses to determine differences in perspectives among surgical and non-surgical PDs. Surgical PDs (76.8% [95% CI, 72.1%-82.0%], I2 = 52%) reported significantly greater rates of disagreement with the score change compared to non-surgical (65.1% [95% CI, 57.9%-73.1%], I2 = 69.7%) (P = .01). Surgical PDs also reported significantly greater rates of agreement that the score change will increase the difficulty in objectively comparing applicants (88.1% [95% CI, 84.6%-91.7%], I2 = 16.4%), compared to non-surgical (81.0% [95% CI, 75.6%-86.8%], I2 = 72.6%) (P = .04). There was less heterogeneity among non-surgical PDs (88.7% [95% CI, 86.2%-91.2%], I2 = 0%), compared to surgical (84.7% [95% CI, 79.0%-90.8%], I2 = 67.3%), regarding expected increases in emphasis on Step 2, although the difference in rates of agreement was not statistically significant. Overall, there is significant heterogeneity in the literature regarding expected changes in the residency application review process. Most PDs reported significant disagreement with the score change, greater expected difficulty in objectively evaluating applicants, and greater emphasis on Step 2, with surgical PDs reporting greater rates of disagreement, greater expected difficulty, and heterogeneity regarding expected increases in emphasis on Step 2, compared to non-surgical. Additionally, there is significant heterogeneity in the overall literature regarding expected changes in the residency application review process. Further research is needed to establish evidence-based guidelines that improve the overall residency application process for all stakeholders.
Collapse
Affiliation(s)
- Nitin Srinivasan
- Division of Upper GI and General Surgery, Keck School of Medicine of USC, Los Angeles, CA, USA
| | - Jamil S Samaan
- Karsh Division of Gastroenterology and Hepatology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Agnes Premkumar
- Department of Surgery, Phoenix Health Sciences Campus, Creighton University School of Medicine, Phoenix, AZ, USA
| | - Nithya D Rajeev
- Division of Upper GI and General Surgery, Keck School of Medicine of USC, Los Angeles, CA, USA
| | - Yee Hui Yeo
- Karsh Division of Gastroenterology and Hepatology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Kamran Samakar
- Division of Upper GI and General Surgery, Keck School of Medicine of USC, Los Angeles, CA, USA
| |
Collapse
|
2
|
Ryan MS, Lomis KD, Deiorio NM, Cutrer WB, Pusic MV, Caretta-Weyer HA. Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:1251-1260. [PMID: 36972129 DOI: 10.1097/acm.0000000000005220] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is professor and associate dean for assessment, evaluation, research and innovation, Department of Pediatrics, University of Virginia, Charlottesville, Virginia, and a PhD student, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Kimberly D Lomis
- K.D. Lomis is vice president, undergraduate medical education innovations, American Medical Association, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-3504-6776
| | - Nicole M Deiorio
- N.M. Deiorio is professor and associate dean for student affairs, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8123-1112
| | - William B Cutrer
- W.B. Cutrer is associate professor of pediatrics and associate dean for undergraduate medical education, Vanderbilt University School of Medicine, Nashville, Tennessee; ORCID: https://orcid.org/0000-0003-1538-9779
| | - Martin V Pusic
- M.V. Pusic is associate professor of emergency medicine and pediatrics, Department of Pediatrics, Harvard Medical School, Boston, Massachusetts; ORCID: https://orcid.org/0000-0001-5236-6598
| | - Holly A Caretta-Weyer
- H.A. Caretta-Weyer is assistant professor and associate residency director, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| |
Collapse
|
3
|
Alnahhal KI, Lyden SP, Caputo FJ, Sorour AA, Rowe VL, Colglazier JJ, Smith BK, Shames ML, Kirksey L. The USMLE® STEP 1 Pass or Fail Era of the Vascular Surgery Residency Application Process: Implications for Structural Bias and Recommendations. Ann Vasc Surg 2023:S0890-5096(23)00239-X. [PMID: 37120072 DOI: 10.1016/j.avsg.2023.04.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 04/17/2023] [Accepted: 04/19/2023] [Indexed: 05/01/2023]
Abstract
MOTIVES BEHIND THE SHIFT USMLE® STEP 1 score reporting has been changed to a binary pass/fail format since January 26, 2022. The motives behind this change were (1) the questionable validity of using USMLE® STEP 1 as a screening tool during the candidate selection process and (2) the negative impact of using standardized examination scores as an initial gatekeeping threshold for the underrepresented in medicine (URiM) candidates applying to GME programs, given their generally lower mean standardized exams scores compared to non-URiM students. The USMLE® administrators justified this change as a tactic to enhance the overall educational experience for all students and to increase the representation of URiM groups. Moreover, they advised the program directors (PDs) to give more attention to other important qualities and components such as the applicant's personality traits, leadership roles and other extracurricular accomplishments, as part of a holistic evaluation strategy. IMPACT OF USMLE® STEP 1 PASS/FAIL: At this early stage, it is unclear how this change will impact Vascular Surgery Integrated residency (VSIR) programs. Several questions are outstanding, most importantly, how VSIR PDs will evaluate applicants absent the variable which heretofore was the primary screening tool. Our previously published survey showed that VSIR PDs will move their attention to other measures such as USMLE® STEP 2CK and letters of recommendation during the VSIR selection process. Furthermore, more emphasis on subjective measures such as the applicant's medical school rank and extracurricular student activities is expected. Given the expected higher weight of USMLE® STEP 2CK in the selection process than ever, many anticipate that medical students will dedicate more of their limited time to its preparation at the expense of both clinical and non-clinical activities. Potentially leaving less time to explore specialty pathways and to determine whether VS is the appropriate career for them. A FRAMEWORK FOR CANDIDATE EVALUATION The critical juncture in the VSIR candidate evaluation paradigm presents an opportunity to thoughtfully transform the process via current (Standardized Letter of Recommendation, USMLE® STEP 2CK, and clinical research) and future (Emotional Intelligence, Structure Interview and Personality Assessment) measures which constitute a framework to follow in the USMLE® STEP 1 pass/fail era.
Collapse
Affiliation(s)
- Khaled I Alnahhal
- Department of Vascular Surgery, Heart Vascular and Thoracic Institute, Cleveland Clinic, Cleveland
| | - Sean P Lyden
- Department of Vascular Surgery, Heart Vascular and Thoracic Institute, Cleveland Clinic, Cleveland
| | - Francis J Caputo
- Department of Vascular Surgery, Heart Vascular and Thoracic Institute, Cleveland Clinic, Cleveland
| | - Ahmed A Sorour
- Department of Vascular Surgery, Heart Vascular and Thoracic Institute, Cleveland Clinic, Cleveland
| | - Vincent L Rowe
- Division of Vascular and Endovascular Surgery, Department of Surgery, David Geffen School of Medicine at UCLA, Los Angeles
| | - Jill J Colglazier
- Division of Vascular and Endovascular Surgery, Mayo Clinic, Rochester
| | - Brigitte K Smith
- Department of Surgery, Division of Vascular Surgery, University of Utah, Salt Lake City
| | - Murray L Shames
- Division of Vascular Surgery, University of South Florida School of Medicine, Tampa
| | - Lee Kirksey
- Department of Vascular Surgery, Heart Vascular and Thoracic Institute, Cleveland Clinic, Cleveland.
| |
Collapse
|
4
|
Ozair A, Bhat V, Detchou DKE. The US Residency Selection Process After the United States Medical Licensing Examination Step 1 Pass/Fail Change: Overview for Applicants and Educators. JMIR MEDICAL EDUCATION 2023; 9:e37069. [PMID: 36607718 PMCID: PMC9862334 DOI: 10.2196/37069] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/05/2022] [Revised: 10/11/2022] [Accepted: 11/29/2022] [Indexed: 06/07/2023]
Abstract
The United States Medical Licensing Examination (USMLE) Step 1, arguably the most significant assessment in the USMLE examination series, changed from a 3-digit score to a pass/fail outcome in January 2022. Given the rapidly evolving body of literature on this subject, this paper aims to provide a comprehensive review of the historical context and impact of this change on various stakeholders involved in residency selection. For this, relevant keyword-based searches were performed in PubMed, Google Scholar, and Scopus to identify relevant literature. Given the unique history of USMLE Step 1 in the US residency selection process and the score's correlation with future performance in board-certifying examinations in different specialties, this scoring change is predicted to significantly impact US Doctor of Medicine students, US Doctor of Osteopathic Medicine students, international medical graduates, and residency program directors, among others. The significance and the rationale of the pass/fail change along with the implications for both residency applicants and educators are also summarized in this paper. Although medical programs, academic institutions, and residency organizing bodies across the United States have swiftly stepped up to ensure a seamless transition and have attempted to ensure equity for all, the conversion process carries considerable uncertainty for residency applicants. For educators, the increasing number of applications conflicts with holistic application screening, leading to the expected greater use of objective measures, with USMLE Step 2 Clinical Knowledge likely becoming the preferred screening tool in lieu of Step 1.
Collapse
Affiliation(s)
- Ahmad Ozair
- Miami Cancer Institute, Baptist Health South Florida, Miami, FL, United States
- Faculty of Medicine, King George's Medical University, Lucknow, India
| | - Vivek Bhat
- St John's Medical College, Bangalore, India
| | - Donald K E Detchou
- Department of Neurosurgery, Hospital of the University of Pennsylvania, Philadelphia, PA, United States
- Thomas William Langfitt Neurosurgical Society, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, United States
| |
Collapse
|
5
|
Sanders EJ, Wu SA, Neuville AJ, Swiatek PR, Gerlach EB, Saltzman MD, Marra G. Trends in leadership at shoulder and elbow fellowships: a cross-sectional demographic review. J Shoulder Elbow Surg 2022; 31:e92-e100. [PMID: 34543746 DOI: 10.1016/j.jse.2021.08.013] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/30/2021] [Revised: 08/02/2021] [Accepted: 08/10/2021] [Indexed: 02/01/2023]
Abstract
OBJECTIVE To determine educational characteristics, trends, and demographics among shoulder and elbow fellowship leaders (FLs). BACKGROUND Fellowship leaders in shoulder and elbow impart lasting impact on trainees and field development. Four previous studies have analyzed the characteristics and career path trends among orthopedic surgery subspecialty FLs (spine, adult reconstruction, trauma, and sports medicine). We characterized the educational backgrounds and demographic composition of all 40 FLs including fellowship directors (FD), fellowship co-directors (co-FD), and associate fellowship directors (associate FD) of 31 American Shoulder and Elbow Surgeons (ASES)-accredited shoulder and elbow fellowships in the United States. We additionally compiled the residency and fellowship institutions that trained FLs as framework for aspiring leaders in orthopedic surgery. METHODS Using the American Shoulder and Elbow Surgeons (ASES) directory website page "ASES-Recognized Shoulder and Elbow Fellowship Programs," we identified all active shoulder and elbow fellowships within the United States as well as associated FL identifiers. Compiled data points include age, sex, ethnicity, residency/fellowship training location, time since education completion until FL appointment, length in FL role, personal research Scopus H-index, and major society and journal leadership position history. RESULTS We analyzed data from all 40 active FLs across 31 ASES-accredited shoulder and elbow fellowships, encompassing 26 FDs, 13 co-FDs, and 1 associate FD. The majority of FLs (97.5%) were male whereas 2.5% were female, with racial/ethnic identification of 80.0% Caucasian, 10.0% Asian/Pacific Islander, and 10.0% Middle Eastern. The mean Scopus H-index of the FLs was 24.63 ± 16.43. The top residency programs for producing future FLs were the University of Pittsburgh, University of Pennsylvania, University of Nebraska/Creighton, Hospital for Special Surgery, and Brown University (all n = 2). The top fellowship programs for producing future FLs were Mayo Clinic (n = 6), Columbia University (n = 6), San Francisco/California Pacific (n = 4), and Washington University in St Louis (n = 4). CONCLUSION Shoulder and elbow fellowship leaders graduate with increased frequency from certain fellowship programs with lesser correlation to residency institutions. Programs demonstrate high retention of prior trainees as future FLs. All FLs are distinguished by high indices of research productivity; however, demographic diversity remains limited, which is comparable to prior orthopedic subspecialty FL investigations.
Collapse
Affiliation(s)
- Eric J Sanders
- Department of Orthopedic Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Scott A Wu
- Department of Orthopedic Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.
| | - Alexander J Neuville
- Department of Orthopedic Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Peter R Swiatek
- Department of Orthopedic Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Erik B Gerlach
- Department of Orthopedic Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Matthew D Saltzman
- Department of Orthopedic Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Guido Marra
- Department of Orthopedic Surgery, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| |
Collapse
|
6
|
Tamakuwala S, Dean J, Kramer KJ, Shafi A, Ottum S, George J, Kaur S, Chao CR, Recanati MA. Potential Impact of Pass/Fail Scores on USMLE Step 1: Predictors of Excellence in Obstetrics and Gynecology Residency Training. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2021; 8:23821205211037444. [PMID: 34805529 PMCID: PMC8597065 DOI: 10.1177/23821205211037444] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/22/2021] [Accepted: 07/16/2021] [Indexed: 06/07/2023]
Abstract
AIM The study aims to determine resident applicant metrics most predictive of academic and clinical performance as measured by the Council of Resident Education in Obstetrics and Gynecology (CREOG) examination scores and Accreditation Council for Graduate Medical Education (ACGME) clinical performance (Milestones) in the aftermath of United States Medical Licensing Examination Scores (USMLE) Step 1 becoming a pass/fail examination. METHODS In this retrospective study, electronic and paper documents for Wayne State University Obstetrics and Gynecology residents matriculated over a 5-year period ending July 2018 were collected. USMLE scores, clerkship grade, and wording on the letters of recommendation as well as Medical Student Performance Evaluation (MSPE) were extracted from the Electronic Residency Application Service (ERAS) and scored numerically. Semiannual Milestone evaluations and yearly CREOG scores were used as a marker of resident performance. Statistical analysis on residents (n = 75) was performed using R and SPSS and significance was set at P < .05. RESULTS Mean USMLE score correlated with CREOG performance and, of all 3 Steps, Step 1 had the tightest association. MSPE and class percentile also correlated with CREOGs. Clerkship grade and recommendation letters had no correlation with resident performance. Of all metrics provided by ERAS, none taken alone, were as useful as Step 1 scores at predicting performance in residency. Regression modeling demonstrated that the combination of Step 2 scores with MSPE wording restored the predictive ability lost by Step 1. CONCLUSIONS The change of USMLE Step 1 to pass/fail may alter resident selection strategies. Other objective markers are needed in order to evaluate an applicant's future performance in residency.
Collapse
Affiliation(s)
| | | | | | - Adib Shafi
- Wayne State University, Detroit, MI, USA
| | | | | | | | | | | |
Collapse
|