1
|
Khodadadi RB, Yetmar ZA, Domonoske CL, Razonable RR. Factors associated with infectious diseases fellowship academic success. MEDICAL EDUCATION ONLINE 2024; 29:2352953. [PMID: 38720561 PMCID: PMC11086035 DOI: 10.1080/10872981.2024.2352953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Accepted: 05/05/2024] [Indexed: 05/12/2024]
Abstract
BACKGROUND A multitude of factors are considered in an infectious disease (ID) training program's meticulous selection process of ID fellows but their correlation to pre and in-fellowship academic success as well as post-fellowship academic success and short-term outcomes is poorly understood. Our goal was to investigate factors associated with subsequent academic success in fellowship as well as post-fellowship short-term outcomes. METHODS In 2022, we retrospectively analyzed deidentified academic records from 39 graduates of the Mayo Clinic Rochester ID Fellowship Program (1 July 2013- 30 June 2022). Data abstracted included demographics, degrees, honor society membership, visa/citizenship status, medical school, residency training program, United States Medical Licensure Exam (USMLE) scores, letters of recommendation, in-training examination (ITE) scores, fellowship track, academic rank, career choice, number of honors, awards, and abstracts/publications prior to fellowship, during training, and within 2 years of graduation. RESULTS Younger fellows had higher USMLE step 1 scores, pre and in-fellowship scholarly productivity, and higher ITE performance. Female fellows had significantly higher USMLE step 3 scores. Prior research experience translated to greater in-fellowship scholarly productivity. Higher USMLE scores were associated with higher ID ITE performance during multiple years of fellowship, but USMLE step 2 clinical knowledge and 3 scores were associated with higher pre and in-fellowship scholarly productivity and receiving an award during fellowship. The USMLE step 1 score did not correlate with fellowship performance beyond year 1 and 2 ITE scores. CONCLUSIONS Multiple aspects of a prospective fellow's application must be considered as part of a holistic review process for fellowship selection. USMLE step 2 CK and 3 scores may predict fellowship performance across multiple domains.
Collapse
Affiliation(s)
- Ryan B. Khodadadi
- Division of Public Health, Infectious Diseases, and Occupational Medicine, Mayo Clinic, Rochester, MN, USA
| | - Zachary A. Yetmar
- Division of Public Health, Infectious Diseases, and Occupational Medicine, Mayo Clinic, Rochester, MN, USA
| | - Cynthia L. Domonoske
- Division of Public Health, Infectious Diseases, and Occupational Medicine, Mayo Clinic, Rochester, MN, USA
| | - Raymund R. Razonable
- Division of Public Health, Infectious Diseases, and Occupational Medicine, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|
2
|
Drake CB, Rhee DW, Panigrahy N, Heery L, Iturrate E, Stern DT, Sartori DJ. Toward precision medical education: Characterizing individual residents' clinical experiences throughout training. J Hosp Med 2024. [PMID: 39103985 DOI: 10.1002/jhm.13471] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/08/2024] [Revised: 06/22/2024] [Accepted: 07/14/2024] [Indexed: 08/07/2024]
Abstract
BACKGROUND Despite the central role of experiential learning in residency training, the actual clinical experiences residents participate in are not well characterized. A better understanding of the type, volume, and variation in residents' clinical experiences is essential to support precision medical education strategies. OBJECTIVE We sought to characterize the entirety of the clinical experiences had by individual internal medicine residents throughout their time in training. METHOD We evaluated the clinical experiences of medicine residents (n = 51) who completed training at NYU Grossman School of Medicine's Brooklyn campus between 2020 and 2023. Residents' inpatient and outpatient experiences were identified using notes written, orders placed, and care team sign-ins; principal ICD-10 codes for each encounter were converted into medical content categories using a previously described crosswalk tool. RESULTS Of 152,426 clinical encounters with available ICD-10 codes, 132,284 were mapped to medical content categories (94.5% capture). Residents' clinical experiences were particularly enriched in infectious and cardiovascular disease; most had very little exposure to allergy, dermatology, oncology, or rheumatology. Some trainees saw twice as many cases in a given content area as did others. There was little concordance between actual frequency of clinical experience and expected content frequency on the ABIM certification exam. CONCLUSIONS Individual residents' clinical experiences in training vary widely, both in number and in type. Characterizing these experiences paves the way for exploration of the relationships between clinical exposure and educational outcomes, and for the implementation of precision education strategies that could fill residents' experiential gaps and complement strengths with targeted educational interventions.
Collapse
Affiliation(s)
- Carolyn B Drake
- Division of Hospital Medicine, Department of Medicine, Internal Medicine Residency Program, NYU Grossman School of Medicine, New York, New York, USA
| | - David W Rhee
- Leon H. Charney Division of Cardiology, Department of Medicine, NYU Grossman School of Medicine, New York, New York, USA
| | - Neha Panigrahy
- NYU Grossman School of Medicine, New York, New York, USA
| | - Lauren Heery
- NYU Grossman School of Medicine, New York, New York, USA
| | - Eduardo Iturrate
- Division of Hospital Medicine, Department of Medicine, DataCore, Enterprise Research Informatics and Epic Analytics, NYU Grossman School of Medicine, New York, New York, USA
| | - David T Stern
- Department of Medicine, Education and Faculty Affairs, NYU Grossman School of Medicine, New York, New York, USA
- Margaret Cochran Corbin VA Medical Center, New York, New York, USA
| | - Daniel J Sartori
- Division of Hospital Medicine, Department of Medicine, Internal Medicine Residency Program, NYU Grossman School of Medicine, New York, New York, USA
| |
Collapse
|
3
|
Gray BM, Vandergrift JL, Stevens JP, Lipner RS, McDonald FS, Landon BE. Associations of Internal Medicine Residency Milestone Ratings and Certification Examination Scores With Patient Outcomes. JAMA 2024; 332:300-309. [PMID: 38709542 PMCID: PMC11074932 DOI: 10.1001/jama.2024.5268] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/19/2023] [Accepted: 03/14/2024] [Indexed: 05/07/2024]
Abstract
Importance Despite its importance to medical education and competency assessment for internal medicine trainees, evidence about the relationship between physicians' milestone residency ratings or the American Board of Internal Medicine's initial certification examination and their hospitalized patients' outcomes is sparse. Objective To examine the association between physicians' milestone ratings and certification examination scores and hospital outcomes for their patients. Design, Setting, and Participants Retrospective cohort analyses of 6898 hospitalists completing training in 2016 to 2018 and caring for Medicare fee-for-service beneficiaries during hospitalizations in 2017 to 2019 at US hospitals. Main Outcomes and Measures Primary outcome measures included 7-day mortality and readmission rates. Thirty-day mortality and readmission rates, length of stay, and subspecialist consultation frequency were also assessed. Analyses accounted for hospital fixed effects and adjusted for patient characteristics, physician years of experience, and year. Exposures Certification examination score quartile and milestone ratings, including an overall core competency rating measure equaling the mean of the end of residency milestone subcompetency ratings categorized as low, medium, or high, and a knowledge core competency measure categorized similarly. Results Among 455 120 hospitalizations, median patient age was 79 years (IQR, 73-86 years), 56.5% of patients were female, 1.9% were Asian, 9.8% were Black, 4.6% were Hispanic, and 81.9% were White. The 7-day mortality and readmission rates were 3.5% (95% CI, 3.4%-3.6%) and 5.6% (95% CI, 5.5%-5.6%), respectively, and were 8.8% (95% CI, 8.7%-8.9%) and 16.6% (95% CI, 16.5%-16.7%) for mortality and readmission at 30 days. Mean length of stay and number of specialty consultations were 3.6 days (95% CI, 3.6-3.6 days) and 1.01 (95% CI, 1.00-1.03), respectively. A high vs low overall or knowledge milestone core competency rating was associated with none of the outcome measures assessed. For example, a high vs low overall core competency rating was associated with a nonsignificant 2.7% increase in 7-day mortality rates (95% CI, -5.2% to 10.6%; P = .51). In contrast, top vs bottom examination score quartile was associated with a significant 8.0% reduction in 7-day mortality rates (95% CI, -13.0% to -3.1%; P = .002) and a 9.3% reduction in 7-day readmission rates (95% CI, -13.0% to -5.7%; P < .001). For 30-day mortality, this association was -3.5% (95% CI, -6.7% to -0.4%; P = .03). Top vs bottom examination score quartile was associated with 2.4% more consultations (95% CI, 0.8%-3.9%; P < .003) but was not associated with length of stay or 30-day readmission rates. Conclusions and Relevance Among newly trained hospitalists, certification examination score, but not residency milestone ratings, was associated with improved outcomes among hospitalized Medicare beneficiaries.
Collapse
Affiliation(s)
- Bradley M. Gray
- Assessment and Research, American Board of Internal Medicine, Philadelphia, Pennsylvania
| | | | - Jennifer P. Stevens
- Division of Pulmonary, Sleep, and Critical Care Medicine, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts
- Harvard Medical School, Boston, Massachusetts
| | - Rebecca S. Lipner
- Assessment and Research, American Board of Internal Medicine, Philadelphia, Pennsylvania
| | - Furman S. McDonald
- J. Edwin Wood Clinic of the Pennsylvania Hospital, Philadelphia
- Academic and Medical Affairs, American Board of Internal Medicine, Philadelphia, Pennsylvania
| | - Bruce E. Landon
- Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts
- Division of General Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts
| |
Collapse
|
4
|
Ong TQ, Krumm B, Wells M, Read S, Harris L, Altomare A, Paniagua M. A Comparison of Remote vs In-Person Proctored In-Training Examination Administration for Internal Medicine. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2024; 99:778-783. [PMID: 38277440 DOI: 10.1097/acm.0000000000005651] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/28/2024]
Abstract
PURPOSE In response to COVID-19, the American College of Physicians provided residents the option to complete the 2020 Internal Medicine In-Training Examination (IM-ITE) via in-person and remote proctoring. This study evaluated the extent to which scores obtained from both testing modalities were comparable. METHOD Data were analyzed from residents from all U.S.-based Accreditation Council for Graduate Medical Education-accredited IM residency programs and participating Canadian and international programs who completed the IM-ITE in 2020. The final sample contained 27,115 IM residents: 9,205 postgraduate year (PGY) 1, 9,332 PGY-2, and 8,578 PGY-3. Testing modality, gender, PGY, time spent on assessment, and native language were used to predict percent-correct scores in a multilevel regression model. This model included all main effects and all 2-way interactions between testing modality and each resident-level demographic variable, allowing those effects to be controlled for. RESULTS Of 27,115 residents studied, 11,354 (42%) tested remotely and 15,761 (58%) in person. Across the parameters of interest (main effect of testing modality and 2-way interactions), the only statistically significant effects were the interaction effects between testing mode (interaction effect: -0.61; 95% confidence interval [CI], -1.01 to -0.21) and PGY (interaction effect: -0.54; 95% CI, -0.95 to -0.13) ( P = .002). Differences between in-person and remote predicted scores were slightly larger for PGY-1 than for PGY-2 and PGY-3 residents, but the magnitude of these differences across residency training was well under one percentage point. Because these statistically significant effects were deemed educationally nonsignificant, the study concluded that performance did not substantively differ across in-person and remote examinees. CONCLUSIONS Residents taking the 2020 IM-ITE performed similarly across in-person and remote proctoring. This study provides evidence of score comparability across the 2 testing modalities and supports continued use of remote proctoring for the IM-ITE.
Collapse
|
5
|
Seaberg PH, Kling JM, Klanderman MC, Mead-Harvey C, Williams KE, Labonte HR, Jain A, Taylor GE, Blair JE. Resident factors associated with American board of internal medicine certification exam failure. MEDICAL EDUCATION ONLINE 2023; 28:2152162. [PMID: 36443907 PMCID: PMC9718560 DOI: 10.1080/10872981.2022.2152162] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Revised: 11/21/2022] [Accepted: 11/22/2022] [Indexed: 06/16/2023]
Abstract
INTRODUCTION Performance on the certifying examinations such as the American Board of Internal Medicine Certification Exam (ABIM-CE) is of great interest to residents and their residency programs. Identification of factors associated with certification exam result may allow residency programs to recognize and intervene for residents at risk of failing. Despite this, residency programs have few evidence-based predictors of certification exam outcome. The change to pass-or-fail score reporting of the USA Medical Licensing Exam (USMLE) Step 1 removes one such predictor. MATERIALS AND METHODS We performed a retrospective study of residents from a medium-sized internal medicine residency program who graduated from 1998 through 2017. We used univariate tests of associations between ABIM-CE result and various demographic and scholastic factors. RESULTS Of 166 graduates, 14 (8.4%) failed the ABIM-CE on the first attempt. Failing the first attempt of the ABIM-CE was associated with older median age on entering residency (29 vs 27 years; P = 0.01); lower percentile rank on the Internal Medicine In-Training Examination (IM-ITE) in each of the first, second, and third years of training (P < 0.001 for all); and lower scores on the USMLE Steps 1, 2 Clinical Knowledge, and 3 (P < 0.05 for all). No association was seen between a variety of other scholastic or demographic factors and first-attempt ABIM-CE result. DISCUSSION Although USMLE step 1 has changed to a pass-or-fail reporting structure, there are still other characteristics that allow residency programs to identify residents at risk of ABIM-CE first time failure and who may benefit from intervention.
Collapse
Affiliation(s)
- Preston H. Seaberg
- Department of Internal Medicine Charleston Division, West Virginia University School of Medicine, Charleston, West Virginia, USA
| | - Juliana M. Kling
- Division of Women’s Health Internal Medicine, Mayo Clinic, Scottsdale, Arizona, USA
| | - Molly C. Klanderman
- Division of Clinical Trials and Biostatistics, Mayo Clinic, Scottsdale, Arizona, USA
| | - Carolyn Mead-Harvey
- Division of Clinical Trials and Biostatistics, Mayo Clinic, Scottsdale, Arizona, USA
| | | | - Helene R. Labonte
- Division of Community Internal Medicine, Mayo Clinic, Scottsdale, Arizona, USA
| | - Atul Jain
- Division of General Internal Medicine, Mayo Clinic, Scottsdale, Arizona, USA
| | - Gretchen E. Taylor
- Division of Hospital Internal Medicine, Mayo Clinic, Phoenix, Arizona, USA
| | - Janis E. Blair
- Division of Infectious Diseases, Mayo Clinic, Phoenix, AZ, USA
| |
Collapse
|
6
|
Ryan MS, Lomis KD, Deiorio NM, Cutrer WB, Pusic MV, Caretta-Weyer HA. Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:1251-1260. [PMID: 36972129 DOI: 10.1097/acm.0000000000005220] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is professor and associate dean for assessment, evaluation, research and innovation, Department of Pediatrics, University of Virginia, Charlottesville, Virginia, and a PhD student, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Kimberly D Lomis
- K.D. Lomis is vice president, undergraduate medical education innovations, American Medical Association, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-3504-6776
| | - Nicole M Deiorio
- N.M. Deiorio is professor and associate dean for student affairs, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8123-1112
| | - William B Cutrer
- W.B. Cutrer is associate professor of pediatrics and associate dean for undergraduate medical education, Vanderbilt University School of Medicine, Nashville, Tennessee; ORCID: https://orcid.org/0000-0003-1538-9779
| | - Martin V Pusic
- M.V. Pusic is associate professor of emergency medicine and pediatrics, Department of Pediatrics, Harvard Medical School, Boston, Massachusetts; ORCID: https://orcid.org/0000-0001-5236-6598
| | - Holly A Caretta-Weyer
- H.A. Caretta-Weyer is assistant professor and associate residency director, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| |
Collapse
|
7
|
Khodadadi RB, Yetmar ZA, Domonoske CL, Razonable RR. Factors Associated with Infectious Diseases Fellowship Academic Success. RESEARCH SQUARE 2023:rs.3.rs-3140095. [PMID: 37546853 PMCID: PMC10402197 DOI: 10.21203/rs.3.rs-3140095/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/08/2023]
Abstract
Background A multitude of factors are considered in an infectious diseases (ID) training program's meticulous selection process of ID fellows but their correlation to pre and in-fellowship academic success as well as post-fellowship academic success and short-term outcomes is poorly understood. Our goal was to investigate factors associated with subsequent academic success in fellowship as well as post-fellowship short-term outcomes. Methods In 2022, we retrospectively analyzed deidentified academic records from 39 graduates of the Mayo Clinic Rochester ID Fellowship Program (July 1, 2013- June 30, 2022). Data abstracted included demographics, degrees, honor society membership, visa/citizenship status, medical school, residency training program, United States Medical Licensure Exam (USMLE) scores, letters of recommendation, in-training examination (ITE) scores, fellowship track, academic rank, career choice, number of honors, awards, and abstracts/publications prior to fellowship, during training, and within 2 years of graduation. Results Younger fellows had higher USMLE step 1 scores, pre and in-fellowship scholarly productivity, and higher ITE performance. Female fellows had significantly higher USMLE step 3 scores. Prior research experience translated to greater in-fellowship scholarly productivity. Higher USMLE scores were associated with higher ID ITE performance during multiple years of fellowship, but USMLE step 2 clinical knowledge and 3 scores were associated with higher pre and in-fellowship scholarly productivity and receiving an award during fellowship. USMLE step 1 score did not correlate with fellowship performance beyond year 1 and 2 ITE scores. Conclusions Multiple aspects of a prospective fellow's application must be considered as part of a holistic reviewprocess for fellowship selection. USMLE step 2 CK and 3 scores may predict fellowship performance across multiple domains.
Collapse
|
8
|
Manstein SM, Laikhter E, Kazei DD, Comer CD, Shiah E, Lin SJ. The Upcoming Pass/Fail USMLE Step 1 Score Reporting: An Impact Assessment From Medical School Deans. Plast Surg (Oakv) 2023; 31:169-176. [PMID: 37188137 PMCID: PMC10170630 DOI: 10.1177/22925503211034838] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2021] [Accepted: 06/22/2021] [Indexed: 11/15/2022] Open
Abstract
Background: The US Medical Licensing Examination (USMLE) Step 1 change to pass/fail has been met with mixed reviews, and the impact on medical student education and residency match is unknown. We surveyed medical school student affairs deans regarding their thoughts on the upcoming transition of Step 1 to pass/fail. Methods: A questionnaire was emailed to medical school deans. Deans were asked to rank the importance of the following after the Step 1 reporting change: Step 2 Clinical Knowledge (Step 2 CK), clerkship grades, letters of recommendation, personal statement, medical school reputation, class rank, Medical Student Performance Evaluation, and research. They were asked how the score change will affect curriculum, learning, diversity, and student mental health. Deans were asked to select 5 specialties they thought would be most affected. Results: Regarding perceived importance of residency applications following the scoring change, the most frequent number 1 choice was Step 2 CK. The majority of deans (93.5%, n = 43) felt that the change to pass/fail would benefit medical student education/learning environment; however, most (68.2%, n = 30) did not believe their school curriculum would change. Students applying to dermatology, neurosurgery, orthopedic surgery, ENT, and plastic surgery were felt to be most affected by the scoring change; 58.7% (n = 27) felt it would not adequately address future diversity. Conclusion: The majority of deans feel the USMLE Step 1 change to pass/fail would benefit medical student education. Deans feel that students applying to traditionally more competitive specialties (ie, programs with fewer overall residency positions available) will be most affected.
Collapse
Affiliation(s)
- Samuel M. Manstein
- Division of Plastic Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
| | - Elizabeth Laikhter
- Division of Plastic Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
| | | | - Carly D. Comer
- Division of Plastic Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
| | - Eric Shiah
- Division of Plastic Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
| | - Samuel J. Lin
- Division of Plastic Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
| |
Collapse
|
9
|
Shirkhodaie C, Avila S, Seidel H, Gibbons RD, Arora VM, Farnan JM. The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:264-273. [PMID: 36512984 DOI: 10.1097/acm.0000000000005061] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
PURPOSE With the change in Step 1 score reporting, Step 2 Clinical Knowledge (CK) may become a pivotal factor in resident selection. This systematic review and meta-analysis seeks to synthesize existing observational studies that assess the relationship between Step 2 CK scores and measures of resident performance. METHOD The authors searched MEDLINE, Web of Science, and Scopus databases using terms related to Step 2 CK in 2021. Two researchers identified studies investigating the association between Step 2 CK and measures of resident performance and included studies if they contained a bivariate analysis examining Step 2 CK scores' association with an outcome of interest: in-training examination (ITE) scores, board certification examination scores, select Accreditation Council for Graduate Medical Education core competency assessments, overall resident performance evaluations, or other subjective measures of performance. For outcomes that were investigated by 3 or more studies, pooled effect sizes were estimated with random-effects models. RESULTS Among 1,355 potential studies, 68 met inclusion criteria and 43 were able to be pooled. There was a moderate positive correlation between Step 2 CK and ITE scores (0.52, 95% CI 0.45-0.59, P < .01). There was a moderate positive correlation between Step 2 CK and ITE scores for both nonsurgical (0.59, 95% CI 0.51-0.66, P < .01) and surgical specialties (0.41, 95% CI 0.33-0.48, P < .01). There was a very weak positive correlation between Step 2 CK scores and subjective measures of resident performance (0.19, 95% CI 0.13-0.25, P < .01). CONCLUSIONS This study found Step 2 CK scores have a statistically significant moderate positive association with future examination scores and a statistically significant weak positive correlation with subjective measures of resident performance. These findings are increasingly relevant as Step 2 CK scores will likely become more important in resident selection.
Collapse
Affiliation(s)
- Camron Shirkhodaie
- C. Shirkhodaie is a medical student, Pritzker School of Medicine, University of Chicago, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-4279-3251
| | - Santiago Avila
- S. Avila is a medical student, Pritzker School of Medicine, University of Chicago, Chicago, Illinois; ORCID: https://orcid.org/0000-0003-3633-4304
| | - Henry Seidel
- H. Seidel is a medical student, Pritzker School of Medicine, University of Chicago, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-7364-1365
| | - Robert D Gibbons
- R.D. Gibbons is professor, Center for Health Statistics and Departments of Medicine and Public Health Sciences, University of Chicago, Chicago, Illinois
| | - Vineet M Arora
- V.M. Arora is professor, Department of Medicine, University of Chicago Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-4745-7599
| | - Jeanne M Farnan
- J.M. Farnan is professor, Department of Medicine, University of Chicago Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-1138-9416
| |
Collapse
|
10
|
Sorah JD, Clark-Garvey S, Collichio FA. Brief Report: Improving Fellow Performance of the Pharmacology Section of the American Society of Clinical Oncology In-Training Exam. JOURNAL OF CANCER EDUCATION : THE OFFICIAL JOURNAL OF THE AMERICAN ASSOCIATION FOR CANCER EDUCATION 2023; 38:141-145. [PMID: 34535877 DOI: 10.1007/s13187-021-02089-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 09/09/2021] [Indexed: 06/13/2023]
Abstract
Though the electronic medical record has streamlined the process for ordering chemotherapy and reduced medical error, it has come at the cost of potentially hindering medical oncology fellows' acquisition of basic principles of chemotherapy administration. Recognizing the need for improving fellow knowledge and comfort level in chemotherapy prescription, we created an anticancer therapeutics curriculum for fellows at the University of North Carolina (UNC). The curriculum, which was comprised of a self-assessment, review session, and chemotherapy writing workshop, was offered voluntarily to all UNC fellows planning to take the medical oncology boards. We distributed a pre-intervention and post-intervention survey. In total, 11 of 18 fellows participated. After our interventions, there were statistically significant improvements in comfortability with calculating a dose of chemotherapy (p = 0.002), writing orders (p = 0.004), and taking the American Society of Clinical Oncology (ASCO) In-Training Exam (ITE) (p = 0.002). Furthermore, we saw a 4% overall improvement in pharmacology subset scores on the ITE compared to the prior year. Overall, our study suggested that the addition of a chemotherapy competency curriculum led to improved outcomes on the ASCO ITE, improved comfortability in chemotherapy administration, and hopefully improved patient care.
Collapse
Affiliation(s)
- Jonathan D Sorah
- Division of Oncology, Department of Medicine, University of North Carolina at Chapel Hill, 170 Manning Drive, CB #7305, Chapel Hill, NC, 27599, USA.
| | - Sean Clark-Garvey
- Division of Oncology, Department of Medicine, University of North Carolina at Chapel Hill, 170 Manning Drive, CB #7305, Chapel Hill, NC, 27599, USA
| | - Frances A Collichio
- Division of Oncology, Department of Medicine, University of North Carolina at Chapel Hill, 170 Manning Drive, CB #7305, Chapel Hill, NC, 27599, USA
| |
Collapse
|
11
|
Lee SH, Phan PH, Desai SV. Evaluation of house staff candidates for program fit: a cohort-based controlled study. BMC MEDICAL EDUCATION 2022; 22:754. [PMID: 36320029 PMCID: PMC9628087 DOI: 10.1186/s12909-022-03801-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Accepted: 10/10/2022] [Indexed: 06/16/2023]
Abstract
BACKGROUND Medical school academic achievements do not necessarily predict house staff job performance. This study explores a selection mechanism that improves house staff-program fit that enhances the Accreditation Council for Graduate Medical Education Milestones performance ratings. OBJECTIVE Traditionally, house staff were selected primarily on medical school academic performance. To improve residency performance outcomes, the Program designed a theory-driven selection tool to assess house staff candidates on their personal values and goals fit with Program values and goals. It was hypothesized cohort performance ratings will improve because of the intervention. METHODS Prospective quasi-experimental cohort design with data from two house staff cohorts at a university-based categorical Internal Medicine Residency Program. The intervention cohort, comprising 45 house staff from 2016 to 2017, was selected using a Behaviorally Anchored Rating Scales (BARS) tool for program fit. The control cohort, comprising 44 house staff from the prior year, was selected using medical school academic achievement scores. House staff performance was evaluated using ACGME Milestones indicators. The mean scores for each category were compared between the intervention and control cohorts using Student's t-tests with Bonferroni correction and Cohen's d for effect size. RESULTS The cohorts were no different in academic performance scores at time of Program entry. The intervention cohort outperformed the control cohort on all 6 dimensions of Milestones by end-PGY1 and 3 of 6 dimensions by mid-PGY3. CONCLUSION Selecting house staff based on compatibility with Residency Program values and objectives may yield higher job performance because trainees benefit more from a better fit with the training program.
Collapse
Affiliation(s)
- Soo-Hoon Lee
- Strome College of Business, Old Dominion University, Norfolk, VA, USA
| | - Phillip H Phan
- Johns Hopkins Carey Business School, Johns Hopkins University, Baltimore, MD, USA.
- Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, MD, USA.
| | - Sanjay V Desai
- Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
12
|
Latif A, Ahsan MJ, Jaffri A, Kapoor V, Tran A, DeVrieze B, Nahas J, Kim M, Porter J, Mirza M. A new dawn in internal medicine education curriculum—a project by the residents, with the residents, for the residents. Proc AMIA Symp 2022; 35:798-801. [DOI: 10.1080/08998280.2022.2109097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2022] Open
Affiliation(s)
- Azka Latif
- Baylor College of Medicine, Houston, Texas
| | | | - Abbis Jaffri
- Creighton University School of Pharmacy and Health Professions, Omaha, Nebraska
| | - Vikas Kapoor
- CHI Health, Good Samaritan Hospital, Kearney, Nebraska
| | - Amy Tran
- Creighton University School of Medicine, Omaha, Nebraska
| | | | - Joseph Nahas
- Creighton University School of Medicine, Omaha, Nebraska
| | - Michael Kim
- Creighton University School of Medicine, Omaha, Nebraska
| | - Joann Porter
- Creighton University School of Medicine, Omaha, Nebraska
| | - Mohsin Mirza
- Creighton University School of Medicine, Omaha, Nebraska
| |
Collapse
|
13
|
Rubright JD, Ong TQ, Jodoin MG, Johnson DA, Barone MA. Revisiting Retake Policy: Analyzing the Success Rates of Examinees With Multiple Attempts on the United States Medical Licensing Examination. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1219-1225. [PMID: 35442911 DOI: 10.1097/acm.0000000000004713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE A testing program's decision making on retake policy considers a number of factors, including fairness to examinees, examination security, examination purpose, and classification accuracy. For high-stakes licensure and certification examinations charged with protection of the public, this includes balancing fairness issues inherent in the potential for false negatives with the public protection need to minimize false positives. Since 2012, the United States Medical Licensing Examination (USMLE) has maintained a policy of ≤ 6 attempts on any examination component. The purpose of this study was to empirically examine the appropriateness of existing USMLE retake policy. METHOD A sample of over 300,000 examinees who were administered each Step examination between 2008 and 2018 was examined to address how pass rates (per attempt and cumulatively) differed for examinees and examinee subgroups via graphic depictions of pass rates, the impact of restricting the number of attempts on examinees and examinee subgroups via logistic regression analyses, and the impact of restricting the number of attempts on classification error via classification error analyses. RESULTS Repeaters displayed consistently lower pass rates on subsequent attempts. Regardless of Step, most passing examinees did so by their fourth attempt. Models predicting the log odds of examinees taking ≥ 5 attempts showed a significant effect of gender, race, and medical school status, although the associated sample sizes are small. Misclassification analyses showed that, as attempts increased, the percentage of false passers increased and percentage of false failers decreased. CONCLUSIONS These considerations led to the USMLE's decision to reduce its attempt limit from 6 to 4 on any Step or Step component effective July 2021. This change balances the competing interests of fairness and examination access to the examinee population, while simultaneously maintaining a minimum standard consistent with the mission of the USMLE program to protect the public.
Collapse
Affiliation(s)
- Jonathan D Rubright
- J.D. Rubright is vice president, Office of Research, National Board of Medical Examiners, Philadelphia, Pennsylvania
| | - Thai Q Ong
- T.Q. Ong is psychometrician, Psychometrics and Data Analysis, National Board of Medical Examiners, Philadelphia, Pennsylvania
| | - Michael G Jodoin
- M.G. Jodoin is senior vice president, Customer and Portfolio Management, National Board of Medical Examiners, Philadelphia, Pennsylvania
| | - David A Johnson
- D.A. Johnson is chief assessment officer, Federation of State Medical Boards, Euless, Texas
| | - Michael A Barone
- M.A. Barone is vice president, Competency Based Assessment, National Board of Medical Examiners, Philadelphia, Pennsylvania
| |
Collapse
|
14
|
Klein R, Koch J, Snyder ED, Volerman A, Simon W, Jassal SK, Cosco D, Cioletti A, Ufere NN, Burnett-Bowie SAM, Palamara K, Schaeffer S, Julian KA, Thompson V. Association of Gender and Race/Ethnicity with Internal Medicine In-Training Examination Performance in Graduate Medical Education. J Gen Intern Med 2022; 37:2194-2199. [PMID: 35710653 PMCID: PMC9296734 DOI: 10.1007/s11606-022-07597-z] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/13/2021] [Accepted: 03/30/2022] [Indexed: 11/24/2022]
Abstract
BACKGROUND Disparities in objective assessments in graduate medical education such as the In-Training Examination (ITE) that disadvantage women and those self-identifying with race/ethnicities underrepresented in medicine (URiM) are of concern. OBJECTIVE Examine ITE trends longitudinally across post-graduate year (PGY) with gender and race/ethnicity. DESIGN Longitudinal analysis of resident ITE metrics at 7 internal medicine residency programs, 2014-2019. ITE trends across PGY of women and URiM residents compared to non-URiM men assessed via ANOVA. Those with ITE scores associated with less than 90% probability of passing the American Board of Internal Medicine certification exam (ABIM-CE) were identified and odds of being identified as at-risk between groups were assessed with chi square. PARTICIPANTS A total of 689 IM residents, including 330 women and URiM residents (48%). MAIN MEASURES ITE score KEY RESULTS: There was a significant difference in ITE score across PGY for women and URiM residents compared to non-URiM men (F(2, 1321) 4.46, p=0.011). Adjusting for program, calendar year, and baseline ITE, women and URiM residents had smaller ITE score gains (adjusted mean change in score between PGY1 and PGY3 (se), non-URiM men 13.1 (0.25) vs women and URiM residents 11.4 (0.28), p<0.001). Women and URiM residents had greater odds of being at potential risk for not passing the ABIM-CE (OR 1.75, 95% CI 1.10 to 2.78) with greatest odds in PGY3 (OR 3.13, 95% CI 1.54 to 6.37). CONCLUSION Differences in ITE over training were associated with resident gender and race/ethnicity. Women and URiM residents had smaller ITE score gains across PGY translating into greater odds of potentially being seen as at-risk for not passing the ABIM-CE. Differences in ITE over training may reflect differences in experiences of women and URiM residents during training and may lead to further disparities.
Collapse
Affiliation(s)
- Robin Klein
- Department of Medicine, Division of General Internal Medicine and Geriatrics, Emory University School of Medicine, 49 Jesse Hill Jr Dr, Atlanta, GA, 30303, USA.
| | - Jennifer Koch
- Department of Medicine, University of Louisville, Louisville, KY, USA
| | - Erin D Snyder
- Department of Medicine, University of Alabama Birmingham School of Medicine, Birmingham, AL, USA
| | - Anna Volerman
- Departments of Medicine and Pediatrics, University of Chicago, Chicago, IL, USA
| | - Wendy Simon
- Department of Medicine, University of California, Los Angeles, Los Angeles, USA
| | - Simerjot K Jassal
- Department of Medicine, VA San Diego Healthcare System, University of California, San Diego, San Diego, USA
| | - Dominique Cosco
- Department of Medicine, Washington University St. Louis, St. Louis, USA
| | - Anne Cioletti
- Department of Medicine, University of Utah, Salt Lake City, USA
| | - Nneka N Ufere
- Department of Medicine, Massachusetts General Hospital, Boston, MA, USA
| | | | - Kerri Palamara
- Department of Medicine, Massachusetts General Hospital, Boston, MA, USA
| | - Sarah Schaeffer
- Department of Medicine, University of California, San Francisco, San Francisco, CA, USA
| | - Katherine A Julian
- Department of Medicine, University of California, San Francisco, San Francisco, CA, USA
| | - Vanessa Thompson
- Department of Medicine, University of California, San Francisco, San Francisco, CA, USA
| |
Collapse
|
15
|
Norcini JJ, Weng W, Boulet J, McDonald F, Lipner RS. Associations between initial American Board of Internal Medicine certification and maintenance of certification status of attending physicians and in-hospital mortality of patients with acute myocardial infarction or congestive heart failure: a retrospective cohort study of hospitalisations in Pennsylvania, USA. BMJ Open 2022; 12:e055558. [PMID: 35470191 PMCID: PMC9058798 DOI: 10.1136/bmjopen-2021-055558] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Abstract
OBJECTIVE To determine whether internists' initial specialty certification and the maintenance of that certification (MOC) is associated with lower in-hospital mortality for their patients with acute myocardial infarction (AMI) or congestive heart failure (CHF). DESIGN Retrospective cohort study of hospitalisations in Pennsylvania, USA, from 2012 to 2017. SETTING All hospitals in Pennsylvania. PARTICIPANTS All 184 115 hospitalisations for primary diagnoses of AMI or CHF where the attending physician was a self-designated internist. PRIMARY OUTCOME MEASURE In-hospital mortality. RESULTS Of the 2575 physicians, 2238 had initial certification and 820 were eligible for MOC. After controlling for patient demographics and clinical characteristics, hospital-level factors and physicians' demographic and medical school characteristics, both initial certification and MOC were associated with lower mortality. The adjusted OR for initial certification was 0.835 (95% CI 0.756 to 0.922; p<0.001). Patients cared for by physicians with initial certification had a 15.87% decrease in mortality compared with those cared for by non-certified physicians (mortality rate difference of 5.09 per 1000 patients; 95% CI 2.12 to 8.05; p<0.001). The adjusted OR for MOC was 0.804 (95% CI 0.697 to 0.926; p=0.003). Patients cared for by physicians who completed MOC had an 18.91% decrease in mortality compared with those cared for by MOC lapsed physicians (mortality rate difference of 6.22 per 1000 patients; 95% CI 2.0 to 10.4; p=0.004). CONCLUSIONS Initial certification was associated with lower mortality for AMI or CHF. Moreover, for patients whose physicians had initial certification, an additional advantage was associated with its maintenance.
Collapse
Affiliation(s)
| | - Weifeng Weng
- American Board of Internal Medicine, Philadelphia, Pennsylvania, USA
| | | | - Furman McDonald
- American Board of Internal Medicine, Philadelphia, Pennsylvania, USA
| | - Rebecca S Lipner
- American Board of Internal Medicine, Philadelphia, Pennsylvania, USA
| |
Collapse
|
16
|
Liu P, Chen SY, Chang YC, Ng CJ, Chaou CH. Multimodal In-training Examination in an Emergency Medicine Residency Training Program: A Longitudinal Observational Study. Front Med (Lausanne) 2022; 9:840721. [PMID: 35355591 PMCID: PMC8959571 DOI: 10.3389/fmed.2022.840721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2021] [Accepted: 02/10/2022] [Indexed: 11/13/2022] Open
Abstract
Background In-training examination (ITE) has been widely adopted as an assessment tool to measure residents' competency. We incorporated different formats of assessments into the emergency medicine (EM) residency training program to form a multimodal, multistation ITE. This study was conducted to examine the cost and effectiveness of its different testing formats. Methods We conducted a longitudinal study in a tertiary teaching hospital in Taiwan. Nine EM residents were enrolled and followed for 4 years, and the biannual ITE scores were recorded and analyzed. Each ITE consisted of 8–10 stations and was categorized into four formats: multiple-choice question (MCQ), question and answer (QA), oral examination (OE), and high-fidelity simulation (HFS) formats. The learner satisfaction, validity, reliability, and costs were analyzed. Results 486 station scores were recorded during the 4 years. The numbers of MCQ, OE, QA, and HFS stations were 45 (9.26%), 90 (18.5%), 198 (40.7%), and 135 (27.8%), respectively. The overall Cronbach's alpha reached 0.968, indicating good overall internal consistency. The correlation with EM board examination was highest for HFS (ρ = 0.657). The average costs of an MCQ station, an OE station, and an HFS station were ~3, 14, and 21 times that of a QA station. Conclusions Multi-dimensional assessment contributes to good reliability. HFS correlates best with the final training exam score but is also the most expensive format among ITEs. Increased testing domains with various formats improve ITE's overall reliability. Program directors must understand each test format's strengths and limitations to bring forth the best combination of exams under the local context.
Collapse
Affiliation(s)
- Pin Liu
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Lin-Kou Medical Center, Taoyuan, Taiwan.,Department of Emergency Medicine, West Garden Hospital, Taipei, Taiwan
| | - Shou-Yen Chen
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Lin-Kou Medical Center, Taoyuan, Taiwan.,Graduate Institute of Clinical Medical Sciences, Division of Medical Education, College of Medicine, Chang Gung University, Taoyuan, Taiwan
| | - Yu-Che Chang
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Lin-Kou Medical Center, Taoyuan, Taiwan.,Chang Gung, Medical Education Research Center, Taoyuan, Taiwan
| | - Chip-Jin Ng
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Lin-Kou Medical Center, Taoyuan, Taiwan
| | - Chung-Hsien Chaou
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Lin-Kou Medical Center, Taoyuan, Taiwan.,Chang Gung, Medical Education Research Center, Taoyuan, Taiwan
| |
Collapse
|
17
|
Tackett S, Jeyaraju M, Moore J, Hudder A, Yingling S, Park YS, Grichanik M. Student well-being during dedicated preparation for USMLE Step 1 and COMLEX Level 1 exams. BMC MEDICAL EDUCATION 2022; 22:16. [PMID: 34983481 PMCID: PMC8728922 DOI: 10.1186/s12909-021-03055-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Accepted: 11/17/2021] [Indexed: 05/15/2023]
Abstract
BACKGROUND Nearly all U.S. medical students engage in a 4-8 week period of intense preparation for their first-level licensure exams, termed a "dedicated preparation period" (DPP). It is widely assumed that student well-being is harmed during DPPs, but evidence is limited. This study characterized students' physical, intellectual, emotional, and social well-being during DPPs. METHODS This was a cross-sectional survey sent electronically to all second-year students at four U.S. medical schools after each school's respective DPP for USMLE Step 1 or COMLEX Level 1 in 2019. Survey items assessed DPP characteristics, cost of resources, and perceived financial strain as predictors for 18 outcomes measured by items with Likert-type response options. Open-ended responses on DPPs' influence underwent thematic analysis. RESULTS A total of 314/750 (42%) students completed surveys. DPPs lasted a median of 7 weeks (IQR 6-8 weeks), and students spent 70 h/week (IQR 56-80 h/week) studying. A total of 62 (20%) reported experiencing a significant life event that impacted their ability to study during their DPPs. Most reported 2 outcomes improved: medical knowledge base (95%) and confidence in ability to care for patients (56%). Most reported 9 outcomes worsened, including overall quality of life (72%), feeling burned out (77%), and personal anxiety (81%). A total of 25% reported paying for preparation materials strained their finances. Greater perceived financial strain was associated with worsening 11 outcomes, with reported amount spent associated with worsening 2 outcomes. Themes from student descriptions of how DPPs for first-level exams influenced them included (1) opportunity for synthesis of medical knowledge, (2) exercise of endurance and self-discipline required for professional practice, (3) dissonance among exam preparation resource content, formal curriculum, and professional values, (4) isolation, deprivation, and anguish from competing for the highest possible score, and (5) effects on well-being after DPPs. CONCLUSIONS DPPs are currently experienced by many students as a period of personal and social deprivation, which may be worsened by perceived financial stress more than the amount of money they spend on preparation materials. DPPs should be considered as a target for reform as medical educators attempt to prevent student suffering and enhance their well-being.
Collapse
Affiliation(s)
- Sean Tackett
- Johns Hopkins Bayview Medical Center, 5200 Eastern Ave, MFL Center Tower Suite 2300, Baltimore, MD 21224 USA
| | | | - Jesse Moore
- Larner College of Medicine at the University of Vermont, Burlington, USA
| | - Alice Hudder
- Lake Erie, College of Osteopathic Medicine, Erie, USA
| | | | - Yoon Soo Park
- Harvard Medical School and Massachusetts General Hospital, Boston, USA
| | - Mark Grichanik
- David Geffen School of Medicine at UCLA, Los Angeles, USA
| |
Collapse
|
18
|
Swails JL, Adams S, Hormann M, Omoruyi E, Aibana O. Mission-Based Filters in the Electronic Residency Application Service: Saving Time and Promoting Diversity. J Grad Med Educ 2021; 13:785-794. [PMID: 35070090 PMCID: PMC8672840 DOI: 10.4300/jgme-d-21-00302.1] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 06/25/2021] [Accepted: 09/29/2021] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Holistic review promotes diversity, but widespread implementation remains limited. OBJECTIVE We aimed to develop a practical approach to incorporate holistic review principles in screening applicants in the Electronic Residency Application Service (ERAS) and to assess the impact on diversity. METHODS Three residency programs (internal medicine [IM], pediatrics, and surgery) at McGovern Medical School developed filters to identify applicants with experiences/attributes aligned with the institutional mission. These filters were retroactively applied to each program's 2019-2020 applicant pool using built-in ERAS capabilities to group applicants by user-defined features. We compared the demographics of applicants reviewed during the cycle with those identified retrospectively through experiences/attributes filters. RESULTS The IM, pediatrics, and surgery programs received 3527, 1341, and 1313 applications, respectively, in 2019-2020. Retrospective use of experiences/attributes filters, without scores, narrowed the IM applicant pool for review to 1301 compared to 1323 applicants reviewed during actual recruitment, while the pediatrics filters identified 514 applicants compared to 384 at baseline. The surgery filters resulted in 582 applicants, but data were missing for baseline comparison. Compared to the baseline screening approach utilizing scores, mission-based filters increased the proportions of underrepresented in medicine applicants selected for review in IM (54.8% [95% CI 52.1-57.5] vs 22.7% [20.4-24.9], P < .0001) and pediatrics (63.2% [95% CI 59.1-67.4] vs 25.3% [20.9-29.6], P < .0001). CONCLUSIONS Program directors can leverage existing ERAS features to conduct application screening in alignment with holistic review principles. Widespread implementation could have important repercussions for enhancing physician workforce diversity.
Collapse
Affiliation(s)
- Jennifer L. Swails
- All authors are with McGovern Medical School, UTHealth
- Jennifer L. Swails, MD, is Program Director, Co-Director of Interprofessional Education, and Associate Professor of Medicine
| | - Sasha Adams
- All authors are with McGovern Medical School, UTHealth
- Sasha Adams, MD, is Program Director and Associate Professor of Surgery
| | - Mark Hormann
- All authors are with McGovern Medical School, UTHealth
- Mark Hormann, MD, is Assistant Dean for Educational Programs, Program Director, and Associate Professor of Pediatrics
| | - Emma Omoruyi
- All authors are with McGovern Medical School, UTHealth
- Emma Omoruyi, MD, MPH, is Associate Program Director and Associate Professor of Pediatrics
| | - Omowunmi Aibana
- All authors are with McGovern Medical School, UTHealth
- Omowunmi Aibana, MD, MPH, is Vice Chair of Diversity and Inclusion, Associate Program Director, and Assistant Professor of Medicine
| |
Collapse
|
19
|
Quaggin SE, Safar-Boueri L. The Art and Science of Medicine … and Standardized Test Scores. J Am Soc Nephrol 2021; 32:2694-2696. [PMID: 34706970 PMCID: PMC8806096 DOI: 10.1681/asn.2021091245] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023] Open
Affiliation(s)
- Susan E. Quaggin
- Division of Nephrology and Hypertension, Northwestern University, Feinberg School of Medicine, Chicago, Illinois,Department of Medicine, Feinberg Cardiovascular and Renal Research Institute, Northwestern University, Chicago, Illinois
| | - Luisa Safar-Boueri
- Division of Nephrology and Hypertension, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| |
Collapse
|
20
|
Berns JS, Weng W, Jaar BG, Lipner RS, Brossman BG, McDonald FS. Analysis of Performance Trends from 2010-2019 on the American Board of Internal Medicine Nephrology Certifying Exam. J Am Soc Nephrol 2021; 32:2714-2723. [PMID: 34706969 PMCID: PMC8806090 DOI: 10.1681/asn.2021020160] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2021] [Accepted: 06/05/2021] [Indexed: 02/04/2023] Open
Abstract
BACKGROUND The pass rate on the American Board of Internal Medicine (ABIM) nephrology certifying exam has declined and is among the lowest of all internal medicine (IM) subspecialties. In recent years, there have also been fewer applicants for the nephrology fellowship match. METHODS This retrospective observational study assessed how changes between 2010 and 2019 in characteristics of 4094 graduates of US ACGME-accredited nephrology fellowship programs taking the ABIM nephrology certifying exam for the first time, and how characteristics of their fellowship programs were associated with exam performance. The primary outcome measure was performance on the nephrology certifying exam. Fellowship program pass rates over the decade were also studied. RESULTS Lower IM certifying exam score, older age, female sex, international medical graduate (IMG) status, and having trained at a smaller nephrology fellowship program were associated with poorer nephrology certifying exam performance. The mean IM certifying exam percentile score among those who subsequently took the nephrology certifying exam decreased from 56.7 (SD, 27.9) to 46.1 (SD, 28.7) from 2010 to 2019. When examining individuals with comparable IM certifying exam performance, IMGs performed less well than United States medical graduates (USMGs) on the nephrology certifying exam. In 2019, only 57% of nephrology fellowship programs had aggregate 3-year certifying exam pass rates ≥80% among their graduates. CONCLUSIONS Changes in IM certifying exam performance, certain trainee demographics, and poorer performance among those from smaller fellowship programs explain much of the decline in nephrology certifying exam performance. IM certifying exam performance was the dominant determinant.
Collapse
Affiliation(s)
- Jeffrey S. Berns
- Department of Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania,Renal, Electrolyte, and Hypertension Division, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania
| | - Weifeng Weng
- American Board of Internal Medicine, Philadelphia, Pennsylvania
| | - Bernard G. Jaar
- Department of Medicine, Johns Hopkins School of Medicine, Baltimore, Maryland,Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland,Welch Center for Prevention, Epidemiology, and Clinical Research, Baltimore, Maryland,Nephrology Center of Maryland, Baltimore, Maryland
| | | | | | - Furman S. McDonald
- American Board of Internal Medicine, Philadelphia, Pennsylvania,J. Edwin Wood Clinic of the Pennsylvania Hospital, Department of Medicine, Philadelphia, Pennsylvania
| |
Collapse
|
21
|
Horton WB, Patrie JT, Duhigg LM, Graham M, True MW, Pelley EM, Dalkin AC. Novel Formative Approach of the ESAP-ITE Provides Strong Predictive Value for ABIM Certification Outcomes. J Endocr Soc 2021; 5:bvab137. [PMID: 34585036 PMCID: PMC8464176 DOI: 10.1210/jendso/bvab137] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/28/2021] [Indexed: 11/19/2022] Open
Abstract
Background The Endocrine Self-Assessment Program In-Training Examination (ESAP-ITE) has the novel formative approach of allowing open access to all questions and answers after secure examination administration is complete, resulting in the creation of an entirely new in-training examination annually. Objective To determine whether scores on the novel ESAP-ITE predict pass/fail outcomes on the American Board of Internal Medicine Endocrinology, Diabetes, and Metabolism Certification Examination (ABIM-ECE). Methods All endocrine fellows-in-training who took the ESAP-ITE between 2016 and 2019 and then subsequently attempted the ABIM-ECE within the same calendar year were included (n = 982). Primary analyses used the ESAP-ITE score from the final year of fellowship training. Covariates included sex, age on date of ABIM-ECE, medical school country, fellowship program region, pass/fail outcomes on the ABIM Internal Medicine Certification Examination, and ESAP-ITE score. All variables were analyzed using multivariable logistic regression. Results ESAP-ITE score (P < 0.001), ABIM Internal Medicine Certification Examination outcome (P < 0.001), and age (P = 0.005) were each significant predictors of passing the ABIM-ECE on the first attempt. ESAP-ITE score was the strongest predictor of passing the ABIM-ECE, and this relationship was such that a score of 75% correct yielded a 97% probability of passing the ABIM-ECE, whereas a score of 50% correct generated only a 70% probability of doing so. Sex, fellowship program region, and medical school country were not significant predictors of ABIM-ECE outcomes. Conclusions In addition to serving as an important learning instrument for endocrine fellowship programs, ESAP-ITE is a robust predictive tool for pass/fail outcomes on the ABIM-ECE.
Collapse
Affiliation(s)
- William B Horton
- Division of Endocrinology and Metabolism, Department of Medicine, University of Virginia School of Medicine, Charlottesville, VA 22908, USA
| | - James T Patrie
- Division of Biostatistics, Department of Public Health Sciences, University of Virginia School of Medicine, Charlottesville, VA 22908, USA
| | - Lauren M Duhigg
- American Board of Internal Medicine, Philadelphia, PA 19106, USA
| | | | - Mark W True
- San Antonio Uniformed Services Health Education Consortium, Brooke Army Medical Center, JBSA Fort Sam Houston, San Antonio, TX 78234, USA
| | - Elaine M Pelley
- Division of Endocrinology, Diabetes, and Metabolism, Department of Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI 53705, USA
| | - Alan C Dalkin
- Division of Endocrinology and Metabolism, Department of Medicine, University of Virginia School of Medicine, Charlottesville, VA 22908, USA
| |
Collapse
|
22
|
Schynoll G, Perog J, Feustel PJ, Smith R. Curriculum Transition From Lecture-Based to Team-Based Learning is Associated With Improved Performance on Internal Medicine In-Training Examination. J Grad Med Educ 2021; 13:691-698. [PMID: 34721799 PMCID: PMC8527942 DOI: 10.4300/jgme-d-20-01164.1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/03/2020] [Revised: 04/05/2021] [Accepted: 06/30/2021] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND Team-based learning (TBL) is an alternative to traditional lectures in graduate medical education, but evidence is scarce regarding its impact on knowledge acquisition and standardized testing performance. OBJECTIVE We examined the association between resident performance on the Internal Medicine In-Training Examination (IM-ITE) and these 2 educational methods. METHODS In 2013, the internal medicine residency program at Albany Medical College transitioned from a lecture-based curriculum to TBL. Residents enrolled in academic years 2011-2012 and 2012-2013 comprised the lecture cohort, and those enrolled in 2015-2016 and 2016-2017 the TBL cohort. Covariates included the type of medical school attended, gender, and United States Medical Licensing Examination Step 2 Clinical Knowledge scores. We performed univariate analysis and multivariable regression to determine the association between covariates and ITE scores. RESULTS Of 120 residents, 60 were in the lecture cohort and 60 in the TBL cohort. The IM-ITE percent correct scores were higher with TBL than lecture (PGY-1 61.0% vs 55.0%, P < .001; PGY-2 69.0% vs 59.7%, P < .001; PGY-3 73.2% vs 61.7%, P < .001). In a multivariable regression analysis of 3 PGYs combined, the transition from lecture to TBL resulted in an increase in IM-ITE Z-score of 0.415 (P < .001), equivalent to 0.415 SD, when including the effects of all covariates. CONCLUSIONS Compared to a lecture-based curriculum, TBL was associated with improved resident medical knowledge acquisition as evidenced by higher IM-ITE scores.
Collapse
Affiliation(s)
- Gerald Schynoll
- All authors are with Albany Medical College
- Gerald Schynoll, MD, MPH, FACP, is Associate Program Director, Internal Medicine Residency, and Associate Professor of Medicine
| | - Justin Perog
- All authors are with Albany Medical College
- Justin Perog, BA, is Team-Based Learning Coordinator, Internal Medicine Residency
| | - Paul J. Feustel
- All authors are with Albany Medical College
- Paul J. Feustel, PhD, is Professor, Department of Neuroscience and Experimental Therapeutics
| | - Raymond Smith
- All authors are with Albany Medical College
- Raymond Smith, MD, FACP, FIDSA, is Program Director, Internal Medicine Residency, and Professor of Medicine
| |
Collapse
|
23
|
Kaell A, Sangwan J, Boryushkina V, Haggerty G. The use of the NEJM knowledge + online platform to supplement traditional pulmonary didactic: a resident-led educational quality improvement project at a community hospital IM GME program. J Community Hosp Intern Med Perspect 2021; 11:425-428. [PMID: 34211642 PMCID: PMC8221144 DOI: 10.1080/20009666.2021.1935595] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Introduction: Many internal medicine residents struggle to prepare for both the ITE and board test. Most existing resources are simply test question banks that are not linked to existing supporting literature from which they can study. Additionally, program directors are unable to track how much time residents are spending or performing on test preparation. We looked to evaluate the benefit of using this online platform to augment our pulmonary didactics and track time and performance on the pulmonary module and ITE pulmonary section. Method: During the month-long live didactic sessions, residents had free access to the pulmonology NEJM K+ platform. A platform-generated post-test was administered with new questions covering the same key elements, including the level of confidence meta-metric. An anonymous feedback survey was collected to assess the residents’ feelings regarding using the NEJM Knowledge+ platform as compared to other prep resources. Results: 44 of 52 residents completed the pre-test. 51/52 completed the month-long didactic sessions and the post-test. Residents’ score improvement from % correct pre-test (M = 46.90, SD = 15.31) to % correct post-test (M = 76.29, SD = 18.49) correlated with levels of mastery (t = 9.60, df = 41, p < .001). The % passing improved from 1/44 (2.3%) pre-test to 35/51 (68.6%) post-test, also correlating with levels of mastery. Accurate confidence correlated with improvement from pre to post test score (r = −51, p = .001). Survey feedback was favorable.
Collapse
Affiliation(s)
- Alan Kaell
- Graduate Medical Education, Mather Hospital Northwell Health, Port Jefferson, New York
| | | | - Varvara Boryushkina
- Internal Medicine Residency, Mather Hospital Northwell Health, Port Jefferson, New York
| | - Greg Haggerty
- Graduate Medical Education, Mather Hospital Northwell Health, Port Jefferson, New York
| |
Collapse
|
24
|
Assessment of Plastic Surgery Residency Applications without United States Medical Licensing Examination Step 1 Scores. Plast Reconstr Surg 2021; 148:219-223. [PMID: 34076626 DOI: 10.1097/prs.0000000000008057] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
SUMMARY The United States Medical Licensing Examination announced the changing of Step 1 score reporting from a three-digit number to pass/fail beginning on January 1, 2022. Plastic surgery residency programs have traditionally used United States Medical Licensing Examination Step 1 scores to compare plastic surgery residency applicants. Without a numerical score, the plastic surgery residency application review process will likely change. This article discusses advantages, disadvantages, and steps forward for residency programs related to the upcoming change. The authors encourage programs to continue to seek innovative methods of objectively and holistically evaluating applications.
Collapse
|
25
|
Hauer KE, Jurich D, Vandergrift J, Lipner RS, McDonald FS, Yamazaki K, Chick D, McAllister K, Holmboe ES. Gender Differences in Milestone Ratings and Medical Knowledge Examination Scores Among Internal Medicine Residents. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:876-884. [PMID: 33711841 DOI: 10.1097/acm.0000000000004040] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE To examine whether there are group differences in milestone ratings submitted by program directors working with clinical competency committees (CCCs) based on gender for internal medicine (IM) residents and whether women and men rated similarly on milestones perform comparably on subsequent in-training and certification examinations. METHOD This national retrospective study examined end-of-year medical knowledge (MK) and patient care (PC) milestone ratings and IM In-Training Examination (IM-ITE) and IM Certification Examination (IM-CE) scores for 2 cohorts (2014-2017, 2015-2018) of U.S. IM residents at ACGME-accredited programs. It included 20,098/21,440 (94%) residents, with 9,424 women (47%) and 10,674 men (53%). Descriptive statistics and differential prediction techniques using hierarchical linear models were performed. RESULTS For MK milestone ratings in PGY-1, men and women showed no statistical difference at a significance level of .01 (P = .02). In PGY-2 and PGY-3, men received statistically higher average MK ratings than women (P = .002 and P < .001, respectively). In contrast, men and women received equivalent average PC ratings in each PGY (P = .47, P = .72, and P = .80, for PGY-1, PGY-2, and PGY-3, respectively). Men slightly outperformed women with similar MK or PC ratings in PGY-1 and PGY-2 on the IM-ITE by about 1.7 and 1.5 percentage points, respectively, after adjusting for covariates. For PGY-3 ratings, women and men with similar milestone ratings performed equivalently on the IM-CE. CONCLUSIONS Milestone ratings were largely similar for women and men. Generally, women and men with similar MK or PC milestone ratings performed similarly on future examinations. Although there were small differences favoring men on earlier examinations, these differences disappeared by the final training year. It is questionable whether these small differences are educationally or clinically meaningful. The findings suggest fair, unbiased milestone ratings generated by program directors and CCCs assessing residents.
Collapse
Affiliation(s)
- Karen E Hauer
- K.E. Hauer is professor, Department of Medicine, University of California, San Francisco, San Francisco, California; ORCID: https://orcid.org/0000-0002-8812-4045
| | - Daniel Jurich
- D. Jurich is manager, psychometrics, National Board of Medical Examiners, Philadelphia, Pennsylvania
| | - Jonathan Vandergrift
- J. Vandergrift is senior research associate, American Board of Internal Medicine, Philadelphia, Pennsylvania
| | - Rebecca S Lipner
- R.S. Lipner is senior vice president for assessment and research, American Board of Internal Medicine, Philadelphia, Pennsylvania
| | - Furman S McDonald
- F.S. McDonald is senior vice president for academic and medical affairs, American Board of Internal Medicine, Philadelphia, Pennsylvania
| | - Kenji Yamazaki
- K. Yamazaki is senior analyst, milestones research and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois
| | - Davoren Chick
- D. Chick is senior vice president for medical education, American College of Physicians, Philadelphia, Pennsylvania
| | - Kevin McAllister
- K. McAllister is assessment officer, National Board of Medical Examiners, Philadelphia, Pennsylvania
| | - Eric S Holmboe
- E.S. Holmboe is chief research, milestone development, and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois
| |
Collapse
|
26
|
Busha ME, McMillen B, Greene J, Gibson K, Milnes C, Ziemkowski P. One Institution's evaluation of family medicine residency applicant data for academic predictors of success. BMC MEDICAL EDUCATION 2021; 21:84. [PMID: 33530993 PMCID: PMC7851804 DOI: 10.1186/s12909-021-02518-w] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/25/2020] [Accepted: 01/08/2021] [Indexed: 05/10/2023]
Abstract
BACKGROUND Family Medicine residencies are navigating recruitment in a changing environment. The consolidation of accreditation for allopathic and osteopathic programs, the high volume of applicants, and the forthcoming transition of the United States Medical Licensing Exam (USMLE) Step 1 to pass/fail reporting all contribute. This retrospective cohort study evaluated which components of a student's academic history best predict readiness for residency. METHODS In 2020, we analyzed applicant data and initial residency data for program graduates at a single residency program between 2013 and 2020. This included undergraduate education characteristics, medical school academic performance, medical school academic problems (including professionalism), STEP exams, location of medical school, and assessments during the first 6 months of residency. Of 110 matriculating residents, assessment data was available for 97 (88%). RESULTS Pre-matriculation USMLE data had a positive correlation with initial American Board of Family Medicine (ABFM) in-training exams. Pre-matriculation exam data did not have a positive correlation with resident assessment across any of the six Accreditation Council for Graduate Medical Education (ACGME) competency domains. A defined cohort of residents with a history of academic struggles during medical school or failure on a USMLE exam performed statistically similarly to residents with no such history on assessments across the six ACGME competency domains. CONCLUSIONS Applicants with a history of academic problems perform similarly in the clinical environment to those without. While a positive correlation between pre-matriculation exams and the ABFM in-training exam was found, this did not extend to clinical assessments across the ACGME competency domains.
Collapse
Affiliation(s)
- Michael E Busha
- Western Michigan University Homer Stryker M.D. School of Medicine, 1000 Oakland Drive, Kalamazoo, MI, 49008, USA.
| | - Brock McMillen
- Indiana University School of Medicine, 1520 North Senate, Indianapolis, IN, 46202, USA
| | - Jeffrey Greene
- Western Michigan University Homer Stryker M.D. School of Medicine, 1000 Oakland Drive, Kalamazoo, MI, 49008, USA
| | - Kristine Gibson
- Western Michigan University Homer Stryker M.D. School of Medicine, 1000 Oakland Drive, Kalamazoo, MI, 49008, USA
| | - Charlotte Milnes
- Western Michigan University Homer Stryker M.D. School of Medicine, 1000 Oakland Drive, Kalamazoo, MI, 49008, USA
| | - Peter Ziemkowski
- Western Michigan University Homer Stryker M.D. School of Medicine, 1000 Oakland Drive, Kalamazoo, MI, 49008, USA
| |
Collapse
|