1
|
Morgan A, Cook M, Christman M, Scott N, Shanks A. Association of Professors of Gynecology and Obstetrics Preparation for Residency Knowledge Assessment scores are more closely associated with first postgraduate year Council on Resident Education in Obstetrics and Gynecology scores than United States Medical Licensing Examination Steps 1 and 2. AJOG GLOBAL REPORTS 2024; 4:100354. [PMID: 38774651 PMCID: PMC11107357 DOI: 10.1016/j.xagr.2024.100354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/24/2024] Open
Affiliation(s)
- Amanda Morgan
- Department of Obstetrics and Gynecology, Indiana University School of Medicine, Indianapolis, IN
| | - Myanna Cook
- Indiana University School of Medicine, Indianapolis, IN
| | - Megan Christman
- Department of Obstetrics and Gynecology, Indiana University School of Medicine, Indianapolis, IN
| | - Nicole Scott
- Department of Obstetrics and Gynecology, Indiana University School of Medicine, Indianapolis, IN
| | - Anthony Shanks
- Department of Obstetrics and Gynecology, Indiana University School of Medicine, Indianapolis, IN
| |
Collapse
|
2
|
Plewa MC, Ledrick DJ, Jenkins K, Orqvist A, McCrea M. Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations? Cureus 2024; 16:e58684. [PMID: 38651085 PMCID: PMC11033967 DOI: 10.7759/cureus.58684] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/21/2024] [Indexed: 04/25/2024] Open
Abstract
PURPOSE The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency. METHODS The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics. RESULTS Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ± 63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively. CONCLUSION USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.
Collapse
Affiliation(s)
- Michael C Plewa
- Emergency Medicine, Mercy Health - St. Vincent Medical Center, Toledo, USA
| | - David J Ledrick
- Emergency Medicine, Mercy Health - St. Vincent Medical Center, Toledo, USA
| | - Kenneth Jenkins
- Emergency Medicine, Ohio University Heritage College of Osteopathic Medicine, Athens, USA
| | - Aaron Orqvist
- Emergency Medicine, Mercy Health - St. Vincent Medical Center, Toledo, USA
| | - Michael McCrea
- Emergency Medicine, Mercy Health - St. Vincent Medical Center, Toledo, USA
| |
Collapse
|
3
|
Hong CX, Russell CB, Southworth EA, Fairchild PS. Use of In-Training Examination Scores as a Fellowship Candidate Evaluation Metric: Time for a Change. UROGYNECOLOGY (PHILADELPHIA, PA.) 2024; 30:394-398. [PMID: 38564624 DOI: 10.1097/spv.0000000000001489] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
ABSTRACT In the field of obstetrics and gynecology (OB/GYN), the Council on Resident Education in Obstetrics and Gynecology (CREOG) administers an annual in-training examination to all OB/GYN residents as a formative educational tool for assessing medical knowledge and promoting self-improvement. Although the CREOG examination is not designed or intended for knowledge certification, many OB/GYN subspecialty fellowship programs request and use CREOG examination scores as a metric to evaluate fellowship candidates. Among the 57 gynecology-based urogynecology fellowship programs, 30 programs (53%) request CREOG examination scores to be submitted by candidates, as of March 2023. Although the use of CREOG examination scores as an evaluation metric may constitute a minor component within the fellowship match process, this practice fundamentally contradicts the intended purpose of the examination as an educational self-assessment. In addition, it introduces the potential for bias in fellowship recruitment, lacks psychometric validity in predicting specialty board examination failure, and shifts the CREOG examination from its original intention as low-stakes self-assessment into a high-stakes examination akin to a certification examination. For these reasons, we call upon the urogynecology community to prioritize the educational mission of the CREOG examination and reconsider the practice of requesting or using CREOG examination scores in the fellowship match progress.
Collapse
Affiliation(s)
- Christopher X Hong
- From the Department of Obstetrics and Gynecology, University of Michigan, Ann Arbor, MI
| | | | | | | |
Collapse
|
4
|
Weaver ML, Carter T, Yamazaki K, Hamstra SJ, Holmboe E, Chaer R, Park YS, Smith BK. The Association of ACGME Milestones With Performance on American Board of Surgery Assessments: A National Investigation of Surgical Trainees. Ann Surg 2024; 279:180-186. [PMID: 37436889 DOI: 10.1097/sla.0000000000005998] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/14/2023]
Abstract
OBJECTIVE To determine the relationship between, and predictive utility of, milestone ratings and subsequent American Board of Surgery (ABS) vascular surgery in-training examination (VSITE), vascular qualifying examination (VQE), and vascular certifying examination (VCE) performance in a national cohort of vascular surgery trainees. BACKGROUND Specialty board certification is an important indicator of physician competence. However, predicting future board certification examination performance during training continues to be challenging. METHODS This is a national longitudinal cohort study examining relational and predictive associations between Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings and performance on VSITE, VQE, and VCE for all vascular surgery trainees from 2015 to 2021. Predictive associations between milestone ratings and VSITE were conducted using cross-classified random-effects regression. Cross-classified random-effects logistic regression was used to identify predictive associations between milestone ratings and VQE and VCE. RESULTS Milestone ratings were obtained for all residents and fellows(n=1,118) from 164 programs during the study period (from July 2015 to June 2021), including 145,959 total trainee assessments. Medical knowledge (MK) and patient care (PC) milestone ratings were strongly predictive of VSITE performance across all postgraduate years (PGYs) of training, with MK ratings demonstrating a slightly stronger predictive association overall (MK coefficient 17.26 to 35.76, β = 0.15 to 0.23). All core competency ratings were predictive of VSITE performance in PGYs 4 and 5. PGY 5 MK was highly predictive of VQE performance [OR 4.73, (95% CI, 3.87-5.78), P <0.001]. PC subcompetencies were also highly predictive of VQE performance in the final year of training [OR 4.14, (95% CI, 3.17-5.41), P <0.001]. All other competencies were also significantly predictive of first-attempt VQE pass with ORs of 1.53 and higher. PGY 4 ICS ratings [OR 4.0, (95% CI, 3.06-5.21), P <0.001] emerged as the strongest predictor of VCE first-attempt pass. Again, all subcompetency ratings remained significant predictors of first-attempt pass on CE with ORs of 1.48 and higher. CONCLUSIONS ACGME Milestone ratings are highly predictive of future VSITE performance, and first-attempt pass achievement on VQE and VCE in a national cohort of surgical trainees.
Collapse
Affiliation(s)
- M Libby Weaver
- Division of Vascular and Endovascular Surgery, University of Virginia, Charlottesville, VA
| | - Taylor Carter
- Department of Surgery, University of North Carolina, Chapel Hill, NC
| | - Kenji Yamazaki
- Accreditation Council for Graduate Medical Education, Chicago, IL
| | - Stanley J Hamstra
- Department of Surgery, University of Toronto, Toronto, Ontario, Canada
- Department of Medical Education, Northwestern University Feinberg School of Medicine, Chicago, IL
| | - Eric Holmboe
- Accreditation Council for Graduate Medical Education, Chicago, IL
| | - Rabih Chaer
- Division of Vascular Surgery, University of Pittsburgh Medical Center, Pittsburgh, PA
| | - Yoon Soo Park
- Massachusetts General Hospital, Harvard Medical School, Boston, MA
| | - Brigitte K Smith
- Division of Vascular Surgery, University of Utah, Salt Lake City, UT
| |
Collapse
|
5
|
Scott NP, Martin TW, Schmidt AM, Shanks AL. Impact of an Online Question Bank on Resident In-Training Exam Performance. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2023; 10:23821205231206221. [PMID: 37822782 PMCID: PMC10563493 DOI: 10.1177/23821205231206221] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Accepted: 06/12/2023] [Indexed: 10/13/2023]
Abstract
OBJECTIVE In-training exams (ITEs) are administered annually to Obstetrics and Gynecology (OBGYN) residents and have been demonstrated to correlate with success on licensing examinations. Our study objective was to determine the impact of a question bank and mock exam on the performance of Council on Resident Education in Obstetrics and Gynecology (CREOG) ITEs. Secondarily, we investigated the correlation between the extent of question bank usage and performance on the exam. METHODS Pre-post intervention study of resident performance on CREOG ITE before and after implementation of the question bank and mock ITE at Indiana University in 2018. Performance was measured as year-to-year improvement in percent correct on ITE exams. Scores were excluded if a resident did not have a prequestion bank score report or if they did not sit for all eligible ITE exams. RESULTS There were 51 OBGYN residents at Indiana University during the study period, with 38 available for analysis (75%). Before implementation, average year-to-year improvement for PGY1-2, PGY2-3 and PGY3-4 classes were 0.60%, 1.0% and -1.6%, respectively. After implementation, all resident classes had significant improvements in ITE scores of 6.6% (P < .01), 9.0% (P < .01), and 7.2% (P < .01), respectively. There was a moderate program-wide correlation between the number of questions completed and the percent improvement on the ITE of R = 0.36 (P = .046). CONCLUSIONS Our study demonstrated that access to a question bank and mock ITE significantly improved CREOG ITE performance of OBGYN residents at Indiana University.
Collapse
Affiliation(s)
- Nicole P. Scott
- Indiana University School of Medicine, Indianapolis, IN, USA
| | | | | | | |
Collapse
|
6
|
Green I, Weaver A, Kircher S, Levy G, Michael Brady R, Flicker AB, Gala RB, Peterson J, Decesare J, Breitkopf D. Impact of Question Bank Use for In-Training Examination Preparation by OBGYN Residents - A Multicenter Study. JOURNAL OF SURGICAL EDUCATION 2022; 79:775-782. [PMID: 35086789 DOI: 10.1016/j.jsurg.2021.12.014] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/23/2021] [Revised: 11/28/2021] [Accepted: 12/28/2021] [Indexed: 06/14/2023]
Abstract
OBJECTIVE To examine the impact of access to and utilization of a commercially available question bank (TrueLearn) for in-training examination (ITE) preparation in Obstetrics and Gynecology (OBGYN). DESIGN This was a retrospective cohort study examining the impact of TrueLearn usage on ITE examination performance outcomes. Produced by the educational arm of the American College of Obstetricians and Gynecologists, the Council on Resident Education in Obstetrics and Gynecology (CREOG) exam is a multiple-choice test given to all residents annually. Residency programs participating in this study provided residency program mean CREOG scores from the year prior (2015), and the first (2016) and second (2017) years of TrueLearn usage. Programs also contributed resident-specific CREOG scores for each resident for 2016 and 2017. This data was combined with each resident's TrueLearn usage data that was provided by TrueLearn with residency program consent. The CREOG scores consisted of the CREOG score standardized to all program years, the CREOG score standardized to the same program year (PGY) and the total percent (%) correct. TrueLearn usage data included number of practice questions completed, number of practice tests taken, average number of days between successive tests, and percent correct of answered practice questions. SETTING OBGYN Residency Training Programs. PARTICIPANTS OBGYN residency programs that purchased and utilized TrueLearn for the 2016 CREOG examination were eligible for participation (n = 14). Ten residency programs participated, which consisted of 212 residents in 2016 and 218 residents in 2017. RESULTS TrueLearn was used by 78.8% (167/212) of the residents in 2016 and 84.9% (185/218) of the residents in 2017. No significant difference was seen in the average CREOG scores available on a per- program level before versus after the first year of implementation either using the CREOG score standardized to all PGYs (mean difference 1.0; p = 0.58) or standardized to the same PGY (mean difference 3.1; p = 0.25). Using resident-level data, there was no significant difference in mean CREOG score standardized to all PGYs between users and non-users of TrueLearn in 2016 (mean, 199.4 vs 196.7; p = 0.41) or 2017 (mean, 198.2 vs 203.4; p = 0.19). The percent of practice questions answered correctly on TrueLearn was positively correlated with the CREOG score standardized to all PGYs (r = 0.47 for 2016 and r = 0.60 for 2017), as well as with the CREOG total percent correct (r = 0.47 for 2016 and r = 0.61 for 2017). Based on a simple linear regression, for every 500 practice questions completed, the CREOG score significantly increased for PGY-2 residents by an average (±SE) of 7.3 ± 2.8 points (p = 0.013); the average increase was 0.7 ± 2.5 (p = 0.79) for PGY-3 residents and 5.8 ± 3.3 points (p = 0.09) for PGY-4 residents. CONCLUSIONS Adoption of an online question bank did not result in higher mean CREOG scores at participating institutions. However, performance on the TrueLearn questions correlated with ITE performance, supporting predictive validity and the use of this question bank as a formative assessment for resident education and exam preparation.
Collapse
Affiliation(s)
- Isabel Green
- Department of Obstetrics and Gynecology, Division of Gynecologic Surgery Mayo Clinic, Rochester, Minnesota.
| | - Amy Weaver
- Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota
| | - Samantha Kircher
- Department of Obstetrics and Gynecology, MetroHealth Medical Center, Cleveland, Ohio
| | - Gary Levy
- Uniformed Services University, Bethesda, Maryland, Tripler Army Medical Center, Honolulu, Hawaii
| | - Robert Michael Brady
- Creighton University School of Medicine - Phoenix, Department of Obstetrics and Gynecology, Phoenix, Arizona
| | - Amanda B Flicker
- Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania
| | - Rajiv B Gala
- Ochsner Health - Baptist, McFarland Medical Plaza, New Orleans, Louisiana
| | - Joseph Peterson
- Ascension Sacred Heart Women's Care Center, Pensacola, Florida
| | - Julie Decesare
- West Florida Medical Group - Obstetrics & Gynecology, Pensacola, Florida
| | - Daniel Breitkopf
- Department of Obstetrics and Gynecology, Division of Gynecologic Surgery Mayo Clinic, Rochester, Minnesota
| |
Collapse
|
7
|
Ottum S, Chao C, Tamakuwala S, Dean J, Shafi A, Kramer KJ, Kaur S, Recanati MA. Can ACGME Milestones predict surgical specialty board passage: an example in Obstetrics and Gynecology. CLIN EXP OBSTET GYN 2021; 48:1048-1055. [PMID: 34720368 PMCID: PMC8555913 DOI: 10.31083/j.ceog4805168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Background: Multiple tools including Accreditation Council for Graduate Medical Education (ACGME) standardized milestones can be utilized to assess trainee and residency program performance. However, little is known regarding the objective validation of these tools in predicting written board passage. Methods: In this retrospective study, data was gathered on n = 45 Wayne State University Obstetrics and Gynecology program graduates over the five-year period ending July 2018. United States Medical Licensing Examination (USMLE) scores, Council on Resident Education in Obstetrics and Gynecology (CREOG) in-training scores and ACGME milestones were used to predict American Board of Obstetrics and Gynecology (ABOG) board passage success on first attempt. Significance was set at p < 0.05. Results: Written board passage was associated with average CREOGs (p = 0.01) and milestones (p = 0.008) while USMLE1 was not significantly associated (p = 0.055). USMLE1 <217 (Positive predictive value (PPV) = 96%). CREOGs <197 (PPV = 100%) and milestones <3.25 (PPV = 100%), particularly practice-based learning and systems-based practice milestones were most strongly correlated with board failure. Using a combination of these two milestones, it is possible to correctly predict board passage using our model (PPV = 86%). Discussion: This study is the first validating the utility of milestones in a surgical specialty by demonstrating their ability to predict board passage. Residents with CREOGs or milestones below thresholds are at risk for board failure and may warrant early intervention.
Collapse
Affiliation(s)
- Sarah Ottum
- Department of Obstetrics and Gynecology, University of Cincinnati, Cincinnati, OH 45267, USA
| | - Conrad Chao
- Department of Maternal-Fetal Medicine, University of New Mexico School of Medicine, Albuquerque, NM 87131, USA
| | - Sejal Tamakuwala
- Department of Obstetrics and Gynecology, Emory University, Atlanta, GA 30322, USA
| | - Joshua Dean
- Department of Obstetrics and Gynecology, Beaumont, Royal Oak, MI 48073, USA
| | - Adib Shafi
- Department of Computer Sciences, Wayne State University, Detroit, MI 48201, USA
| | - Katherine Jennifer Kramer
- Department of Obstetrics and Gynecology, St. Vincent's Catholic Medical Centers, New York, NY 10011, USA
| | - Satinder Kaur
- Department of Obstetrics and Gynecology, Wayne State University, Detroit, MI 48201, USA
| | - Maurice-Andre Recanati
- Department of Obstetrics and Gynecology and NIH-Women's Reproductive Health Research Scholar, Wayne State University, Detroit, MI 48201, USA
| |
Collapse
|
8
|
Tsai S, Sun M, Asbury ML, Weber JM, Truong T, Deans E. Novel Spaced Repetition Flashcard System for the In-training Examination for Obstetrics and Gynecology. MEDICAL SCIENCE EDUCATOR 2021; 31:1393-1399. [PMID: 34457982 PMCID: PMC8368326 DOI: 10.1007/s40670-021-01320-z] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/11/2021] [Indexed: 06/13/2023]
Abstract
INTRODUCTION Electronic flashcards allow repeated information exposure over time along with active recall. It is increasingly used for self-study by medical students but remains poorly implemented for graduate medical education. The primary goal of this study was to determine whether a flashcard system enhances preparation for the in-training examination in obstetrics and gynecology (ob-gyn) conducted by the Council on Resident Education in Obstetrics and Gynecology (CREOG). METHODS Ob-gyn residents at Duke University were included in this study. A total of 883 electronic flashcards were created and distributed. CREOG scores and flashcard usage statistics, generated internally by interacting with the electronic flashcard system, were collected after the 2019 exam. The primary outcome was study aid usage and satisfaction. The secondary outcome was the impact of flashcard usage on CREOG exam scores. RESULTS Of the 32 residents, 31 (97%) participated in this study. Eighteen (58%) residents used the study's flashcards with a median of 276 flashcards studied over a median of 3.7 h. All of the flashcard users found the study aid helpful, and all would recommend them to another ob-gyn resident. Using the flashcards to study for the 2019 CREOG exam appeared to correlate with improvement in scores from 2018 to 2019, but did not achieve statistical significance after adjusting for post-graduate year (beta coefficient = 10.5; 95% confidence interval = - 0.60,21.7; p = 0.06). DISCUSSION This flashcard resource was well received by ob-gyn residents for in-training examination preparation, though it was not significantly correlated with improvement in CREOG scores after adjusting for post-graduate year.
Collapse
Affiliation(s)
- Shelun Tsai
- Department of Obstetrics and Gynecology, Duke University Medical Center, Durham, NC USA
| | - Michael Sun
- Department of Psychiatry, Duke University Medical Center, Durham, NC USA
| | - Melinda L. Asbury
- Department of Psychiatry, University of North Carolina School of Medicine, Chapel Hill, NC USA
- Department of Psychiatry, Duke University Medical Center, Durham, NC USA
| | - Jeremy M. Weber
- Department of Biostatistics and Bioinformatics, Duke University Medical Center, Durham, NC USA
| | - Tracy Truong
- Department of Biostatistics and Bioinformatics, Duke University Medical Center, Durham, NC USA
| | - Elizabeth Deans
- Department of Obstetrics and Gynecology, Duke University Medical Center, Durham, NC USA
| |
Collapse
|
9
|
Ost SR, Wells D, Goedecke PJ, Tolley EA, Kleinman M, Thompson NS. Relationship Between Standardized Test Scores and Board Certification Exams in a Combined Internal Medicine/Pediatrics Residency Program. Cureus 2021; 13:e13567. [PMID: 33815979 PMCID: PMC8008765 DOI: 10.7759/cureus.13567] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/04/2022] Open
Abstract
Background Combined Internal Medicine/Pediatrics (Med/Peds) residencies rely on categorical program data to predict pass rates for the American Board of Internal Medicine Certifying Exam (ABIM-CE) and the American Board of Pediatrics Certifying Exam (ABP-CE). There is insufficient literature describing what best predicts a Med/Peds resident passing board exams. In this study, we aimed to determine how standardized test scores predict performance on ABIM-CE and ABP-CE for Med/Peds residents. Methodology We analyzed prior exam scores for 91/96 (95%) residents in a Med/Peds program from 2008 to 2017. Scores from the United States Medical Licensing Examination (USMLE) Steps 1 and 2 Clinical Knowledge (CK) and In-Training Exams in Internal Medicine (ITE-IM) and Pediatrics (ITE-P) were analyzed with the corresponding ABIM-CE and ABP-CE first-time scores. Linear and logistic regression were applied to predict board scores/passage. Results USMLE 1 and 2 CK, ITE-IM, and ITE-P scores had a linear relationship with both ABIM-CE and ABP-CE scores. In the linear regression, adjusted R2 values showed low-to-moderate predictive ability (R2 = 0.11-0.35), with the highest predictor of ABIM-CE and ABP-CE being USMLE Step 1 (0.35) and Postgraduate Year 1 (PGY-1) ITE-IM (0.33), respectively. Logistic regression showed odds ratios of passing board certifications ranging from 1.05 to 1.53 per point increase on the prior exam score. The PGY-3 ITE-IM was the best predictor of passing both certifying exams. Conclusions In one Med/Peds program, USMLE Steps 1 and 2 and all ITE-IM and ITE-P scores predicted certifying exam scores and passage. This provides Med/Peds-specific data to allow individualized resident counseling and guide programmatic improvements targeted to board performance.
Collapse
Affiliation(s)
- Shelley R Ost
- General Internal Medicine, University of Tennessee Health Science Center College of Medicine, Memphis, USA
| | - Daniel Wells
- General Pediatrics, University of Tennessee Health Science Center College of Medicine, Memphis, USA
| | - Patricia J Goedecke
- Preventive Medicine, University of Tennessee Health Science Center College of Medicine, Memphis, USA
| | - Elizabeth A Tolley
- Preventive Medicine, University of Tennessee Health Science Center College of Medicine, Memphis, USA
| | - Michael Kleinman
- General Internal Medicine, University of Tennessee Health Science Center College of Medicine, Memphis, USA
| | - Natascha S Thompson
- General Internal Medicine, University of Tennessee Health Science Center College of Medicine, Memphis, USA
| |
Collapse
|
10
|
McCrary HC, Colbert-Getz JM, Poss WB, Smith BK. A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores. J Grad Med Educ 2021; 13:43-57. [PMID: 33680301 PMCID: PMC7901636 DOI: 10.4300/jgme-d-20-00111.1] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/07/2020] [Revised: 05/04/2020] [Accepted: 10/23/2020] [Indexed: 11/06/2022] Open
Abstract
BACKGROUND In-training examinations (ITEs) are intended for low-stakes, formative assessment of residents' knowledge, but are increasingly used for high-stake purposes, such as to predict board examination failures. OBJECTIVE The aim of this review was to investigate the relationship between performance on ITEs and board examination performance across medical specialties. METHODS A search of the literature for studies assessing the strength of the relationship between ITE and board examination performance from January 2000 to March 2019 was completed. Results were categorized based on the type of statistical analysis used to determine the relationship between ITE performance and board examination performance. RESULTS Of 1407 articles initially identified, 89 articles underwent full-text review, and 32 articles were included in this review. There was a moderate-strong relationship between ITE and board examination performance, and ITE scores significantly predict board examination scores for the majority of studies. Performing well on an ITE predicts a passing outcome for the board examination, but there is less evidence that performing poorly on an ITE will result in failing the associated specialty board examination. CONCLUSIONS There is a moderate to strong correlation between ITE performance and subsequent performance on board examinations. That the predictive value for passing the board examination is stronger than the predictive value for failing calls into question the "common wisdom" that ITE scores can be used to identify "at risk" residents. The graduate medical education community should continue to exercise caution and restraint in using ITE scores for moderate to high-stakes decisions.
Collapse
|
11
|
Dabbagh A, Elyassi H, Sabouri AS, Vahidshahi K, Ziaee SAM. The Role of Integrative Educational Intervention Package (Monthly ITE, Mentoring, Mocked OSCE) in Improving Successfulness for Anesthesiology Residents in the National Board Exam. Anesth Pain Med 2020; 10:e98566. [PMID: 32547933 PMCID: PMC7260396 DOI: 10.5812/aapm.98566] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2019] [Revised: 11/12/2019] [Accepted: 11/14/2019] [Indexed: 02/05/2023] Open
Abstract
BACKGROUND National Board of Anesthesiology (NBA) pass rate is an important and critical step in clinical residency programs. OBJECTIVES This study was designed to assess the relationship between an integrative educational intervention (IEI) and the relative annual pass rate (RAPR). RAPR is defined as ratio of NBA pass rate of Shahid Beheshti University of Medical Sciences (SBMU) to the NBA pass rate of all the anesthesiology residency programs across Iran. METHODS In a descriptive-analytic retrospective study from 2012 to 2019, RAPR was calculated. IEI was implanted in the latter 4years period of this time interval includes: (1) individualized mentorship for residents by faculty members; (2) monthly in-training examination (ITE) in written; and (3) periodical mocked OSCE exam. Spearman's correlation coefficient was used to assess correlation between integrative educational intervention and RAPR results. P value less than 0.05 was considered statistically significant. RESULTS There was a statistically significant relationship between "integrative educational intervention program" and the RAPR results: Spearman's correlation coefficient = 0.655 (P value = 0.039). CONCLUSIONS The IEI package of Anesthesiology Department, SBMU showed a significant relationship with improvements in successfulness for anesthesiology residents in the National Board Exam (RAPR trend). More prolonged studies could prevail further aspects of these interventions.
Collapse
Affiliation(s)
- Ali Dabbagh
- Anesthesiology Department, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
- Anesthesiology Research Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran
- Corresponding Author: Anesthesiology Research Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Hedayatollah Elyassi
- Anesthesiology Department, Anesthesiology Research Center, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - A. Sassan Sabouri
- Anesthesiology Department, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
- Anesthesiology Department, Massachusetts General Hospital, Harvard Medical School, Boston, United States
| | - Kourosh Vahidshahi
- Pediatric Cardiology Department, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | | | | |
Collapse
|
12
|
Chern I, Antalan C, Aeby T, Hiraoka M. The Effect of a Targeted Educational Activity on Obstetrics and Gynecology Resident In-Training Examination Scores. HAWAI'I JOURNAL OF MEDICINE & PUBLIC HEALTH : A JOURNAL OF ASIA PACIFIC MEDICINE & PUBLIC HEALTH 2018; 77:161-165. [PMID: 30009094 PMCID: PMC6037829] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
The objective of this study was to compare the examination scores before and after implementation of a study program based on high yield topics on the Council of Resident Education in Obstetrics and Gynecology (CREOG) In-Training Examination. This prospective cohort study compared scores from academic years 2012 to 2014 of University of Hawai'i obstetrics and gynecology residents who participated in a directed study program based on selected high yield topics from the CREOG Test Item Summary Booklet. Topics were considered high yield if more than 75% of the program residents answered the topic questions incorrectly during the immediately preceding CREOG In-Training Examination administration. Residents were assigned topics to research and present at monthly teaching sessions. The presentations were made accessible in a wiki website. The intervention was initiated in 2012 and discontinued in 2013. The primary outcome was the difference among CREOG In-Training Examination scores before the study program, during the study program, and after the study program was discontinued. Only scores of residents who sat for all three exams were included. Eleven residents were present during the duration of the study period and sat for all three CREOG examinations. During the year of the educational activity, paired individual resident CREOG exams scores increased significantly from the 2012 CREOG administration (mean = 194.7) to the 2013 CREOG administration (mean = 208.2). These findings demonstrate that the CREOG Test Item Summary Booklet and the wiki platform can be used to effectively direct educational efforts resulting in improvements in CREOG examination performance.
Collapse
Affiliation(s)
- Ingrid Chern
- Department of Obstetrics, Gynecology and Women's Health, John A. Burns School of Medicine, University of Hawai'i, Honolulu, HI
| | - Clyner Antalan
- Department of Obstetrics, Gynecology and Women's Health, John A. Burns School of Medicine, University of Hawai'i, Honolulu, HI
| | - Tod Aeby
- Department of Obstetrics, Gynecology and Women's Health, John A. Burns School of Medicine, University of Hawai'i, Honolulu, HI
| | - Mark Hiraoka
- Department of Obstetrics, Gynecology and Women's Health, John A. Burns School of Medicine, University of Hawai'i, Honolulu, HI
| |
Collapse
|