Orthopaedic in-training examination scores: a correlation with USMLE results

Do the selection criteria of internal medicine residency program predict resident performance?

BACKGROUND

Well-performing physician reflects the success of the residency program in selecting the best candidates for training. This study aimed to evaluate the selection criteria, mainly the United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) results and applicants' status as international or locally trained applicants, used by the medical education department and the internal medicine residency program in Hamad Medical Corporation in Qatar to predict the residents' performance during their training.

METHODS

A retrospective chart review was performed for three batches of graduates who started residency training in 2011, 2012, and 2013. Each group completed 4 years of training. The USMLE Step 2 CK status of the applicant, in-training exam (ITE) scores, formative evaluation scores, Arab Board written and clinical exams pass rate, and other indicators were analyzed. Statistical analysis included chi squares and independent t-test to identify associations. Multivariable analyses were conducted using logistic and linear regressions to test for adjusted associations.

RESULTS

The study included 118 (81 international/37 locally trained applicants) internal medicine residents. The ITE score correlated positively with the USMLE Step 2 CK score (r = 0.621, r = 0.587, r = 0.576, r = 0.571, p < 0.001) over the 4 years of training and among the international compared with locally trained applicants (p < 0.001). The rate of passing part 1 and 2 written exam of the Arab Board was higher in international than in local applicants, whereas clinical Arab Board exam and formative evaluation were not associated with any criteria.

CONCLUSIONS

Higher USMLE Step 2 CK score correlated with better performance on ITE but not with other performance indicators, whereas international applicants did better in both ITE and Arab Board written exam than local applicants. These variables may provide reasonable predictors of well-performing physicians.

Identifying Best Practices for and Utilities of the Pharmacy Curriculum Outcome Assessment Examination

Objective. A review was conducted to determine implementation strategies, utilities, score interpretation, and limitations of the Pharmacy Curriculum Outcome Assessment (PCOA) examination. Methods. Articles were identified through the PubMed and American Journal of Pharmaceutical Education, and International Pharmaceutical Abstracts databases using the following terms: "Pharmacy Curriculum Outcomes Assessment," "pharmacy comprehensive examination," and "curricular assessment." Studies containing information regarding implementation, utility, and predictive values for US student pharmacists, curricula, and/or PGY1/PGY2 residents were included. Publications from the Academic Medicine Journal, the Accreditation Council for Pharmacy Education (ACPE), and the American Association of Colleges of Pharmacy (ACCP) were included for background information and comparison of predictive utilities of comprehensive examinations in medicine. Results. Ten PCOA and nine residency-related publications were identified. Based on published information, the PCOA may be best used as an additional tool to identify knowledge gaps for third-year student pharmacists. Conclusion. Administering the PCOA to students after they have completed their didactic coursework may yield scores that reflect student knowledge. Predictive utility regarding the North American Pharmacy Licensure Examination (NAPLEX) and potential applications is limited, and more research is required to determine ways to use the PCOA.

Analysis of the Korean Orthopedic In-Training Examination: The Hip and Pelvis Section

Purpose

The purpose of this study was to analyze the questions in the hip and pelvis section of the Korean Orthopaedic In-Training Examination (KOITE).

Materials and Methods

We analyzed all KOITE questions pertaining to hip and pelvis surgery between 2010 and 2014. A thorough analysis of the contents was performed after categorizing as tested topics, imaging modalities used, taxonomic classification, and recommended references. We also analyzed the scores of the hip and pelvis section of the KOITE.

Results

Seventy-five of five-hundred questions (weight, 15.0%) were related to the hip and pelvis. Trauma including fracture and dislocation (26/75, 34.7%) was asked more commonly than disease and basics. The description-only questions (65/75, 86.7%) were the most frequently asked. According to taxonomic classification, taxonomy 3 (decision; 39/75, 52.0%) was most frequently asked. Campbell's Operative Orthopedics (52/75, 69.3%) was the reference that covered most of the questions.

Conclusion

This analysis of the hip and pelvis section of KOITE could be used for resident training programs in teaching hospitals.

Selection for Surgical Training: An Evidence-Based Review

PURPOSE

The predictive relationship between candidate selection criteria for surgical training programs and future performance during and at the completion of training has been investigated for several surgical specialties, however there is no interspecialty agreement regarding which selection criteria should be used. Better understanding the predictive reliability between factors at selection and future performance may help to optimize the process and lead to greater standardization of the surgical selection process.

METHODS

PubMed and Ovid MEDLINE databases were searched. Over 560 potentially relevant publications were identified using the search strategy and screened using the Cochrane Collaboration Data Extraction and Assessment Template.

RESULTS

57 studies met the inclusion criteria. Several selection criteria used in the traditional selection demonstrated inconsistent correlation with subsequent performance during and at the end of surgical training. The following selection criteria, however, demonstrated good predictive relationships with subsequent resident performance: USMLE examination scores, Letters of Recommendation (LOR) including the Medical Student Performance Evaluation (MSPE), academic performance during clinical clerkships, the interview process, displaying excellence in extracurricular activities, and the use of unadjusted rank lists.

CONCLUSIONS

This systematic review supports that the current selection process needs to be further evaluated and improved. Multicenter studies using standardized outcome measures of success are now required to improve the reliability of the selection process to select the best trainees.

Does Residency Selection Criteria Predict Performance in Orthopaedic Surgery Residency?

BACKGROUND

More than 1000 candidates applied for orthopaedic residency positions in 2014, and the competition is intense; approximately one-third of the candidates failed to secure a position in the match. However, the criteria used in the selection process often are subjective and studies have differed in terms of which criteria predict either objective measures or subjective ratings of resident performance by faculty.

QUESTIONS/PURPOSES

Do preresidency selection factors serve as predictors of success in residency? Specifically, we asked which preresidency selection factors are associated or correlated with (1) objective measures of resident knowledge and performance; and (2) subjective ratings by faculty.

METHODS

Charts of 60 orthopaedic residents from our institution were reviewed. Preresidency selection criteria examined included United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, Medical College Admission Test (MCAT) scores, number of clinical clerkship honors, number of letters of recommendation, number of away rotations, Alpha Omega Alpha (AOA) honor medical society membership, fourth-year subinternship at our institution, and number of publications. Resident performance was assessed using objective measures including American Board of Orthopaedic Surgery (ABOS) Part I scores and Orthopaedics In-Training Exam (OITE) scores and subjective ratings by faculty including global evaluation scores and faculty rankings of residents. We tested associations between preresidency criteria and the subsequent objective and subjective metrics using linear correlation analysis and Mann-Whitney tests when appropriate.

RESULTS

Objective measures of resident performance namely, ABOS Part I scores, had a moderate linear correlation with the USMLE Step 2 scores (r = 0.55, p < 0.001) and number of clinical honors received in medical school (r = 0.45, p < 0.001). OITE scores had a weak linear correlation with the number of clinical honors (r = 0.35, p = 0.009) and USMLE Step 2 scores (r = 0.29, p = 0.02). With regards to subjective outcomes, AOA membership was associated with higher scores on the global evaluation (p = 0.005). AOA membership also correlated with higher global evaluation scores (r = 0.60, p = 0.005) with the strongest correlation existing between AOA membership and the "interpersonal and communication skills" subsection of the global evaluations.

CONCLUSIONS

We found that USMLE Step 2, number of honors in medical school clerkships, and AOA membership demonstrated the strongest correlations with resident performance. Our goal in analyzing these data was to provide residency programs at large a sense of which criteria may be "high yield" in ranking applicants by analyzing data from within our own pool of residents. Similar studies across a broader scope of programs are warranted to confirm applicability of our findings. The continually emerging complexities of the field of orthopaedic surgery lend increasing importance to future work on the appropriate selection and training of orthopaedic residents.

The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination

PURPOSE

To explore the relationship between United States Medical Licensing Examination (USMLE) Step 1 scores, yearly in-service training exam (ITE) scores, and passing the American Board of Internal Medicine certifying examination (ABIM-CE).

METHOD

The authors conducted a retrospective database review of internal medicine residents from the Medical College of Wisconsin from 2004 through 2012. Residents' USMLE Step 1, ITE, and ABIM-CE scores were extracted. Pearson rho, chi-square, and logistic regression were used to determine whether relationships existed between the scores and if Step 1 and ITE scores correlate with passing the ABIM-CE.

RESULTS

There were 241 residents, who participated in 728 annual ITEs. There were Step 1 scores for 195 (81%) residents and ABIM-CE scores for 183 (76%). Step 1 and ABIM-CE scores had a modest correlation (rho: 0.59), as did ITE and ABIM-CE scores (rho: 0.48-0.67). Failing Step 1 or being in the bottom ITE quartile during any year of testing markedly increased likelihood of failing the boards (Step 1: relative risk [RR]: 2.4; 95% CI: 1.0-5.9; first-year residents' RR: 1.3; 95% CI: 1.0-1.6; second-year residents' RR: 1.3; 95% CI: 1.1-1.5; third-year residents' RR: 1.3; 95% CI: 1.1-1.5).

CONCLUSIONS

USMLE Step 1 and ITE scores have a modest correlation with board scores. Failing Step 1 or scoring in the bottom quartile of the ITE increased the risk of failing the boards. What effective intervention, if any, program directors may use with at-risk residents is a question deserving further research.

It was the night before the interview: perceptions of resident applicants about the preinterview reception

OBJECTIVE

Hosting a reception for prospective interns the evening before the interview has become a well-established expectation. It is thought that these initial impressions significantly influence the ranking process. Despite these well-held beliefs, there has been a paucity of studies exploring the preinterview reception.

DESIGN

A survey tool was created and piloted to ensure validity. The survey was then administered to a fourth-year class of allopathic medical students immediately after interviews but before Match Day.

SETTING

A university, teaching hospital.

PARTICIPANTS

Fourth-year allopathic medical students.

RESULTS

The response rate was 100% (n = 69). Ninety-six percent of programs hosted an event. Although these events were minimally stressful (86%), the same percent felt that not attending would limit their knowledge of the program, and 66% felt that it would negatively affect their application. Forty percent believe this event to be extremely important to residency programs in selecting interns. Ninety-five percent are attended by residents only, and approximately half were at a casual restaurant. Most applicants (97%) never paid for their own meal, and 69% felt that if they did, it would leave a negative impression of the program.

CONCLUSIONS

Candidates believe the preinterview reception is important in the selection process, that failing to attend would negatively affect their application, and provides insight about the program. Alcohol is often provided but rarely has a negative effect. Applicants prefer an informal setting with unfettered interactions with the residents.

Associations between residency selection strategies and doctor performance: a meta-analysis

OBJECTIVES

The purpose of this study was to use meta-analysis to establish which of the information available to the resident selection committee is associated with resident or doctor performance.

METHODS

Multiple electronic databases were searched to 4 September 2012. Two reviewers independently selected studies that met the present inclusion criteria and extracted data in duplicate; disagreement was resolved by consensus. Risk for bias was assessed using a customised bias assessment tool. Measures of association were converted to a common effect size (Hedges' g). Meta-analysis was performed using the random-effects model for each selection strategy and all outcomes without pooling. Sensitivity analysis for each selection strategy-outcome pair was performed with pooling of effect size.

RESULTS

Eighty studies involving a total of 41 704 participants were included in the meta-analysis. Seventeen different selection strategies and 17 outcomes were assessed across these studies. The strongest positive associations referred to examination-based selection strategies, such as the US Medical Licensing Examination (USMLE) Step 1, and examination-based outcomes, such as scores on in-training examinations. Moderate positive associations were present for medical school marks and both examination-based and subjective outcomes. Minimal or no associations were seen for the selection tools represented by interviews, reference letters and deans' letters.

CONCLUSIONS

Standardised examination performance and medical school grades show the strongest associations with current measures of doctor performance. Deans' letters, reference letters and interviews all show a lower than expected strength of association given the relative value often assigned to them during resident doctor selection. Objective selection strategies are potentially the most useful to residency selection committees based on current evaluative methods. However, reports in the literature of validated long-term doctor performance outcomes are scant.

Academic and professional career outcomes of medical school graduates who failed USMLE Step 1 on the first attempt

This study sought to determine the academic and professional outcomes of medical school graduates who failed the United States Licensing Examination Step 1 on the first attempt. This retrospective cohort study was based on pooled data from 2,003 graduates of six Midwestern medical schools in the classes of 1997-2002. Demographic, academic, and career characteristics of graduates who failed Step 1 on the first attempt were compared to graduates who initially passed. Fifty medical school graduates (2.5 %) initially failed Step 1. Compared to graduates who initially passed Step 1, a higher proportion of graduates who initially failed Step 1 became primary care physicians (26/49 [53 %] vs. 766/1,870 [40.9 %]), were more likely at graduation to report intent to practice in underserved areas (28/50 [56 %] vs. 419/1,939 [ 21.6 %]), and more likely to take 5 or more years to graduate (11/50 [22.0 %] vs. 79/1,953 [4.0 %]). The relative risk of first attempt Step 1 failure for medical school graduates was 13.4 for African Americans, 7.4 for Latinos, 3.6 for matriculants >22 years of age, 3.2 for women, and 2.3 for first generation college graduates. The relative risk of not being specialty board certified for those graduates who initially failed Step 1 was 2.2. Our observations regarding characteristics of graduates in our study cohort who initially failed Step 1 can inform efforts by medical schools to identify and assist students who are at particular risk of failing Step 1.

How should unmatched orthopaedic surgery applicants proceed?

BACKGROUND

Obtaining an orthopaedic surgery residency is competitive. Advisors must understand what factors may help unmatched candidates reapply successfully.

QUESTIONS/PURPOSES

We determined (1) the attitude of leaders of orthopaedic surgery residency programs toward interviewing unmatched students; (2) whether a surgical internship or a research year is preferred in considering reapplicants; (3) the importance of United States Medical Licensing Examination (USMLE) scores, recommendations, and Alpha Omega Alpha (AOA) membership; and (4) whether academic and nonacademic programs evaluate reapplicants differently.

METHODS

We sent an anonymous 19-question survey to 151 Accreditation Council for Graduate Medical Education (ACGME)-accredited orthopaedic surgery residency programs in five waves, 1 week apart (December 5, 2009-January 5, 2010). Investigators were blinded to the respondents' identities.

RESULTS

Ninety-one of the 151 programs (60%) responded. Sixty-eight of the 91 programs (75%) stated they rarely accept unmatched applicants. Sixty-eight programs (75%) agreed an unmatched applicant should do a surgery internship for 1 year. Of the 36 programs that recommended a research year, 32 were academic programs. Academic programs were more likely than nonacademic programs to view as important new recommendations (85% versus 67%), minimum scores of 220 on Step I (67% versus 49%) and Step II (64% versus 36%), and AOA membership (85% versus 67%).

CONCLUSIONS

By completing a surgical internship, unmatched students may increase their chances of matching. Students considering academic programs should ensure their academic record meets certain benchmarks and may consider a research year but risk limiting their acceptance to academic programs.

A six year head-to-head comparison of osteopathic and allopathic applicants to a university-based, allopathic general surgery residency

OBJECTIVE

The number of osteopathic physicians is increasing as is the number applying to allopathic general surgery residency programs. A lack of knowledge of osteopathic schooling leads to a potential applicant bias in favor of allopathic applicants, but the 2 groups have not been compared head to head.

DESIGN

Applications over a 6-year period to an allopathic general surgery residency program were reviewed. Demographics, examination scores, employment, education, and research experience were catalogued into a database. Allopathic applicants were compared with osteopathic applicants utilizing statistical analysis.

SETTING

A university teaching hospital.

PARTICIPANTS

Allopathic and osteopathic applicants to an allopathic general surgery residency program.

RESULTS

A total of 1290 applications were reviewed; 1155 allopathic and 135 osteopathic applications. Other than race, the 2 cohorts are similar in age, gender, and citizenship. The groups are not significantly different with regard to the number of letter of recommendations, volunteer activities, scholarly works, and advanced degrees. Graduates of both proceed directly to residency. A significantly higher percentage of allopathic graduates reported their United States medical licensing examination (USMLE) scores, yet when osteopaths released their USMLE transcript, they scored significantly higher on the USMLE Step 1 examination and required fewer attempts to pass. These differences do not apply to the USMLE Step 2 examination.

CONCLUSIONS

No single screening tool exists for selecting a successful general surgery resident. We are seeing increased numbers of osteopathic applicants. Many criteria used to evaluate applicants do not apply to osteopathic applicants, but our comparison of common selection variables on the Electronic Residency Application Service (ERAS) application did not demonstrate an overall difference. While our analysis demonstrated a statistically higher USMLE Step 1 score by osteopathic applicants, they may only self-report favorable data.

The use of a surgery-specific written examination in the selection process of surgical residents

OBJECTIVE

Selection of surgical residents is a difficult task, and program directors are interested in identifying the best candidates. Among the qualities being sought after is the ability to acquire surgical knowledge, and eventually do well on their board examinations. During the interview process, many programs use results from the United States Medical Licensing Exam (USMLE) to identify residents they think will do well academically. The purpose of this study was to evaluate a different method of identifying such residents, through the use of a surgery-specific written exam (SSWE).

DESIGN

A retrospective review of residents in our program between 2004 and 2012 was done. A 50-question SSWE was designed and administered to candidates on the day of their interview. Scores on the SSWE and the USMLE were compared with results on the American Board of Surgery In-Training Exam (ABSITE). Correlation coefficients were calculated and compared.

SETTING

Community based General Surgery residency program.

PARTICIPANTS

Resident applicants.

RESULTS

Forty-three residents had scores available from the SSWE, USMLE Part 1 (USMLE-1), and Part 2 (USMLE-2). There were ABSITE scores available for 38 in postgraduate year (PGY) 1. USMLE-1 had a statistically significant correlation (r = 0.327, p = 0.045) with the ABSITE score in PGY-1 (ABSITE-1), while with USMLE-2 had slightly less correlation (r = 0.314, p = 0.055) with ABSITE-1. However, the SSWE had a much stronger correlation (r = 0.656, p < 0.001) than either of them.

CONCLUSIONS

An SSWE is a good method to identify residents who will later do well on the ABSITE. It is a better method than using the more general USMLE. Since the ABSITE has been shown to correlate with performance on board examinations, residency programs interested in identifying candidates that will do well on their board examinations, should consider incorporating an SSWE into their application process.

Use of an integrated, anatomic-based, orthopaedic resident education curriculum: a 5-year retrospective review of its impact on orthopaedic in-training examination scores

INTRODUCTION

Experts have called for a comprehensive didactic curriculum in orthopaedic residency training. This study examined the effects of an anatomic-based, integrated conference program on annual Orthopaedic In-Training Examination (OITE) scores at a single orthopaedic residency program.

METHODS

We implemented a new, integrated, anatomic-based curriculum in January 2005. Differences between scores were analyzed by postgraduate year (PGY) of training. OITE scores (percentile ranking and raw scores) of year 1 (PGY-2) through year 4 (PGY-5) residents exposed to the curriculum (2005-2009) were compared to prior PGY-2 through PGY-5 residents (2000-2004) who had experienced the previous unstructured curriculum. To evaluate for cohort effects, United States Medical Licensing Examination (USMLE) Step I scores for these 2 groups were also compared.

RESULTS

Eight residents were exposed to the new conference program and 8 to the prior conference program. All residents' percentile rankings improved after exposure to the curriculum, although improvement was not statistically significant for all participants. The most dramatic improvements in OITE scores were seen for PGY-4 and PGY-5 residents, which improved from 65th to 91st percentile (P  =  .03) and from 66th to 91st percentile (P  =  .06), respectively. There were no differences between the cohorts in USMLE Step I scores.

DISCUSSION

Initiation of an integrated, anatomic-based, resident conference program had a positive impact on resident performance on the OITE.

Performance in the Duke-Elder ophthalmology undergraduate prize examination and future careers in ophthalmology

AIMS

Cognitive factors (eg, academic achievement) have had a significant role in selecting postgraduate surgical trainees in the past. This project sought to determine the role of a national undergraduate ophthalmology prize examination (Duke-Elder examination) in the selection of postgraduate ophthalmology trainees. This would also serve as a quality assurance exercise for the assessment, in which the ultimate aim is to encourage trainees into ophthalmology.

METHODS

A retrospective analysis of the top 20 ranked candidates in the Duke-Elder examination from 1989 to 2005 (except 1995) was carried out to determine which of them subsequently entered the ophthalmic training and General Medical Council Specialist Registers.

RESULTS

Out of the top 20 candidates in the exam, 29.5% went into specialist training in ophthalmology. Some appeared in the top 20 more than once, with 56% of them going into ophthalmic training, but they had a similar median time to enter training as those who appeared in the top 20 once. There was no significant evidence to suggest that the overall median ranking scores between the UK medical schools differed (P=0.23; Kruskal-Wallis test). However, there was a marked difference in frequency of top 20 candidates from each medical school, which could not be explained by the size of the medical school alone.

CONCLUSION

It is difficult to conclude from these findings the importance that the Duke-Elder examination has in the selection of trainees into ophthalmology. The role of cognitive factors in selection into postgraduate medical/surgical training is discussed, along with the potential academic criteria, which may influence interview scores.

Using United States Medical Licensing Examination(®) (USMLE) examination results to predict later in-training examination performance among general surgery residents

OBJECTIVE

To examine the relationship between and predictive nature of United States Medical Licensing Examination(®) (USMLE) Step 1, Step 2, and American Board of Surgery In-Training Examination (ABSITE) postgraduate year (PGY) 1-5 scores from 2 general surgery programs from 1999-2009, with a goal of discerning how Step 1 and Step 2 scores should be used in resident selection and screening.

DESIGN

A descriptive, retrospective, correlational study was conducted using data from existing program records.

SETTING

Two accredited Midwestern community teaching hospital general surgery residency programs.

PARTICIPANTS

Data were collected from the records of N = 34 residents completing the programs in the years 1999-2009.

RESULTS

Although a statistically significant correlation exists between USMLE Step 1 and ABSITE PGY 3 scores, Step 2 scores were more highly correlated to ABSITE scores from all years of residency training. In both hierarchical and simple regression models, Step 2 scores were superior in predicting ABSITE PGY 1-5 scores, with the strongest predictability for PGY 3 and PGY 5 scores.

CONCLUSIONS

USMLE Step 1 scores showed limited utility in predicting later ABSITE scores whereas Step 2 scores were more predictive of all years of ABSITE scores. These findings should prompt additional research into the relationship between examination performances at different points along the general surgery education continuum. In the mean time, general surgery program directors and faculty might do well to examine the relationships between Step 1 and Step 2 scores and ABSITE scores in their own programs to evaluate the usefulness of considering either score when ranking potential residents for selection into a program.

Correlation of USMLE Step 1 scores with performance on dermatology in-training examinations

BACKGROUND

Although United States Medical Licensing Examination (USMLE) Step 1 was not designed to predict resident performance, scores are used to compare residency applicants. Multiple studies have displayed a significant correlation among Step 1 scores, in-training examination (ITE) scores, and board passage, although no such studies have been performed in dermatology.

OBJECTIVE

The purpose of this study is to determine if this correlation exists in dermatology, and how much of the variability in ITE scores is a result of differences in Step 1 scores. This study also seeks to determine if it is appropriate to individualize expectations for resident ITE performance.

METHODS

This project received institutional review board exemption. From 5 dermatology residency programs (86 residents), we collected Step 1 and ITE scores for each of the 3 years of dermatology residency, and recorded passage/failure on boards. Bivariate Pearson correlation analysis was used to assess correlation between USMLE and ITE scores. Ordinary least squares regression was computed to determine how much USMLE scores contribute to ITE variability.

RESULTS

USMLE and ITE score correlations were highly significant (P < .001). Correlation coefficients with USMLE were: 0.467, 0.541, and 0.527 for ITE in years 1, 2, and 3, respectively. Variability in ITE scores caused by differences in USMLE scores were: ITE first-year residency = 21.8%, ITE second-year residency = 29.3%, and ITE third-year residency = 27.8%.

LIMITATIONS

This study had a relatively small sample size, with data from only 5 programs.

CONCLUSIONS

There is a moderate correlation between USMLE and ITE scores, with USMLE scores explaining ∼26% of the variability in ITE scores.

Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination?: a multicenter study

BACKGROUND

The US Medical Licensing Examination (USMLE) and Orthopaedic In-Training Examination (OITE) are commonly used to select medical students or residents, respectively. Knowing how well these examinations predict performance on the American Board of Orthopaedic Surgery (ABOS) Part I certifying examination is important to provide evaluations for medical students and residents. Previous studies comparing the OITE scores with the ABOS Part 1 scores have been limited to one program.

QUESTIONS/PURPOSES

Therefore, we compared the scores on the USMLE Step 1 and OITE examinations with those on the ABOS Part I certifying examination using data from four ACGME approved residency programs.

METHODS

We reviewed 202 resident files from 1996 to 2008 from four programs in the same geographic region. Of those, 181 (90%) had complete records. De-identified data were used to compare USMLE Step 1 scores, OITE percentile rank scores, and ABOS Part I percentile rank scores. Pearson coefficients and receiver operator curves were calculated to assess the relationships between tests.

RESULTS

We found a correlation of 0.53 between the USMLE Step 1 and ABOS Part I, and an average correlation of 0.50 for postgraduate years (PGY) 2 through 5 OITE scores and ABOS Part I. There was a stepwise increase in correlation from PGY 2 through PGY 5 between the OITE scores and ABOS scores. Those who averaged in the 27th percentile or lower on the OITE had a 57% chance of failing the ABOS Part I examination.

CONCLUSION

USMLE Step 1 scores correlated with ABOS Part I certifying examination scores, and we therefore believe it may be used as one factor in resident selection. Use of the OITE scores in guiding education and feedback appears to be justified.

Relationship among United States Medical Licensing Step I, orthopedic in-training, subjective clinical performance evaluations, and american board of orthopedic surgery examination scores: a 12-year review of an orthopedic surgery residency program

OBJECTIVE

To improve the understanding of relationships among United States Medical Licensing Examination (USMLE Step I), Orthopedic In-Training Examination (OITE), Subjective Clinical Performance Evaluations, and American Board of Orthopedic Surgery Examination Part I (Abos-I) and Part II (Abos-II), which would help residency programs better achieve their educational mission.

DESIGN

A 12-year descriptive study of retrospectively collected data.

SETTING

One residency program with 47 resident participants.

RESULTS

Residents that failed Abos-I and Abos-II had lower program mean OITE year-in-training (YIT) percentile rank scores. The program mean OITE YIT percentile rank score had a moderate relationship with Abos-I (% correct) score (r = 0.68, p < 0.0001) and an insignificant relationship with USMLE Step I (3-digit) score (r = 0.22, p = 0.13). Residents with upper quartile (>or=220) USMLE Step I (3-digit) scores for our program had higher program mean OITE YIT percentile rank scores and Abos-I (% correct) scores than residents with lower quartile scores (<or=202). Residents who scored in the upper quartile (>or=55) for the program mean OITE YIT percentile rank score had higher Abos-I (% correct) scores than residents who did not. Residents who scored in the lower quartile for the third postgraduate year (PGY-3) program OITE YIT percentile rank score or for the program mean OITE YIT percentile rank score had a 5.2 and 5.8 time greater Abos-I failure risk, respectively. The program PGY-3 OITE YIT percentile rank score was the strongest Abos-I (% correct) score discriminator. Resident Abos-I (% correct), program mean OITE YIT, and program PGY-3 OITE YIT percentile rank scores were the strongest discriminators for Abos-II passage. Residents with a program mean OITE YIT percentile rank score >or=28, program PGY-3 OITE YIT percentile rank score >or=39, and USMLE Step I (3-digit) score >or=207 were more likely to pass Abos-I and II. Residents that had lower quartile USMLE Step I (3-digit) scores for our program had a 2.3 time greater Abos-I failure risk. Program residents with >or=2 below-average subjective clinical performance evaluations had lower Abos-I (% correct) scores but had similar Abos-I and II pass rates.

CONCLUSION

Our program uses the USMLE Step I (3-digit) score as a preacceptance estimate of likely supplemental guided mentoring needs. Program mean OITE YIT percentile rank and PGY-3 OITE YIT percentile rank scores help identify educational deficiencies and predict eventual Abos-I and II passage. Subjective clinical performance evaluations provide important supplemental information regarding professionalism, communication, and patient care skills.

Ophthalmology resident selection: current trends in selection criteria and improving the process

OBJECTIVE

To document and assess current ophthalmology resident selection practices as well as to initiate discussion on how best to improve the process.

DESIGN

Online survey comprising 56 questions.

PARTICIPANTS

Program directors, chairpersons, or members of the resident selection committee representing 65 United States ophthalmology residency programs accredited by the Accreditation Council on Graduate Medical Education.

METHODS

Study participants completed an online, anonymous survey consisting primarily of multiple choice questions, with single or multiple answers.

MAIN OUTCOME MEASURES

Ophthalmology resident selection practices were evaluated and included: screening of applications, interview processes, selection factors, and formation of rank lists; recommendations given to applicants; and respondent satisfaction with the current selection process.

RESULTS

As a group, survey respondents deemed the following factors most important in resident selection: interview performance (95.4%), clinical course grades (93.9%), letters of recommendation (83.1%), and board scores (80%). Statistical analyses deemed that the best predictors of resident performance are interviews, clinical course grades, recommendation letters, and ophthalmology rotation performance.

CONCLUSIONS

Ophthalmology resident selection is a relatively subjective process, continuing to rely heavily on cognitive factors. Because these factors are not always indicative of ultimate resident quality, it would be helpful if ophthalmology training programs improved selection practices to discern who most likely will become a successful resident and future ophthalmologist. Long-term studies correlating applicant attributes with residency and postresidency success are needed to recommend guidelines for a more standardized selection process.

Correlation of United States Medical Licensing Examination and Internal Medicine In-Training Examination performance

The Internal Medicine In-Training Examination (ITE) is administered during residency training in the United States as a self-assessment and program assessment tool. Performance on this exam correlates with outcome on the American Board of Internal Medicine Certifying examination. Internal Medicine Program Directors use the United States Medical Licensing Examination (USMLE) to make decisions in recruitment of potential applicants. This study was done to determine a correlation of USMLE Steps 1, 2 and 3 results with ITE scores in each level of Internal Medicine training. A retrospective review of all residents graduating from an Internal Medicine program from 1999 to 2006 was done. Subjects included had data for all USMLE Steps and ITE during all years of training. Thirty-one subjects were included in the study. Correlations of USMLE Steps 1, 2 and 3 were done with ITE scores (percent correct) in each year of training. Pearson's correlation coefficient (r) was determined for each pairing and a t test to determine statistical significance of the correlation was done. Statistical significance was defined as P value <0.05. The r values for USMLE Step 1 and ITE percent correct in PGY I, II and III were 0.46, 0.55 and 0.51 respectively. Corresponding r values for USMLE Step 2 and ITE percent correct were 0.79, 0.70 and 0.72; for USMLE Step 3 these values were 0.51, 0.37 and 0.51 respectively for each training year. USMLE scores are correlated with ITE scores. This correlation was strongest for USMLE Step 2.

Orthopaedic in-training examination performance: a nine-year review of a residency program database

BACKGROUND

The Orthopaedic In-Training Examination (OITE) provides an objective way for both the residency program and the resident to monitor progress.

METHODS

This longitudinal descriptive study tracks the OITE performance of 16 residents from one orthopaedic surgery residency program over nine years (1997-2005). Domain comparisons are made to national averages (N > 3000) by mean difference and split middle celeration line assessment trend analysis to evaluate program strengths and weaknesses. Further evaluation by postgraduate year (PGY) is made of domains that were deemed to be in need of moderate attention.

RESULTS

Resident performance for the medically related issues, rehabilitation, and sports medicine domains were deemed acceptable. Resident performance on the orthopaedic science, orthopaedic diseases, hip and knee reconstruction, spine, and shoulder and elbow domains were deemed to be in need of minor attention. Musculoskeletal trauma, pediatric orthopaedics, foot and ankle, and hand domains were deemed to be in need of moderate attention. Further analysis of these domains by PGY revealed less positive slopes for the pediatric orthopaedics and musculoskeletal trauma domains, and a negative slope for the hand domain between PGY-3 and PGY-4, each indicative of lower domain-specific OITE scores than the national mean. A similar, less positive slope was identified between PGY-2 and PGY-3 for the foot and ankle domain.

CONCLUSIONS

A longitudinal descriptive review of orthopaedic surgery resident OITE performance enables the departmental education committee to identify curricular content areas that require attention, select the most efficient modes of educational information delivery, facilitate learning through the application of innovative educational technologies, establish measurable educational program goals, and more prescriptively allocate personnel and equipment resources.

Comparison of surgical residency applicants from U.S. medical schools with U.S.-born and foreign-born international medical school graduates

OBJECTIVE

Compare characteristics of U.S. medical school graduates with U.S.-born and foreign-born international medical school (IMG) graduates.

DESIGN

Retrospective analysis.

SETTING

East Carolina University, a tertiary care teaching hospital.

PARTICIPANTS

Demographic data, United States Medical Licensing Examination (USMLE) scores, attempts needed to achieve a passing score, number of scholarly works, attainment of an advanced degree, and employment history since medical school graduation were obtained from all Electronic Residency Application Service applications to a general surgery residency for the 2007 match.

RESULTS

In all, 572 applicants were evaluated. Comparing U.S. graduates with U.S.-born IMGs and foreign-born IMGs. IMGs are older (mean, 28.9 vs 29.9 vs 33.0 years, respectively), more frequently male (70% vs 80% vs 86%, respectively), and hold more advanced degrees (11% vs 13% vs 19%, respectively). Mean time between graduation and application to residency was 0.3 years for U.S. graduates, 1.5 years for U.S.-born IMGs, and 7.7 years for foreign-born IMGs. Although mean USMLE Step 1 scores were similar (206 vs 200 vs 202, respectively), IMGs more frequently required multiple attempts to achieve a passing score (9% vs 20% vs 24%, respectively). Mean USMLE Step 2 scores were lower (213 vs 201 vs 203, respectively), and IMGs again required more attempts to achieve a passing score (11% vs 22% vs 19%, respectively). U.S. graduates produced an average of 1.7 scholarly works compared with 0.9 scholarly works for U.S.-born IMGS and 3.9 scholarly works for foreign-born IMGs. U.S.-born graduates held a mean of 0.3 jobs since graduation with most positions being preliminary surgery residents. U.S.-born IMGs held an average of 2.2 jobs most frequently in research or U.S. preliminary surgery, and foreign-born IMGs held an average of 5.1 jobs; most frequently, positions included international surgery or research.

CONCLUSION

IMGs are older, frequently male, hold more advanced degrees, and produce more scholarly works but require multiple attempts to pass the USMLE. IMGs also hold more jobs after graduation with most positions being in research or surgery.

Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents?

BACKGROUND

Residency selection committees commonly utilize USMLE scores as criteria to screen residency applicants.

OBJECTIVES

The purpose of this study is to evaluate the relationship between United States Medical Licensing Examination (USMLE) and American Board of Emergency Medicine (ABEM) in-training examination scores (ITEs).

METHODS

In an Accreditation Council for Graduate Medical Education-accredited emergency medicine residency program, data were collected for this retrospective cohort study for the classes of 2002-2006. USMLE Step 1 and 2 scores and the ABEM ITEs were recorded for each post-graduate year (PGY) within the aforementioned time frame. Step 1 and 2 scores were compared to consecutive PGY ABEM ITEs to evaluate for an association.

RESULTS

There were 51 USMLE Step 1 and 39 Step 2 scores available for comparison with 153 ABEM ITEs. The mean USMLE Step 1 and Step 2 scores were 228.9 (range 197-252) and 228.4 (range 168-259), respectively. The mean in-training percentiles for the PGY 1, 2, and 3 years were 40.4, 68.3, and 81.7, respectively. The R-squared values for the Step 1 scores compared to the PGY 1, 2, and 3 years' ITEs were 0.25, 0.18, and 0.16, respectively. The R-squared values for Step 2 scores as compared to the ABEM ITEs for the PGY 1, 2, and 3 years were 0.43, 0.44, and 0.38, respectively. Residents who scored below 200 on either USMLE Step 1 or Step 2 had significantly lower mean ABEM ITEs than residents who scored above 200 (p < 0.05) and were 10-fold more likely than residents who scored above 220 to score below the 70th percentile in their PGY3 ABEM ITE.

CONCLUSIONS

USMLE Step 1 scores are mildly correlated and Step 2 scores are moderately correlated with ABEM ITEs. Scoring below 200 on either test is associated with significantly lower ABEM ITEs.

A survey of collection development for United States Medical Licensing Examination (USMLE) and National Board Dental Examination (NBDE) preparation material

OBJECTIVE

The research sought to ascertain medical and dental libraries' collection development policies, evaluation methods, purchase decisions, and issues that relate to print and electronic United States Medical Licensing Examination (USMLE) and National Board Dental Examination (NBDE) preparation materials.

METHODS

The investigators surveyed librarians supporting American Association of Medical Colleges (AAMC)-accredited medical schools (n = 58/125) on the USMLE and librarians supporting American Dental Association (ADA)-accredited dental schools (n = 23/56) on the NBDE. The investigators analyzed the data by cross-tabulating and filtering the results using EFM Continuum web survey software. Investigators also surveyed print and electronic USMLE and NBDE preparation materials from 2004-2007 to determine the number of publications and existence of reviews.

RESULTS

A majority of responding AAMC libraries (62%, n = 58) provide at least 1 electronic or online USMLE preparation resource and buy an average of 11.6 print USMLE titles annually. Due to a paucity of NBDE print and electronic resources, ADA libraries bought significantly fewer print resources, and only 1 subscribed to an electronic resource. The most often reported evaluation methods for both populations were feedback from medical or dental students, feedback from medical or dental faculty, and online trials. Some AAMC (10%, n = 58) and ADA libraries (39%, n = 23) libraries reported that no evaluation of these materials occured at their libraries.

CONCLUSIONS

From 2004-2007, publishers produced 45 USMLE preparation resources (total n = 546) to every 1 NBDE preparation resource (total n = 12). Users' needs, institutional missions and goals, financial status, and official collection policies most often underlie decisions to collect or not collect examination preparation materials. Evaluating the quality of examination preparation materials can be problematic due to lack of published reviews, lack of usability testing by libraries, and librarians' and library users' unfamiliarity with the actual content of examinations. Libraries must integrate faculty and students into the purchase process to make sure examination preparation resources of the highest quality are purchased.

Associations between United States Medical Licensing Examination (USMLE) and Internal Medicine In-Training Examination (IM-ITE) scores

BACKGROUND

Little is known about the associations of previous standardized examination scores with scores on subsequent standardized examinations used to assess medical knowledge in internal medicine residencies.

OBJECTIVE

To examine associations of previous standardized test scores on subsequent standardized test scores.

DESIGN

Retrospective cohort study.

PARTICIPANTS

One hundred ninety-five internal medicine residents.

METHODS

Bivariate associations of United States Medical Licensing Examination (USMLE) Steps and Internal Medicine In-Training Examination (IM-ITE) scores were determined. Random effects analysis adjusting for repeated administrations of the IM-ITE and other variables known or hypothesized to affect IM-ITE score allowed for discrimination of associations of individual USMLE Step scores on IM-ITE scores.

RESULTS

In bivariate associations, USMLE scores explained 17% to 27% of the variance in IME-ITE scores, and previous IM-ITE scores explained 66% of the variance in subsequent IM-ITE scores. Regression coefficients (95% CI) for adjusted associations of each USMLE Step with IM-ITE scores were USMLE-1 0.19 (0.12, 0.27), USMLE-2 0.23 (0.17, 0.30), and USMLE-3 0.19 (0.09, 0.29).

CONCLUSIONS

No single USMLE Step is more strongly associated with IM-ITE scores than the others. Because previous IM-ITE scores are strongly associated with subsequent IM-ITE scores, appropriate modeling, such as random effects methods, should be used to account for previous IM-ITE administrations in studies for which IM-ITE score is an outcome.

Re-engineering the resident applicant selection process in ophthalmology: a literature review and recommendations for improvement

The current resident selection process for ophthalmology has undergone little change over the last several years and remains highly dependent on the traditional selection factors (i.e., grades, honors, letters of recommendation, and an interview). Unfortunately, these selection factors have not been shown to be consistently predictive of future resident performance. In addition, the Accreditation Council for Graduate Medical Education (ACGME) has mandated implementation of six new competencies in resident training in the USA and the current selection process does not directly recruit for these competencies. We propose an implementation strategy to re-engineer and improve the resident selection process in ophthalmology and potentially develop assessments that would be predictive of actual downstream resident performance that would encompass the ACGME related competencies. An intra-departmental Task Force for the ACGME Competencies reviewed a PubMed literature search regarding resident selection. A content expert (AGL) gleaned selected "good practices" from the literature review and summarized the results. Specific recommendations were reviewed for topicality to ophthalmology and where possible for feasibility, reliability, and validity. We summarize several good practices identified from the literature review and propose an implementation matrix for aligning the resident application process with the ACGME competencies that might include: using a standardized and consolidated academic score for the cognitive domains; converting the letter of recommendation format into a letter of evaluation; standardizing the letters of evaluation, including the "Dean's letter"; using behavior specific interview techniques with standardized questions; and developing a specialty based consensus for the selection of traits specific to ophthalmology that might predict success. The resident selection process for ophthalmology might be improved by implementation of specific good practices from the literature. Ophthalmology should strive to develop applicant selection tools that might be useful for predicting residency performance and that would align with the ACGME competency mandate for tools to predict future performance as a physician.

Does US Medical Licensing Examination Step l score really matter in surgical residency match outcomes (and should it)?

BACKGROUND

The relationship between US Medical Licensing Examination Step 1 scores and core surgical-specialty match outcomes has not been well defined.

STUDY DESIGN

With IRB approval, we measured associations between aggregate Step 1 scores and other specialty-specific, match-process variables for 3 surgical-specialty matches. Chi-square tests measured differences between proportions of US students and independent applicants (ie, all non-US allopathic student applicants) who matched. Independent samples t-tests compared differences in Step 1 scores between matched- and unmatched-applicant groups. Pearson correlations measured the magnitude and direction of associations between matched-applicants' Step 1 scores and other variables of interest and between Step 1 scores for all match participants and percentage of positions filled by US students (two-tailed p values).

RESULTS

Step 1 scores were lower for unmatched- than matched-applicant groups for each specialty examined (each p < 0.0001). Matched-applicant groups' Step 1 scores positively correlated with each unmatched-applicant groups' Step 1 scores (r =.82, p < 0.0001), Step 1 gap between matched- and unmatched-applicant groups' scores (r = .40, p = 0.035), percentage of positions filled by US students (r = .62, p < 0.0001), and mean number of applications filed/applicant (r = .50, p < 0.0001). Step 1 scores for all match participants correlated with percentage of positions filled by US students (r = .61, p = 0.0006).

CONCLUSIONS

Step 1 scores were closely related to match process outcomes and match participation itself, with increasing Step 1 scores among both matched- and unmatched-applicant groups as specialty selectivity increased.

The American Society for Clinical Pathology resident in-service examination: does resident performance provide insight into the effectiveness of clinical pathology education?

The resident in-service examination in pathology is an in-training exercise that is taken by virtually all pathology residents in the United States as well as by some participants in Canada, Ireland, and Lebanon. Although all of the anatomic pathology topics in the examination, with only one exception-forensic pathology, show significant improvement in scores over the 4 years of residency training, three areas of clinical pathology training (laboratory administration, clinical chemistry, and microbiology) show significantly lower improvement in performance over the years of residency training. By contrast, transfusion medicine, hematopathology and the special topics section of the examination all demonstrate improved performance by residents over time. While the reason behind these differences must remain speculative at this time, these findings suggest that measures to improve effectiveness in clinical pathology training might be suggested by examining the differences between residency training practices between higher and lower performing areas of clinical pathology.

Key criteria for selection of radiology residents: results of a national survey

RATIONALE AND OBJECTIVES

We sought to identify the criteria that academic radiology departments in the United States consider for selecting their residents.

MATERIALS AND METHODS

In a cross-sectional study, a validated survey was sent to all the program directors of radiology residency programs. A total of 25 variables were studied. Descriptive statistics and correlations were calculated by the chi2 test. Nonparametric correlations were calculated with the Kruskal-Wallis rank test. Statistical significance was set at 5% alpha-error level (P<.05).

RESULTS

We had a response rate of 53.1% (77 of 145). All responders participate in the National Resident Matching Program (NRMP), and 93.5% fill all their positions through NRMP. The preinterview selection criteria showed no significant difference by size, region, or affiliation with a medical school. An "interviewing body" carries out the interview process in 87.3% of the cases. Residents and fellows are part of the interviewing body in 76.5% of the programs, the body has the final word in accepting candidates in 62.9% of the programs, 55.4% of the programs use score sheets during interviews with candidates, and only 6.5% of the programs perform panel interviews. Programs associated with a medical school are significantly more likely to have more members in their interviewing body and to use score sheets when evaluating candidates, and panel interviews (more than one candidate or interviewer) are significantly more common among programs in the northeast region.

CONCLUSION

All preinterview selection criteria and some interview structural characteristics are independent of the program's size, region, or affiliation with a medical school. More research regarding optimal preselection and interview processes is needed, and closer attention should be paid to the NRMP process if current practices are to be maintained.