Is a faculty developed pretest equivalent to pre-third year GPA or USMLE step 1 as a predictor of third-year internal medicine clerkship outcomes?

Investigating the validity of web-enabled mechanistic case diagramming scores to assess students' integration of foundational and clinical sciences

As medical schools have changed their curricula to address foundational and clinical sciences in a more integrated fashion, teaching methods such as concept mapping have been incorporated in small group learning settings. Methods that can assess students' ability to apply such integrated knowledge are not as developed, however. The purpose of this project was to assess the validity of scores on a focused version of concept maps called mechanistic case diagrams (MCDs), which are hypothesized to enhance existing tools for assessing integrated knowledge that supports clinical reasoning. The data were from the medical school graduating class of 2018 (N = 136 students). In 2014-2015 we implemented a total of 16 case diagrams in case analysis groups within the Mechanisms of Health and Disease (MOHD) strand of the pre-clinical curriculum. These cases were based on topics being taught during the lectures and small group sessions for MOHD. We created an overall score across all 16 cases for each student. We then correlated these scores with performance in the preclinical curriculum [as assessed by overall performance in MOHD integrated foundational basic science courses and overall performance in the Clinical and Professional Skills (CAPS) courses], and standardized licensing exam scores [United States Medical Licensing Exam (USMLE)] Step 1 (following core clerkships) and Step 2 Clinical Knowledge (at the beginning of the fourth year of medical school). MCD scores correlated with students' overall basic science scores (r = .46, p = .0002) and their overall performance in Clinical and Professional Skills courses (r = .49, p < .0001). In addition, they correlated significantly with standardized exam measures, including USMLE Step 1 (r = .33, p ≤ .0001), and USMLE Step 2 CK (r = .39, p < .0001). These results provide preliminary validity evidence that MCDs may be useful in identifying students who have difficulty in integrating foundational and clinical sciences.

Exploring the Predictors of Post-Clerkship USMLE Step 1 Scores

Theory: We used two theoretical frameworks for this study: a) experiential learning, whereby learners construct new knowledge based on prior experience, and learning grows out of a continuous process of reconstructing experience, and b) deliberate practice, whereby the use of testing (test-enhanced learning) promotes learning and produces better long-term retention. Hypothesis: We hypothesized that moving the USMLE Step 1 exam to follow the clerkship year would provide students with a context for basic science learning that may enhance exam performance. We also hypothesized that examination performance variables, specifically National Board of Medical Examiners (NBME) Customized Basic Science Examinations and NBME subject examinations in clinical disciplines would account for a moderate to large amount of the variance in Step 1 scores. Thus we examined predictors of USMLE Step 1 scores when taken after the core clerkship year. Method: In 2011, we revised our medical school curriculum and moved the timing of Step 1 to follow the clerkship year. We performed descriptive statistics, an ANCOVA to compare Step 1 mean scores for three graduating classes of medical students before and after the curriculum changes, and stepwise linear regression to investigate the association between independent variables and the primary outcome measure after curriculum changes. Results: 993 students took the Step 1 exam, which included graduating classes before (2012-2014, N = 491) and after (2015-2017, N = 502) the curriculum change. Step 1 scores increased significantly following curricular revision (mean 218, SD 18.2, vs. 228, SD 16.7, p < 0.01) after controlling for MCAT and undergraduate GPA. Overall, 66.4% of the variance in Step 1 scores after the clerkship year was explained by: the mean score on fourteen pre-clerkship customized NBME exams (p < 0.01, 57.0% R²); performance on the surgery NBME subject exam (p < 0.01, 3.0% R²); the pediatrics NBME subject exam (p < 0.01, 2.0% R²); the Comprehensive Basic Science Self-Assessment (p < .01, 2.0% R²) ; the internal medicine NBME subject exam (p < 0.01, 0.03% R²), pre-clerkship Integrated Clinical Skills score (p < 0.01, 0.05% R²), and the pre-matriculation MCAT (p < 0.01, 0.01% R²). Conclusion: In our institution, nearly two-thirds of the variance in performance on Step 1 taken after the clerkship year was explained mainly by pre-clerkship variables, with a smaller contribution emanating from clerkship measures. Further study is needed to uncover the specific aspects of the clerkship experience that might contribute to success on high stakes licensing exam performance.

Adaptation of Medical Students During Clinical Training: Effects of Holistic Preclinical Education on Clerkship Performance

THEORY

Taiwan's medical undergraduate program at a university or medical center is a continuation of 12 years of compulsory citizenship education rooted in holistic philosophies. Students acquire both technical knowledge and nontechnical attributes, which are necessary for success in further work and life. The early clinical learning experiences of medical students are primarily acquired through clerkships. These clerkships require medical students to apply and extend what they learned during their preclinical education; however, previous studies have explored this issue through examining fragmentary factors such as preclinical course grades and traits but not undertaking comprehensive, whole-person investigations.

HYPOTHESES

To account for the potential benefits of a holistic approach in medical students' learning, we propose three hypotheses: Medical students' preclinical performance on Taiwan's technical and nontechnical higher education assessments are positively associated with their clinical competence (Hypothesis 1) and psychological well-being (Hypothesis 2) during clerkships, and medical students' psychological well-being during clerkships is positively associated with their clinical competence (Hypothesis 3).

METHOD

We studied a cohort of 65 medical students engaged in clerkships from September 2013 to April 2015. Their preclinical technical knowledge scores-formal curricular grades received from course instructors-were obtained from their medical school's archival dataset. Their nontechnical attributes-moral and social performance scores received from student mentors and physical performance scores received from course instructors-were also obtained from the school's archival data set. The medical students' competence in their 2-year clinical clerkships was measured using the objective structured clinical examination scores from the end of both clerkship years. The medical students' psychological well-being during their 2-year clerkships was measured according to burnout level, which was determined using routine online surveys that employed validated, structured, and self-administered questionnaires at each specialty rotation. Multiple regressions and linear mixed-effects model were employed for statistical analysis.

RESULTS

Our study revealed that higher preclinical technical knowledge predicted superior clinical competence and a higher level of burnout during clerkships. By contrast, higher preclinical nontechnical attributes (i.e., higher preclinical moral, social, and physical performance) predicted lower level of burnout. However, no relationship was discovered between clerkship burnout and the clinical competence of the medical students.

CONCLUSIONS

Our study verified the value of a holistic education that encompasses both technical knowledge and nontechnical attributes during the preclinical learning stage for medical students. Our findings can serve as a reference for medical educators designing preclinical educational programs for medical students.

Factors associated with performance in an internal medicine clerkship

The purpose of this retrospective study was to examine the relationship between demographic and educational variables and student performance on an internal medicine (IM) clerkship in order to target areas for educational intervention and potential early remediation. This study examined data associated with third-year medical student performance (N = 505) during the IM clerkship at Baylor Scott & White, Temple/Texas A&M Health Science Center College of Medicine from 2005 to 2011. Multiple regression analysis (N = 341) showed that a model containing the following variables was significantly associated with scores on the National Board of Medical Examiners (NBME) subject exam, accounting for 46.5% of the variance: Objective Structured Clinical Exam (OSCE), Medical College Admissions Test (MCAT), US Medical Licensing Exam (USMLE) Step 1, second-year grade point average (GPA), and clinical evaluation. A model containing USMLE Step 1, clinical evaluation, and NBME was significantly associated with OSCE score, accounting for 30% of the variance. Additionally, a model containing age, MCAT score, undergraduate GPA, NBME subject exam score, and OSCE was significantly associated with clinical evaluation score, accounting for 22% of score variance. Age accounted for the most unique variance in clinical evaluation score. Gender and IM interest group were not significantly associated with any outcome variable. In conclusion, in contrast to previous studies in the field, we did not find a significant association between undergraduate GPA and NBME score. Our findings supply further evidence that the OSCE, typically believed to be a clinical performance exam, actually assesses a broader set of domains. Interest group membership did not confer any academic benefit to medical students in IM clerkships in our study.

Predictors of medical school clerkship performance: a multispecialty longitudinal analysis of standardized examination scores and clinical assessments

BACKGROUND

Evidence suggests that poor performance on standardized tests before and early in medical school is associated with poor performance on standardized tests later in medical school and beyond. This study aimed to explore relationships between standardized examination scores (before and during medical school) with test and clinical performance across all core clinical clerkships.

METHODS

We evaluated characteristics of 435 students at Mayo Medical School (MMS) who matriculated 2000-2009 and for whom undergraduate grade point average, medical college aptitude test (MCAT), medical school standardized tests (United States Medical Licensing Examination [USMLE] 1 and 2; National Board of Medical Examiners [NBME] subject examination), and faculty assessments were available. We assessed the correlation between scores and assessments and determined USMLE 1 cutoffs predictive of poor performance (≤10th percentile) on the NBME examinations. We also compared the mean faculty assessment scores of MMS students vs visiting students, and for the NBME, we determined the percentage of MMS students who scored at or below the tenth percentile of first-time national examinees.

RESULTS

MCAT scores correlated robustly with USMLE 1 and 2, and USMLE 1 and 2 independently predicted NBME scores in all clerkships. USMLE 1 cutoffs corresponding to poor NBME performance ranged from 220 to 223. USMLE 1 scores were similar among MMS and visiting students. For most academic years and clerkships, NBME scores were similar for MMS students vs all first-time examinees.

CONCLUSIONS

MCAT, USMLE 1 and 2, and subsequent clinical performance parameters were correlated with NBME scores across all core clerkships. Even more interestingly, faculty assessments correlated with NBME scores, affirming patient care as examination preparation. USMLE 1 scores identified students at risk of poor performance on NBME subject examinations, facilitating and supporting implementation of remediation before the clinical years. MMS students were representative of medical students across the nation.

Relationship between clinical experiences and internal medicine clerkship performance

OBJECTIVES

This study was conducted to assess the associations between several clerkship process measures and students' clinical and examination performance in an internal medicine clerkship.

METHODS

We collected data from the internal medicine clerkship at one institution over a 3-year period (classes of 2010-2012; n = 507) and conducted correlation and multiple regression analyses. We examined the associations between clerkship process measures (student-reported number of patients evaluated, percentage of core problems encountered, total number of core problems encountered, total number of clinics attended) and four clerkship outcomes (clinical points [a weighted summation of a student's clinical grade recommendations], ambulatory clinical points [the out-patient portion of clinical points], examination points [a weighted summation of scores on three clerkship examinations], and National Board of Medical Examiners examination score).

RESULTS

After controlling for pre-clerkship ability and gender, percentage of core problems was significantly associated with ambulatory clinical points (b = 3.84, total model R(2) = 0.14). Further, number of patients evaluated was significantly associated with clinical points (b = 0.19, total model R(2) = 0.22), but only for students who undertook first-quarter clerkships, who reported higher numbers of patients.

CONCLUSIONS

Notwithstanding a few positive (but small) associations, the results from this study suggest that clinical exposure is, at best, weakly associated with internal medicine clerkship performance.

Clerkship directors' practices with respect to preparing students for and using the National Board of Medical Examiners Subject Exam in medicine: results of a United States and Canadian Survey

PURPOSE

Clerkship directors' practices regarding the National Board of Medical Examiners (NBME) subject exam in medicine are important in enhancing educational evaluation policy. The study's purpose was to determine clerkship directors' use of the subject exam in medicine and related learning activities in the context of curricula and outcomes of the directors' internal medicine clerkships.

METHOD

The authors conducted a survey of directors of internal medicine clerkships in 2007. They performed descriptive statistical and multivariate analyses on all responses.

RESULTS

Of 110 clerkship directors, 82 responded to the survey, for an overall response rate of 75%. Eighty-eight percent of the clerkship directors required the NBME subject examination in medicine. The mean minimum passing score was 62 (SD = 4.2); this score was not adjusted throughout the academic year, and it contributed 20% to 25% of the final grade. Most (89%) clerkships allowed students a retake after a failed first attempt. Most clerkship directors prepared students for the NBME subject exam in their programs through some combination of lectures, independent self-study, and review sessions with exam-preparation review books. However, 42% of clerkship directors lacked a specific strategy for a retake after a failure.

CONCLUSION

Clerkship directors' use of the NBME subject exam in medicine is high. Most allow a retake after a first failure, and a combination of strategies is currently provided to help students prepare. A need exists to develop remediation plans for students who fail the exam. This report may serve as a reference for curricular and programmatic clerkship decisions.

USMLE Step 1 Scores as a Significant Predictor of Future Board Passage in Pediatrics

PURPOSE

To explore the relationship(s) between USMLE, In-Training Exam, and American Board of Pediatrics (ABP) board-certifying exam scores within a Pediatric residency-training program.

METHODS

Data were abstracted from records of graduating residents from the Pediatric residency program at the University of Florida College of Medicine Jacksonville from 1999 to 2005. Seventy (70) residents were identified and their files reviewed for the following information: USMLE Step 1 and 2 scores, in-training exam results and eventual board scores as reported by the ABP. Correlation and regression analyses were performed and compared across all tests.

RESULTS

The correlation coefficients between the three types of tests were all statistically significant. Using logistic regression, however, only USMLE Step 1 scores (compared to Step 2) had a statistically significant association with board performance. Interestingly, none of the three in-training exam scores had any additional impact on predicting board performance given one's USMLE Step 1 score. USMLE Step 1 scores greater than 220 were associated with nearly a 95 per cent passage rate on the board-certifying exam.

CONCLUSIONS

The data suggests that performance on USMLE Step 1 is an important predictor of a resident's chances of passing the pediatric boards. This information, which is available when a resident initiates training, can be used to identify those at risk of not passing the boards. While Step 1 scores should not be used as a sole determinant in the recruiting process, individual learning plans can be developed and implemented early in training to maximize one's ability to pass the certifying exam.

Effect of electronic versus print format and different reading resources on knowledge acquisition in the third-year medicine clerkship

BACKGROUND

Many medical students have access to electronic and print resources. It is not known if the format or the choice of resources effects knowledge acquisition over an entire clerkship.

PURPOSE

To determine if the format (electronic or print) or choice of reading materials is associated with knowledge acquisition during the 3-year internal medicine clerkship.

METHODS

This was a prospective cohort study. During the last week of the clerkship, students took the National Board of Medical Examiners (NBME) exam and completed a survey ranking the reading materials used.

RESULTS

One hundred and fourteen 3rd-year internal medicine clerkship students participated. The most commonly reported resources were UpToDate (99%), a review or question book (93%), and Harrison's Principles of Internal Medicine (82%). Multivariate analysis showed a significant beneficial effect of any use of Harrison's (linear regression, p < .04, absolute difference 3 points, SD = 1.4) on NBME score. The format of the most important resource-electronic (61%) or print (39%)-did not predict the NBME score, p > .80.

CONCLUSIONS

The format (electronic or print) of resources did not appear to effect NBME score in the medicine clerkship. A secondary analysis showed the use of Harrison's textbook was associated with improved knowledge acquisition.