The Validity of MCAT Scores in Predicting Students' Performance and Progress in Medical School: Results From a Multisite Study

Pre-Admission Standardized Tests and the Relationship with Anesthesiologist Assistant Student Performance: A Retrospective Analysis

Background

The Anesthesiologist Assistant (AA) profession is gaining popularity, and as such, the number of applicants and the number of AA programs are also increasing. There is no published data on what makes a good AA student - multiple studies exist in this regard for anesthesiology residents and medical students. The Medical College Admissions Test (MCAT) and the Graduate Record Examination (GRE) have shown to have some correlation with student success in other health professions. There is no data to suggest that they can indicate potential success or academic problems during AA school. As such, this study evaluated the impact of these standardized test scores and student success in both the preclinical and clinical curricula at the Emory AA program.

Methods

A retrospective analysis of MCAT scores and GRE scores was performed. 2-tailed Pearson coefficients were calculated to see if there was a relationship between higher MCAT and GRE scores and performance in our program.

Results

2- tailed Pearson coefficients showed a moderately strong correlation between MCAT scores and our AA program's science curriculum. There was a weak correlation as well between GRE scores here. The correlation dropped slightly for the MCAT and performance during our clinical curriculum, and a similar drop was noted for the GRE.

Discussion

Our study shows that the MCAT appears to more strongly correlated with better performance in both the science curriculum of our program and also with our clinical program. The study likely implies that the MCAT appears to be a better gauge as to how a student will do in an AA program in comparison to the GRE. Further studies are needed to understand how to appropriately admit students who will successfully complete the rigorous AA program curriculum both at Emory and beyond.

Early practice makes histology masters: The use of a formative assessment quiz to prepare histology learners for a high-stakes final examination

Assessment of learners in the anatomical sciences is a complex task as it not only tests students' fact knowledge, but also the analysis of visual information. Sometimes, novice histology learners must acquire image recognition skills of microscopic structures in a short time frame. This paper describes a strategy of offering first year dental students at the University of Michigan a short, non-grade contributing, formative assessment quiz in order to better prepare them for a high-stakes, final summative histology examination. Data collected over 7 years indicate that students, who made use of this formative assessment opportunity, performed significantly better in their summative examination than students, who did not. Especially early practice quiz users profited most. Students, who used the practice quiz until they achieved a perfect score, also had statistically higher final examination scores. Students, who did not use the practice quiz, had a significantly lower cumulative D1-year Grade Point Average (D1-GPA) than students, who did, indicating that academically weaker students often underuse supporting learning resources. In general, scores from the Dental Admission Test (DAT) were weak predictors of learning success in the DENT 510 histology course. In contrast, the D1-GPA values had a medium strength positive correlation with final histology examination scores. The major problem that was encountered with this practice quiz strategy was that many students waited until close to the day of the final examination before taking advantage of this feedback opportunity, thereby reducing its potential benefit for improving their overall learning strategy for histology.

To Grade or Not to Grade (the Quiz): The Impact of Two Formative Assessment Grading Approaches on Summative Assessment Outcomes in an Integrated Pre-clinical Curriculum

Introduction

Formative assessments have overall been shown to improve summative evaluations in medical education. However, it remains unclear if utilizing them for course credit in an integrated curriculum over multiple subspecialties is beneficial for student acquisition of knowledge. We set out to determine if grading formative quizzes had an effect on student acquisition of knowledge via summative assessments.

Materials and Methods

In 2020, quizzes remained mandatory, but were not graded for course credit. We collected and compared formative (quiz) and summative (unit exam) score data from student cohorts when quizzes were graded versus not graded. Medical college admission test (MCAT) score, gender, and underrepresented in medicine (URM) status data were utilized to determine if they had effects on outcomes. We used a predefined region of indifference (± 5%) and second-generation p-values to determine if there were meaningful differences in average summative exam scores.

Results

Despite a drop in quiz scores after removing course credit, differences in average summative exam scores fully resided within the region of indifference for three of five course blocks, indicating no meaningful differences in summative exam scores existed for those blocks. In other blocks, the difference was not fully nested within the region of indifference; however, all data overlapped with this region, implying exam score differences were inconclusive for these blocks.

Conclusions

Our study demonstrates that there is either no meaningful effect on summative assessments or no conclusive detrimental effects when mandatory quizzes are not graded for course credit.

The predictive validity of MCAT scores and undergraduate GPA for COMLEX-USA licensure exam performance of students enrolled in osteopathic medical schools

CONTEXT

Osteopathic (Doctor of Osteopathic Medicine [DO]) medical students account for more than 25 % of all medical students in the United States.

OBJECTIVES

This study examined the predictive validity of Medical College Admission Test (MCAT) total scores and cumulative undergraduate grade point averages (UGPAs) for performance on the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) Level 1 and Level 2-CE (Cognitive Evaluation) licensure examinations administered by the National Board of Osteopathic Medical Examiners (NBOME). Additionally, the study examined the degree to which MCAT total scores and UGPAs provide comparable prediction of student performance by key sociodemographic variables.

METHODS

This study involved a collaborative effort between the Association of American Medical Colleges (AAMC), the American Association of Colleges of Osteopathic Medicine (AACOM) and the NBOME. Data were examined for 39 accredited DO-granting medical schools in the United States during the 2017 application cycle. Researchers utilized three regression models that included MCAT total scores, cumulative UGPA, and combined MCAT total scores and cumulative UGPA to determine predictive validity. Researchers also examined the comparability of prediction for sociodemographic variables by examining the differences between observed and predicted error for both scores and pass/fail success rates.

RESULTS

Medium to large correlations were discernible between MCAT total scores, UGPA, and COMLEX-USA examination outcomes. For both COMLEX-USA Level 1 and Level 2-CE scores and pass/fail outcomes, MCAT scores alone provided superior predictive value to UGPA alone. However, MCAT scores and UGPA utilized in conjunction provided the best predictive value. When predicting both licensure examination scores and pass/fail outcomes by sociodemographic variables, all three models provided comparable predictive accuracy.

CONCLUSIONS

Findings from this comprehensive study of DO-granting medical schools provide evidence for the value-added benefit of taking MCAT scores and UGPA into consideration, particularly when these measures are utilized in conjunction. Further, findings provide evidence indicating that individuals from different sociodemographic backgrounds who enter medical school with similar MCAT scores and UGPA perform similarly on licensure examination outcome measures.

Psychological and emotional experiences of participants in a medical school, early assurance admissions program targeting students from groups underrepresented in medicine

BACKGROUND

There are growing number of pathway programs, with an early assurance of admission, that target undergraduate students from groups underrepresented in medicine (URiM) to enable their competitiveness for and matriculation to medical school, including the Penn Access Summer Scholars (PASS) program. The psychological and emotional experiences of students in these programs, however, have not been previously described.

METHODS

Students from the summer 2021 cohort of the PASS program were interviewed using a structured set of questions that explored four specific areas: (i) the application process; (ii) the benefits and value of being in the PASS program; (iii) the emotional and psychological challenges and stresses of being in the PASS program; (iv) feelings and emotions about not taking the MCAT or having to interview at multiple schools. The transcribed, de-identified interviews were then subjected to a qualitative analysis.

RESULTS

Students in PASS reported that the program was valuable to them in that it reduced the stress of the pre-medical process; relieved worry and anxiety surrounding the MCAT; enabled development of supportive relationships and provided meaningful exposures to the medical profession and biomedical research. Despite this, students reported feelings of imposterism, guilt, and fear of disappointing, along with varying degrees of regret over not taking the MCAT and not interviewing at more than one medical school.

CONCLUSIONS

URiM and other marginalized students participating in early assurance admissions programs likely enter medical school with a range of positive and negative emotions as a result of their participation in these programs. These data can be used to inform the development of programing and other initiatives that further support the transition and success of these students in medical school.

The Supreme Court Decision on Affirmative Action-Fewer Black Physicians and More Health Disparities for Minoritized Groups

This Viewpoint discusses the US Supreme Court’s June 2023 ruling on affirmative action and its repercussions for Black physicians and health equity for racial and ethnic minority groups.

Athletes in medicine: A systematic review of performance of athletes in medicine

INTRODUCTION

As interest in medicine grows, admissions committees must review an increasingly competitive pool of medical school and residency candidates. Nearly all admissions committees have moved towards a holistic review, which considers an applicant's experiences and attributes in addition to academic metrics. As such, identifying nonacademic predictors of success in medicine is necessary. Parallels between skills necessary to succeed in athletics and in medicine have been drawn, including teamwork, discipline and resiliency. This systematic review synthesises the current literature to evaluate the relationship between participation in athletics and performance in medicine.

METHODS

The authors searched five databases to conduct a systematic review following PRISMA guidelines. Included studies assessed medical students, residents or attending physicians in the United States or Canada and used prior athletic participation as a predictor or explanatory variable. The review examined associations between prior athletic participation and outcomes in medical school, residency and/or as an attending physician.

RESULTS

Eighteen studies evaluating medical students (78%), residents (28%) or attending physicians (6%) met inclusion criteria for this systematic review. Twelve (67%) studies specifically assessed participants based on skill level, and five (28%) studies specifically assessed participants based on type of athletic participation (team versus individual). Sixteen studies (89%) found that former athletes performed significantly better than their counterparts (p < 0.05). These studies found significant associations between prior athletic participation and better outcomes in multiple performance indicators, including exam scores, faculty ratings, surgical errors and burnout.

CONCLUSIONS

Current literature, although limited, suggests that prior participation in athletics may be a predictor of success in medical school and residency. This was demonstrated through objective scoring methods, such as USMLE, and subjective outcomes, such as faculty ratings and burnout. Specifically, multiple studies indicate that former athletes demonstrated increased surgical skill proficiency and decreased burnout as medical students and residents.

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm

PURPOSE

To explore whether a machine-learning algorithm could accurately perform the initial screening of medical school applications.

METHOD

Using application data and faculty screening outcomes from the 2013 to 2017 application cycles (n = 14,555 applications), the authors created a virtual faculty screener algorithm. A retrospective validation using 2,910 applications from the 2013 to 2017 cycles and a prospective validation using 2,715 applications during the 2018 application cycle were performed. To test the validated algorithm, a randomized trial was performed in the 2019 cycle, with 1,827 eligible applications being reviewed by faculty and 1,873 by algorithm.

RESULTS

The retrospective validation yielded area under the receiver operating characteristic (AUROC) values of 0.83, 0.64, and 0.83 and area under the precision-recall curve (AUPRC) values of 0.61, 0.54, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The prospective validation yielded AUROC values of 0.83, 0.62, and 0.82 and AUPRC values of 0.66, 0.47, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The randomized trial found no significant differences in overall interview recommendation rates according to faculty or algorithm and among female or underrepresented in medicine applicants. In underrepresented in medicine applicants, there were no significant differences in the rates at which the admissions committee offered an interview (70 of 71 in the faculty reviewer arm and 61 of 65 in the algorithm arm; P = .14). No difference in the rate of the committee agreeing with the recommended interview was found among female applicants (224 of 229 in the faculty reviewer arm and 220 of 227 in the algorithm arm; P = .55).

CONCLUSIONS

The virtual faculty screener algorithm successfully replicated faculty screening of medical school applications and may aid in the consistent and reliable review of medical school applicants.

Does medical education reform change who is selected? A national cross-sectional survey from China

OBJECTIVES

Approximately 10 years ago, China introduced an education plan to improve the overall quality of medical education and to better serve the population's health needs. Many medical schools were then recognised and financed by China's Ministry of Education to develop and operationalise new pilot programmes (PPs) aligned with this plan. These ran in parallel with the traditional programmes (TPs). One way to achieve the plan's first aim, improving the quality of medical education, is to select academically stronger candidates. We, thus, examined and compared who were selected into PPs and TPs.

DESIGN

Cross-sectional study.

SETTING

Data were collected from 123 medical schools across China via the 2021 China Medical Student Survey.

PARTICIPANTS

Participants were undergraduate clinical medicine students across all year groups.

PRIMARY AND SECONDARY OUTCOME MEASURES

Medical school selection was via the National College Entrance Examination (NCEE). Medical students' NCEE performance and their sociodemographics were used as the primary and secondary outcome measures. Mann-Whitney or χ2 tests were used to compare the means between educational programmes (PPs vs TPs) and various selection outcomes. Multilevel mixed-effects regressions were employed to account for school idiosyncratic selection results.

RESULTS

Of the 204 817 respondents, 194 163 (94.8%) were in a TP and 10 654 (5.2%) a PP. PP respondents (median=75.2, IQR=69.5-78.8) had significantly higher NCEE scores than their TP counterparts (median=73.9, IQR=68.5-78.7). Holding constant their NCEE score, PP respondents were significantly more likely to come from urban areas, not be first-generation college students, and have parents with higher occupational status and income.

CONCLUSIONS

Assuming quality can be indicated by prior academic achievement at the point of selection, PPs achieved this mission. However, doing so limited medical students' diversity. This may be unhelpful in achieving the Education Plan's goal to better serve China's health needs.

Multivariate Modeling of Student Performance on NBME Subject Exams

Aim This study sought to determine whether it was possible to develop statistical models which could be used to accurately correlate student performance on clinical subject exams based on their National Board of Medical Examiner (NBME) self-assessment performance and other variables, described below, as such tools are not currently available.  Methods Students at a large public medical school were provided fee vouchers for NBME self-assessments before clinical subject exams. Multivariate regression models were then developed based on how self-assessment performance correlated to student success on the subsequent subject exam (Medicine, Surgery, Family Medicine, Obstetrics-Gynecology, Pediatrics, and Psychiatry) while controlling for the proximity of the self-assessment to the exam, USMLE Step 1 score, and the academic quarter. Results The variables analyzed satisfied the requirements of linear regression. The correlation strength of individual variables and overall models varied by discipline and outcome (equated percent correct or percentile, Model R² Range: 0.1799-0.4915). All models showed statistical significance on the Omnibus F-test (p<0.001). Conclusion The correlation coefficients demonstrate that these models have weak to moderate predictive value, dependent on the clinical subject, in predicting student performance; however, this varies widely based on the subject exam in question. The next step is to utilize these models to identify struggling students to determine if their use reduces failure rates and to further improve model accuracy by controlling for additional variables.

Prior Enlisted Medical Students at the Uniformed Services University: Outcomes During Four Years of Medical School

INTRODUCTION

The Uniformed Services University (USU) implemented the Enlisted to Medical Degree Preparatory Program (EMDP2) with the goal of enhancing the diversity of the military physician corps. Programs like EMDP2 can assist students in making the social and intellectual transition from undergraduate studies to medical school and beyond. These types of programs are also opportunities to reduce health disparities and prepare students to work in multicultural settings. The purpose of this study was to evaluate whether there was any significant difference in performance between USU medical students who had attended the EMDP2 and those who had not.

MATERIALS AND METHODS

We compared the results of National Board of Medical Examiners (NBME) Clinical Science Subjects, United States Medical Licensing Examination (USMLE) Step 1, and USMLE Step 2 Clinical Knowledge exams of EMDP2 learners from the School of Medicine classes of 2020 to 2023 to those of four similarly sized cohorts of their peers who varied by age and prior military service.

RESULTS

We found that the performance of EMDP2 graduates was comparable to their peers who followed more traditional and other alternative paths to medical school. For example, regression models showed that EMDP2 status was not a statistically significant predictor of average clerkship NBME exam score, nor of USMLE Step 1 failure.

CONCLUSION

EMDP2 graduates performed on a par with their medical school peers, and EMDP2 status does not appear to influence NBME or USMLE performance. EMDP2 provides a focused curriculum and addresses the mandate to make medical education opportunities available to a more diverse population.

Surveying the Middle Ground: A Thematic Analysis of the Medical School Applications of Standard Performers

INTRODUCTION

As gatekeepers to the medical profession, admissions committees make important decisions through the analysis of quantitative (e.g. test scores and grade point averages) and qualitative (e.g. letters of recommendation and personal statements) data. One area of the application that deserves additional study is the Work and Activities section, where students describe their extracurricular experiences. Previous research has found different themes that sometimes overlap in the applications of exceptional performing and low performing medical students, but it is unclear if these themes are present in the applications of standard performers.

MATERIALS AND METHODS

An exceptional performing medical student is defined as one who was inducted into both the Alpha Omega Alpha Honor Medical Society and the Gold Humanism Honor Society. A low performing medical student is one who was referred to the Student Promotions Committee (SPC) where an administrative action was made. A standard performing medical student is defined as someone who was neither a member of an honor society nor referred to the SPC during medical school. Using a constant comparative method, the applications of standard performers who graduated from the Uniformed Services University between 2017 and 2019 was assessed using exceptional performer themes (success in a practiced activity, altruism, teamwork, entrepreneurship, wisdom, passion, and perseverance) and low performer themes (witnessing teamwork, embellishment of achievement, and description of a future event). The presence of novel themes was also assessed. The absolute number of themes and the diversity of themes were determined. Demographic data were collected (age, gender, number of Medical School Admissions Test [MCAT] attempts, highest MCAT score, and cumulative undergraduate grade point average), and descriptive statistics was performed.

RESULTS

A total of 327 standard performers were identified between 2017 and 2019. No novel themes were found after coding 20 applications. All exceptional performer themes were identified in the population of standard performers. The low performing theme of embellishment of achievement was not found. Standard performers had a lower number of total exceptional performer themes and a lower diversity of exceptional performer themes compared to low and exceptional performers, but standard performers also had a lower number and less diversity of low performing themes compared to low performers.

CONCLUSIONS

This study suggests that the diversity and frequency of exceptional performing themes in a medical school application may be useful in distinguishing between exceptional performers and other performers, though the small sample size limits quantitative conclusions. Low performing themes may be specific to low performers and thus could be useful to admissions committees. Future studies should include a larger sample size and could assess for predictive validity evidence of these exceptional performing and low performing themes through a blinded protocol.

Student Performance on Course Objectives in a First-Year Medical School Foundational Science Course Correlate with USMLE Step 1 Scores

PURPOSE

Early identification of students at risk for poor United States Medical Licensing Examination^® (USMLE) Step 1 examination (Step 1) performance allows medical schools to provide targeted intervention for those students. Therefore, determination of metrics that identify struggling students is necessary for proper intervention. We hypothesize that; 1) student performance on pre-matriculation metrics will correlate with their Molecular and Cellular Foundations of Medicine (FDNS) course performance and 2) student performance in the FDNS course and on specific FDNS course objectives will correlate with their Step 1 performance.

METHODS

This was a retrospective cohort study analyzing data for students matriculating to the University of South Carolina School of Medicine Greenville in 2018 and 2019. Linear regression analysis was conducted to assess the correlation between pre-matriculation metrics, performance in the FDNS course, performance on FDNS objectives, and USMLE Step 1 performance. Adjusted R-squared (adjusted r²) values were compared with a p-value at <0.05.

RESULTS

The FDNS course grade correlated with pre-matriculation metrics of science undergraduate grade point average (uGPA), total uGPA, and the Medical College Admission Test (MCAT), adjusted r² of 0.139, 0.121, 0.223, respectively. The FDNS course grade showed a stronger correlation to USMLE Step 1 performance (adjusted r² = 0.257) than pre-matriculation metrics. USMLE Step 1 performance strongly correlated with FDNS course performance when two objectives, pertaining to anabolic and catabolic processes, regulation of cell cycle, and DNA replication and repair, were combined, adjusted r² of 0.357.

CONCLUSION

The FDNS course grade and performance on specific course objectives could serve as a predictor for USMLE Step 1 performance and provides a more targeted and concise approach to identification of low-performing students and subsequent intervention.