What is the impact of commercial test preparation courses on medical examination performance?

Medical Students' Perceptions and Use of Formal and Informal Curriculum Resources

Introduction  Resource overload describes the feeling medical students experience in choosing formal (faculty-prescribed) and informal study resources (not faculty-prescribed). This study aims to characterize students' use and perceptions of formal and informal study resources to inform their use in medical education. Methods  This is a mixed-methods study utilizing a convenience sample of first-year medical students enrolled at the University of Central Florida College of Medicine during the academic year 2020-2021. A 40-question, five-point Likert scale, survey based on Keller's Attention, Relevance, Confidence, and Satisfaction (ARCS) Model of Motivational Design was distributed to medical students during the end of their first year of medical school. Multivariate analysis of variance determined differences between formal and informal resources for each construct. Interviews were also conducted by first-year medical students and analyzed using thematic analysis. Learning logs were completed during the beginning of the medical students' second year to assess daily study habits. Results  Fifty-one students completed the survey with a response rate of 42.5%. Informal resources scored higher across all constructs: attention (formal: 3.4±1.2, informal: 4.0±1.1; p<.0125), relevance (formal: 3.8±1.1, informal: 4.3±1.0; p<.0125), confidence (formal: 3.2±1.2, informal: 4.1±1.1; p<.0125), satisfaction (formal: 2.8±1.2, informal: 3.6±1.2; p<.0125) (Likert scale 1-5, Mean±SD). Students found formal resources lacked depth and organization while informal resources allowed for concise understanding with retention cues. Learning log data reported similar use of formal and informal resources during week 1 (88.2% formal vs. 87.8% informal) and week 2 (84.6% formal vs. 82.6% informal). Conclusions  Students preferred informal resources based on ARCS constructs. However, the actual usage of formal and informal resources was similar. Formal resources align more with curricular assessments, but informal resources aid student retention and understanding. Therefore, students find both formal and informal resources necessary for success. Faculty should consider integrating informal curriculum resources to optimize student learning.

Final-year medical student mental wellness during preparation for the examination for specialty in Turkey: a cross-sectional survey study

BACKGROUND

In Turkey, most final-year medical students prepare for the Examination for Specialty in Medicine in a high-stress environment. To the best of our knowledge, this is the first study on final-year medical student general psychological distress during preparation for the Examination for Specialty in Turkey. We aim to evaluate psychological distress and understand the variables associated with depression, anxiety, and stress levels among final-year medical students preparing for the Examination for Specialty.

METHODS

A self-reporting, anonymous, cross-sectional survey with 21 items consisting of demographic variables, custom variables directed for this study, and the DASS-21 was utilized. Survey results were expounded based on univariate analysis and multivariate linear regression analysis.

RESULTS

Our study revealed four variables associated with impaired mental wellness among final-year medical students during preparation for the examination for Specialty: attendance to preparatory courses, duration of preparation, consideration of quitting studying, and psychiatric drug usage/ongoing psychotherapy.

DISCUSSION

Considering that physician mental wellness is one of the most crucial determinants of healthcare quality, impaired mental wellness among future physicians is an obstacle to a well-functioning healthcare system. Our study targets researchers and authorities, who should focus on medical student mental wellness, and medical students themselves.

Factors Leading to Successful Performance on U.S. National Licensure Exams for Medical Students: A Scoping Review

PURPOSE

To synthesize the evidence of the factors leading to successful performance on knowledge-based national licensure exams (NLEs) for medical students.

METHOD

The authors conducted a scoping review to summarize the peer-reviewed empiric literature that used United States Medical Licensing Examination (USMLE) Step 1 or Step 2 Clinical Knowledge or Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 or Level 2 Cognitive Evaluation scores as outcomes. The authors searched PubMed and Scopus without date restrictions through April 30, 2021. Two reviewers independently screened and selected studies for inclusion. Data were summarized narratively and with descriptive statistics.

RESULTS

The authors screened 1,185 unique citations and included 233 full-text studies in their review. Of these, 201 (86%) were studies of USMLE exams, 31 (13%) were studies of COMLEX exams, and 1 (0.4%) reported on both. The authors classified 29 studies (12%) as informing NLE preparation, 163 (70%) as attempting to identify predictive variables, and 76 (33%) as using NLE scores for program evaluation. Preparation studies found that the number of practice test items, practice exam scores, and less time in dedicated preparation correlated with higher NLE scores. Use of other commercial resources or study strategies was not consistently associated with higher scores. Predictive studies found the strongest relationships between individuals' performance on past assessments and their NLE scores.

CONCLUSIONS

The factors leading to successful performance on knowledge-based NLEs align with well-known principles from the cognitive sciences. Learners build on existing foundations of knowledge (reflected in their prior academic performance) and are likely to learn more efficiently with testing and spaced learning over time. While commercial test preparation resources are ubiquitous, there is no evidence that a single resource gives students a competitive advantage on NLEs. Developing habits of regular and continuous learning is necessary for clinical practice and successful NLE performance.

Association Between NAPLEX Preparation Program Characteristics and First-Time Pass Rates

Objective. The objectives of this study were to (1) describe characteristics of preparation programs for the North American Pharmacist Licensure Examination (NAPLEX) that are currently used by colleges and schools of pharmacy and (2) evaluate these program characteristics in relation to first-attempt NAPLEX pass rates.Methods. This cross-sectional study was based on an online survey administered between February and March 2020. Assessment leads from 143 PharmD programs were invited to answer questions on their schools' PharmD program characteristics and various aspects of NAPLEX preparation programs. The study included regression analyses to investigate associations between the NAPLEX first-attempt pass rates and PharmD demographic characteristics as well as between first-attempt pass rates and various aspects of the NAPLEX preparation programs. Finally, common themes from open-ended questions were identified.Results. Fifty-eight participants completed the survey out of 132 successfully delivered email invitations (response rate = 44%). Fifty participants (86%) indicated that their PharmD program offers a NAPLEX preparation program. Our data indicate that offering a NAPLEX preparation program was not significantly associated with higher NAPLEX first-attempt pass rates. The analysis identified possible explanations for this lack of association, including student concerns with balancing a preparation program with advanced pharmacy practice experiences (APPEs) and the faculty workload associated with delivering such programs.Conclusion. The current findings show no association between offering a NAPLEX preparation program and NAPLEX first-attempt pass rates. Future research should continue to examine the impact of these programs on individual school pass rates and factors that may enhance student motivation to engage in these programs.

Exploring students' [pre-pandemic] use and the impact of commercial-off-the-shelf learning platforms on students' national licensing exam performance: A focused review - BEME Guide No. 72

BACKGROUND

Commercial-off-the-shelf learning platforms developed for medical education (herein referred to as MedED-COTS) have emerged as a resource used by a majority of medical students to prepare for licensing examinations. As MedED-COTS proliferate and include more functions and features, there is a need for an up-to-date review to inform medical educators on (a) students' use of MedED-COTS outside the formal medical school curriculum, (b) the integration of MedED-COTS into the formal curriculum, and (c) the potential effects of MedED-COTS usage on students' national licensing exam scores in the USA.

METHODS

Due to the limited number of studies published on either the use or integration of MedED-COTS, a focused review of literature was conducted to guide future research and practice. Data extraction and quality appraisal were conducted independently by three reviewers; with disagreements resolved by a fourth reviewer. A narrative synthesis was completed to answer research questions, contextualize results, and identify trends and issues in the findings reported by the studies included in the review.

RESULTS

Results revealed consistent positive correlations between students' use of question banks and their licensing exam performance. The limited number of integration studies, combined with a number of methodological issues, makes it impossible to isolate specific effects or associations of integrated commercial resources on standardized test or course outcomes. However, consistent positive correlations, along with students' pervasive use and strong theoretical foundations explaining the results, provide evidence for integrating MedED-COTS into medical school curricula and highlight the need for further research.

CONCLUSIONS

Based on findings, we conclude that students use exam preparation materials broadly and they have a positive impact on exam results; the literature on integration of MedED-COTS into formal curriculum and the use by students of resources outside of exam preparation is scant.

Analysis of Preparatory Courses for the Practical Component of the PLAB Exam for International Medical Graduates in the UK

BACKGROUND

Medical workforce shortages are a major threat to the future of the UK National Health Service (NHS) and countless other healthcare systems globally. Reliance on international medical graduates is likely to continue and may increase, although these doctors face many educational and professional challenges. The Professional and Linguistic Assessment Board (PLAB) test, delivered by the General Medical Council (GMC), is the "gateway" exam that doctors must pass through in order to practice in the UK. Despite the central importance of this exam, no official or accredited training courses are provided or approved by the GMC.

METHODS

This study used online and social media searches to identify and categorise preparatory courses available for the practical (clinical) component of the PLAB assessment.

RESULTS

A total of 13 courses were identified, delivered predominantly in the cities of Manchester and London, ranging from 1 to 28 days in duration and from £24.99 to £649.99 in cost. Most courses were organised by previous PLAB candidates and websites focussed on testimonials from previous participants rather than educational credentials or professional clinical experience of teachers running the courses. Courses were all provided by private education companies, and none were delivered by universities or National Health Service organisations.

DISCUSSION

A variety of preparatory services and courses exist for the PLAB assessment, although the public information about them is variable and limited. Further research is required to examine the quality of these courses and the potential for alternative avenues of training for international medical graduates preparing to practice in the UK.

Understanding Excellence: a Qualitative Analysis of High-Performing Learner Study Strategies

INTRODUCTION

Remediation in medical education tends to focus on the struggling learner. However, understanding successful learners may provide valuable insights to problematic academic behavior. This study explored core study strategies reported by high-performing medical school students.

METHOD

In the Fall of 2018 and 2019, high-performing first- and second-year medical students, defined as those who had performed over 90% on a national standardized assessment, were requested via e-mail to describe the study strategies that they believe contributed to their success. Student responses were analyzed using a template-driven approach, which drew upon common frameworks from cognitive learning theory.

RESULTS

Thirty-seven high-performing students provided brief unstructured descriptions of their study strategies. Retrieval practice-the cognitive learning strategy of attempting to draw out information to active recall, typically through means of self-quizzing, practice questions, or flashcards-was the most commonly endorsed strategy, followed by spaced repetition. Use of outside commercial resources was commonly referenced, and approximately one-third endorsed a perception that some form of regularly practiced self-care contributed to their academic success.

DISCUSSION

Retrieval practice, particularly combined with some form of spaced repetition, is often attributed as contributing to the success of high-performing learners. The use of commercial resources bears further exploration and consideration. Finally, the finding that self-care was identified as a contribution to academic performance goes beyond a focus on self-care solely for medical student well-being. This is an important distinction for both medical education and future research in this area.

Diversity and Inclusion-Why Does It Matter

Otolaryngology has historically lagged behind other specialties with respect to diversity, equity and inclusion (DEI) and remains one of the least diverse specialties as it relates to gender, race and ethnicity. Strategies aimed at increasing DEI include programs designed to provide mentorship, coaching, and sponsorship. Pipeline efforts, inclusivity on committees, bi-directional communication, and equal pay are additional DEI efforts that have been successful in recruiting and retaining those under-represented in medicine (URiM). Closing the equity gap requires commitment; daily action and measuring progress is required. Finally, use feedback to make refinements as opportunities exist to continually improve DEI efforts.

The economic costs of selecting medical students: An Australian case study

CONTEXT

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained.

METHODS

A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated.

RESULTS

A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%).

CONCLUSIONS

The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

Demographic and socioeconomic characteristics of Canadian medical students: a cross-sectional study

BACKGROUND

While the importance of medical students' demographic characteristics in influencing the scope and location of their future practice is recognized, these data are not systematically collected in Canada. This study aimed to characterize and compare the demographics of Canadian medical students with the Canadian population.

METHODS

Through an online survey, delivered in 2018, medical students at 14 English-speaking Canadian medical schools provided their age, sex, gender identity, ethnicity, educational background, and rurality of the area they grew up in. Respondents also provided information on parental income, occupation, and education as markers of socioeconomic status. Data were compared to the 2016 Canadian Census.

RESULTS

A total of 1388 students responded to the survey, representing a response rate of 16.6%. Most respondents identified as women (63.1%) and were born after 1989 (82.1%). Respondents were less likely, compared to the Canadian Census population, to identify as black (1.7% vs 6.4%) (P < 0.001) or Aboriginal (3.5% vs. 7.4%) (P < 0.001), and have grown up in a rural area (6.4% vs. 18.7%) (P < 0.001). Respondents had higher socioeconomic status, indicated by parental education (29.0% of respondents' parents had a master's or doctoral degree, compared to 6.6% of Canadians aged 45-64), occupation (59.7% of respondents' parents were high-level managers or professionals, compared to 19.2% of Canadians aged 45-64), and income (62.9% of respondents grew up in households with income >$100,000/year, compared to 32.4% of Canadians). Assessment of non-response bias showed that our sample was representative of all students at English-speaking Canadian medical schools with respect to age, though a higher proportion of respondents were female. Additionally, there were no differences between early and late respondents with respect to ethnicity, rurality, and parental income, occupation, and education.

CONCLUSIONS

Canadian medical students have different socioeconomic characteristics compared to the Canadian population. Collecting and analyzing these characteristics can inform evidence-based admissions policies.

How Physicians Prepare for Maintenance of Certification Exams: A Qualitative Study

PURPOSE

Little is known about how board-certified physicians prepare for their periodic maintenance of certification (MOC) examinations. This qualitative study explores how physicians experience MOC exam preparation: how they prepare for the exams and decide what to study and how exam preparation compares with what they normally do to keep their medical knowledge current.

METHOD

Between September 2016 and March 2017, the authors interviewed 80 primary care physicians who had recently taken either the American Board of Family Medicine or American Board of Internal Medicine MOC exam. They analyzed transcripts and notes from these interviews looking for patterns and emergent themes, using the constant comparative method and a social practice theory perspective.

RESULTS

Most interviewees studied for their MOC exams by varying from their routines for staying current with medical knowledge, both by engaging with a different scope of information and by adopting different study methods. Physicians described exam preparation as returning to a student/testing mindset, which some welcomed and others experienced negatively or with ambivalence.

CONCLUSIONS

What physicians choose to study bounds what they can learn from the MOC exam process and, therefore, also bounds potential improvements to their patient care. Knowing how physicians actually prepare, and how these activities compare with what they do when not preparing for an exam, can inform debates over the value of requiring such exams, as well as conversations about how certification boards and other key stakeholders in physicians' continuing professional development could improve the MOC process.

First-Year Experience Implementing an Adaptive Learning Platform for First- and Second-Year Medical Students at the Lake Erie College of Osteopathic Medicine

Context

The Lake Erie College of Osteopathic Medicine (LECOM) is a large, diverse medical school spread across 3 campuses, which makes it challenging to deliver comparable learning experiences to all students. Osmosis is a Web and mobile application that can integrate with a variety of existing curricula and, through its content creation and sharing features, can foster student cohesion in an online environment.

Objective

To analyze the first year of use for the Osmosis platform among LECOM students at each campus and to identify barriers to this use.

Methods

Medical education at LECOM is delivered across 3 campus locations using 3 main pathways: the lecture-discussion, problem-based learning, and directed-study pathways. Beginning in the 2016-2017 academic year, all LECOM first-year medical students could use the Osmosis platform free of charge; second-year students were informed about the platform but were required to pay for a subscription. For all students, engagement metrics were tracked within the platform, and periodic student surveys and informal student feedback supplemented these metrics.

Results

Of 1135 LECOM students, 567 (50%) signed up for an account, including 416 (73%) of 573 first-year and 151 (27%) of 562 second-year students. In aggregate, students created 17,901 items and answered 123,050 practice questions and flashcards. Student use of the platform varied considerably, with the highest use at the Erie campus, where the platform was championed by faculty, and among a portion of second-year students preparing for board examinations. Some students were "superusers"; 2 students completed more than 20,000 items, and 5 created more than 1000 items each. The greatest barriers to use of the platform were preferences for previous study methods and lack of time to learn new study habits.

Conclusion

Although the use of the platform across LECOM campuses was uneven, it was greatest when there was overt support by faculty and when students were already motivated to use the platform. The lessons learned during the first year of the program will be used to improve use of the platform. The authors offer their insights into this new technology.

Comprehensive Medical College Admission Test Preparatory Course as a Strategy to Encourage Premedical Students to Pursue Osteopathic Medicine in Rural Areas

Context

Comprehensive Medical College Admission Test (MCAT) preparatory courses could potentially promote interest among premedical students to pursue careers in osteopathic medicine in underserved areas.

Objective

To determine whether a comprehensive 16-week course centered on MCAT preparation and exposure to an osteopathic medical school setting will promote interest among premedical students to become osteopathic physicians in the rural Southwest.

Methods

At the Burrell College of Osteopathic Medicine in Las Cruces, New Mexico, undergraduate premedical students from the surrounding rural and urban areas completed an all-inclusive 16-week MCAT preparation course. Students were required to have completed medical school prerequisite courses and have a minimum 3.0 grade point average. The program included interactive instructional sessions for teaching material pertinent to the MCAT, an introduction to osteopathic philosophy, workshops for application preparation, a tour of the medical school facilities, full-length practice tests, and mock interviews. After course completion with at least 80% attendance, a survey was conducted to identify effectiveness and outcomes of the course. The survey used a Likert scale that ranged from 1 (strongly disagree) to 5 (strongly agree).

Results

Of the 170 student participants, 163 completed the survey. After completing the course, participants felt more knowledgeable about the true nature of medical school and osteopathic medicine (weighted averages, 4.26-4.40) than before the course. Compared with attitudes before taking the course, participants were more inclined to attend an osteopathic medical school and practice rural medicine in the southwestern United States (weighted averages, 4.16-4.45). Participants who completed the course also felt that they were better prepared to take the MCAT (weighted average, 4.37).

Conclusion

Participant knowledge and attitudes about practicing osteopathic medicine were enhanced after they completed the comprehensive MCAT preparatory course. These results suggest that offering similar courses in osteopathic medical schools throughout the country may improve the outlook of creating a diverse physician workforce that provides health care in rural areas.

The construct validity of HPAT-Ireland for the selection of medical students: unresolved issues and future research implications

Aptitude tests are widely used in selection. However, despite certain advantages their use remains controversial. This paper aims to critically appraise five sources of evidence for the construct validity of the Health Professions Admission Test (HPAT)-Ireland, an aptitude test used for selecting undergraduate medical students. The objectives are to identify gaps in the evidence, draw comparisons with other aptitude tests and outline future research directions. Our appraisal of the literature found that stakeholder feedback indicates that there is reasonable evidence for test content validity for two of the three sections of HPAT-Ireland. By contrast the Non-Verbal Reasoning section is widely criticised as having limited relevance to medical school performance and future clinical practice. In terms of concurrent validity there is a significant small to medium, negative correlation with school exit examinations, but not consistently so across all studies (r = -0.18, -0.28, 0.017). Likewise predictive validity studies vary, from negative to moderate strength correlations with examination performance during early years at medical school. Five studies indicate that HPAT-Ireland is supported in principle by the majority of stakeholders. While one consequence of its introduction is that successful applicants are now coming from more diverse academic backgrounds, there is no evidence that the socio-economic background of medical school entrants has been altered significantly. Negative perceptions of unfairness relating to gender, coaching and socio-economics remain. The evidence to date suggests that while there are slight gender differences, initially favouring males, these vary year on year. In conclusion, the attitudes towards, and performance of, HPAT-Ireland is not unlike that of other aptitude tests widely used internationally. The main justifications for its introduction have been achieved, in that Ireland no longer relies exclusively on a single measure of academic record for selection to medical school. However a number of areas require further research and exploration.

Race/Ethnicity in Medical Education: An Analysis of a Question Bank for Step 1 of the United States Medical Licensing Examination

Phenomenon: There is growing concern over racial/ethnic bias in clinical care, yet how best to reduce bias remains challenging, in part because the sources of bias in medical education are poorly understood. One possible source is the routinized use of race/ethnicity in lectures, assessment, and preparatory materials, including question banks for licensing examinations. Because students worldwide use question banks to prepare for the United States Medical Licensing Examination, we examined how race/ethnicity was used in one of the most commonly recommended question banks.

APPROACH

We analyzed the use of race/ethnicity in all 2,211 questions in a question bank for Step 1 of the United States Medical Licensing Examination for the following: the frequency of mentions of racial/ethnic groups, whether the use of race/ethnicity was merely descriptive or was central to any part of the question, and whether the question associated race/ethnicity with genetic difference.

FINDINGS

In sum, 455 of the 2,011 (20.6%) of the questions in the question bank referred to race/ethnicity in the question stem, answer, or educational objective. The frequency of mentions of racial/ethnicity was disproportionate to the U.S.

POPULATION

85.8% referred to White/Caucasians, 9.70% to Black/African Americans, 3.16% to Asian, 0.633% to Hispanics, and 0.633% to Native Americans. No cases referred to Native Hawaiians/Pacific Islanders. The proportion of mentions of race/ethnicity classified as either a routine descriptor or central to the case varied by racial/ethnic category. The association between genetics and disease in cases also varied by racial/ethnic category. Insights. The routinized use of race/ethnicity with no specific goal in preparation materials, such as question banks, risks contributing to racial bias. The implications of routinized use extend to assessment in medical education. Race/ethnicity should be used only when referring to social experiences of groups relevant to their health, not as a proxy for genetics, social class, or culture.

Preparation for a postgraduate specialty examination by medical students in Turkey: processes and sources of anxiety

PHENOMENON: Interns in Turkey must endeavor to study for a specialty exam during their internship. The preparation process for the specialty exam and the effect of this process on the students' anxiety has not been studied comprehensively. The purpose of this study was to evaluate the interns' preparation time for the specialty exam, their perception of how the preparation process affects their training, and which factors are related to their test anxiety.

APPROACH

A cross-sectional study was conducted with 6th-year students (interns). A questionnaire asked participants to report health status, academic achievement, exam-related anxiety, and trait anxiety. Two open-ended questions asked about views regarding the specialty exam. Multiple linear regression was used to identify the significant predictors of anxiety level due to the exam.

FINDINGS

The average duration of exam preparations of participating interns (n = 214) was 16.8 months and 14.3 hours/week. Participating interns' health status, economic level, perception of academic achievement, time allocated to study for the exam, time remaining until the exam, and trait anxiety level demonstrated a relationship with anxiety level due to the exam (R =.35, R(2) =.13, p <.001). In the open-ended questions, the most frequent opinion regarding the importance of the Examination for Specialty in Medicine was "Value attributed to specialization" (43%). The most frequent response regarding the contribution of studying for the specialty exam to their general professional skills was "Rehearsal/recall." INSIGHTS: Participating interns spent an appreciable amount of time preparing for the specialty exam. Although participating interns value this exam, they appear to believe that preparing for it will contribute only moderately to their professional competencies, while increasing their anxiety level. The internship curriculum, requirements, and timing of the specialty exam should be reconsidered.

Practice effects in medical school entrance testing with the undergraduate medicine and health sciences admission test (UMAT)

BACKGROUND

The UMAT is widely used for selection into undergraduate medical and dental courses in Australia and New Zealand (NZ). It tests aptitudes thought to be especially relevant to medical studies and consists of 3 sections - logical reasoning and problem solving (UMAT-1), understanding people (UMAT-2) and non-verbal reasoning (UMAT-3). A substantial proportion of all candidates re-sit the UMAT. Re-sitting raises the issue as to what might be the precise magnitude and determinants of any practice effects on the UMAT and their implications for equity in subsequent selection processes.

METHODS

Between 2000 and 2012, 158,909 UMAT assessments were completed. From these, 135,833 cases were identified where a candidate had sat once or more during that period with 117,505 cases (86.5%) having sat once, 14,739 having sat twice (10.9%), 2,752 thrice (2%) and 837, 4 or more times (0.6%). Subsequent analyses determined predictors of multiple re-sits as well as the magnitude and socio-demographic determinants of any practice effects.

RESULTS

Increased likelihood of re-sitting the UMAT twice or more was predicted by being male, of younger age, being from a non-English language speaking background and being from NZ and for Australian candidates, being urban rather than rurally based. For those who sat at least twice, the total UMAT score between a first and second attempt improved by 10.7 ± 0.2 percentiles, UMAT-1 by 8.3 ± 0.2 percentiles, UMAT-2 by 8.3 ± 0.2 percentiles and UMAT-3 by 7.7 ± 0.2 percentiles. An increase in total UMAT percentile score on re-testing was predicted by a lower initial score and being a candidate from NZ rather than from Australia while a decrease was related to increased length of time since initially sitting the test, older age and non-English language background.

CONCLUSIONS

Re-sitting the UMAT augments performance in each of its components together with the total UMAT percentile score. Whether this increase represents just an improvement in performance or an improvement in understanding of the variables and therefore competence needs to be further defined. If only the former, then practice effects may be introducing inequity in student selection for medical or dental schools in Australia or NZ.

Socio-economic predictors of performance in the Undergraduate Medicine and Health Sciences Admission Test (UMAT)

BACKGROUND

Entry from secondary school to Australian and New Zealand undergraduate medical schools has since the late 1990's increasingly relied on the Undergraduate Medicine and Health Sciences Admission Test (UMAT) as one of the selection factors. The UMAT consists of 3 sections - logical reasoning and problem solving (UMAT-1), understanding people (UMAT-2) and non-verbal reasoning (UMAT-3). One of the goals of using this test has been to enhance equity in the selection of students with the anticipation of an increase in the socioeconomic diversity in student cohorts. However there has been limited assessment as to whether UMAT performance itself might be influenced by socioeconomic background.

METHODS

Between 2000 and 2012, 158,909 UMAT assessments were completed. From these, 118,085 cases have been identified where an Australian candidate was sitting for the first time during that period. Predictors of the total UMAT score, UMAT-1, UMAT-2 and UMAT-3 scores were entered into regression models and included gender, age, school type, language used at home, deciles for the Index of Relative Socioeconomic Advantage and Disadvantage score, the Accessibility/Remoteness Index of Australia (ARIA), self-identification as being of Aboriginal or Torres Strait Islander origin (ATSI) and current Australian state or territory of abode.

RESULTS

A lower UMAT score was predicted by living in an area of relatively higher social disadvantage and lower social advantage. Other socioeconomic indicators were consistent with this observation with lower scores in those who self-identified as being of ATSI origin and higher scores evident in those from fee-paying independent school backgrounds compared to government schools. Lower scores were seen with increasing age, female gender and speaking any language other than English at home. Divergent effects of rurality were observed, with increased scores for UMAT-1 and UMAT-2, but decreasing UMAT-3 scores with increasing ARIA score. Significant state-based differences largely reflected substantial socio-demographic differences across Australian states and territories.

CONCLUSIONS

Better performance by Australian candidates in the UMAT is linked to an increase in socio-economic advantage and reduced disadvantage.This observation provides a firm foundation for selection processes at medical schools in Australia that have incorporated affirmative action pathways to quarantine places for students from areas of socio-economic disadvantage.

The effect of commercial coaching on selection test performance

AIM

To investigate whether commercial coaching affects performance in the Undergraduate Medical and Health Sciences Admission Test (UMAT).

METHOD

The effect of coaching on the UMAT scores of 402 final high school students applying for medicine was examined using t-tests and multiple regression analyses that controlled for academic performance, high school type and gender.

RESULTS

Over half (56.2%) of the participants had enrolled in commercial coaching. Coaching had no influence on UMAT Section 1 or 2, even after controlling for academic performance, gender and type of high school. UMAT Section 3 was significantly higher for coached students, but this was only the case for high ability students. Coaching had a slight negative effect for lower ability students. Those attending academically selective high school also had higher UMAT scores.

CONCLUSION

The results have implications for the fairness and validity of the UMAT. The differential effect of coaching possibly occurred because high academic ability students are able to learn solution rules for non-verbal tests. We propose that the effect of school type relates to a competitive culture that drives extended test practice in academically selective schools.

Standardized patient-narrated web-based learning modules improve students' communication skills on a high-stakes clinical skills examination

BACKGROUND

Use of web-based standardized patient (SP) modules is associated with improved medical student history-taking and physical examination skills on clinical performance examinations (CPX), but a benefit for communication skills has not been shown.

AIM

We describe an innovative web-based SP module using detailed SP and faculty commentary to teach communication skills.

SETTING

A public medical school in 2008-2009.

PARTICIPANTS

Fourth-year medical students.

PROGRAM DESCRIPTION

A 90-minute web-based module with three simulated clinical encounters was narrated by an expert clinician and SP to explain expected history-taking, physical examination, and communication skills behaviors. All 147 students were encouraged to review the module one month before the CPX.

PROGRAM EVALUATION

One hundred and six students (72%) viewed the web-based module. Students who watched the module performed significantly higher on the CPX communication score (+2.67%, p < 0.01) and overall score (+2.12%, p = 0.03), even after controlling for USMLE Step 1 and clerkship summary ratings. Use of the module did not significantly affect history/physical examination scores (+1.89%, p = 0.12).

DISCUSSION

Students who watched an optional web-based SP module prior to the CPX performed higher than those who did not on communication skills. The web-based module appears to be an effective CPX preparatory activity to enhance communication performance.

Effects of comprehensive educational reforms on academic success in a diverse student body

CONTEXT

Calls for medical curriculum reform and increased student diversity in the USA have seen mixed success: performance outcomes following curriculum revisions have been inconsistent and national matriculation of under-represented minority (URM) students has not met aspirations. Published innovations in curricula, academic support and pipeline programmes usually describe isolated interventions that fail to affect curriculum-level outcomes.

METHODS

United States Medical Licensing Examination (USMLE) Step 1 performance and graduation rates were analysed for three classes of medical students before (matriculated 1995-1997, n=517) and after (matriculated 2003-2005, n=597) implementing broad-based reforms in our education system. The changes in pipeline recruitment and preparation programmes, instructional methods, assessment systems, academic support and board preparation were based on sound educational principles and best practices.

RESULTS

Post-reform classes were diverse with respect to ethnicity (25.8% URM students), gender (51.8% female), and Medical College Admissions Test (MCAT) score (range 20-40; 24.1% scored ≤ 25). Mean±standard deviation MCAT scores were minimally changed (from 27.2±4.7 to 27.8±3.6). The Step 1 failure rate decreased by 69.3% and mean score increased by 14.0 points (effect size: d=0.67) overall. Improvements were greater among women (failure rate decreased by 78.9%, mean score increased by 15.6 points; d=0.76) and URM students (failure rate decreased by 76.5%, mean score increased by 14.6 points; d=0.74), especially African-American students (failure rate decreased by 93.6%, mean score increased by 20.8 points; d=1.12). Step 1 scores increased across the entire MCAT range. Four- and 5-year graduation rates increased by 7.1% and 5.8%, respectively.

CONCLUSIONS

The effect sizes in these performance improvements surpassed those previously reported for isolated interventions in curriculum and student support. This success is likely to have resulted from the broad-based, mutually reinforcing nature of reforms in multiple components of the education system. The results suggest that a narrow reductionist view of educational programme reform is less likely to result in improved educational outcomes than a system perspective that addresses the coordinated functioning of multiple aspects of the academic enterprise.

Does a trauma course improve resident performance on the trauma domain of the OITE?

BACKGROUND

Lecture-based courses are common in orthopaedic surgery; however, their effectiveness in improving medical knowledge has not been widely assessed. The Orthopaedic Trauma Association conducts a resident trauma course that occurs four to six weeks prior to the Orthopaedic In-Training Examination (OITE). The purposes of this study were to determine whether residents who had taken this course performed better on the trauma domain of the OITE and to compare the effect of the course with the effect of a focused rotation in orthopaedic trauma.

METHODS

The percentile scores on the trauma domain of the OITE for forty-seven residents in postgraduate year (PGY) 2 were reviewed. Twenty-nine residents who took a trauma course within six weeks before the OITE were compared with eighteen residents who took the course after the OITE. As a secondary analysis, the effect of a trauma rotation for the same PGY-2 residents before and after the OITE was also assessed. Baseline orthopaedic knowledge was assessed with use of the overall OITE percentage scores.

RESULTS

The trauma domain scores were not significantly better among the residents who took the course (mean and standard deviation, 62.2 ± 9.9) compared with those who had not taken it (mean, 59.2 ± 6.2) (p = 0.2). A significant difference (p = 0.04) was found for a trauma rotation, with the residents who had a trauma rotation scoring better (mean, 64.5 ± 6.8) than the residents who had not (mean, 58.8 ± 8.9). The subgroup of residents who had both a course and a trauma rotation before the test scored significantly better (p = 0.01) than the subgroup that had neither of these experiences before the test (mean, 65.6 ± 8.0 and 56.9 ± 6.4, respectively).

CONCLUSIONS

An orthopaedic trauma course alone does not enhance an orthopaedic resident's trauma medical knowledge as assessed on a standardized test. However, a trauma rotation before, and in particular the combination of both a course and a trauma rotation, can improve trauma test scores.

Impacting student anxiety for the USMLE Step 1 through process-oriented preparation

BACKGROUND

Standardized examinations are the key components of medical education. The USMLE Step 1 is the first of these important milestones. Success on this examination requires both content competency and efficient strategies for study and review. Students employ a wide variety of techniques in studying for this examination, with heavy reliance on personal study habits and advice from other students. Nevertheless, few medical curricula formally address these strategies.

METHODS

In response to student-generated critique at our institution, a five-part seminar series on process-oriented preparation was developed and implemented to address such concerns. The series focused on early guidance and preparation strategies for Step 1 and the many other important challenges in medical school. Emphasis was placed on facilitating conversation and mentorship opportunities between students.

RESULTS & CONCLUSIONS

A profoundly positive experience was reported by our medical students that included a decreased anxiety level for the Step 1 examination.

Correlation between an email based board review program and American Board of Pediatrics General Pediatrics Certifying Examination scores

OBJECTIVE

To investigate the impact of a weekly email based board review course on individual resident performance on the American Board of Pediatrics (ABP) General Pediatrics Certifying Examination for pediatric residents and, specifically, residents with low ABP In-training Examination (ITE) scores.

METHODS

Weekly board-type questions were emailed to all pediatric residents from 2004-2007. Responses to board-type questions were tracked, recorded, and correlated with ITE scores and ABP General Pediatrics Certifying Examination Scores.

RESULTS

With regard to total number of questions answered, only total number of questions answered correctly had a significant positive correlation with standard board scores (n = 71, r = 0.24, p = 0.047). For "at risk" residents with ITE scores <or= 200 (n = 21), number of questions answered in PL 3 year (r = 0.51, p = 0.018) and number of questions answered correctly for all PL years (r = 0.59, p = 0.005) had significant positive correlations with standard board scores.

CONCLUSIONS

Participating regularly in the email-based board review course, answering board style questions, and answering correctly to board style questions were associated with higher standard board scores. This benefit existed for all but was especially prominent among those with poor in-training examination scores.

A survey of collection development for United States Medical Licensing Examination (USMLE) and National Board Dental Examination (NBDE) preparation material

OBJECTIVE

The research sought to ascertain medical and dental libraries' collection development policies, evaluation methods, purchase decisions, and issues that relate to print and electronic United States Medical Licensing Examination (USMLE) and National Board Dental Examination (NBDE) preparation materials.

METHODS

The investigators surveyed librarians supporting American Association of Medical Colleges (AAMC)-accredited medical schools (n = 58/125) on the USMLE and librarians supporting American Dental Association (ADA)-accredited dental schools (n = 23/56) on the NBDE. The investigators analyzed the data by cross-tabulating and filtering the results using EFM Continuum web survey software. Investigators also surveyed print and electronic USMLE and NBDE preparation materials from 2004-2007 to determine the number of publications and existence of reviews.

RESULTS

A majority of responding AAMC libraries (62%, n = 58) provide at least 1 electronic or online USMLE preparation resource and buy an average of 11.6 print USMLE titles annually. Due to a paucity of NBDE print and electronic resources, ADA libraries bought significantly fewer print resources, and only 1 subscribed to an electronic resource. The most often reported evaluation methods for both populations were feedback from medical or dental students, feedback from medical or dental faculty, and online trials. Some AAMC (10%, n = 58) and ADA libraries (39%, n = 23) libraries reported that no evaluation of these materials occured at their libraries.

CONCLUSIONS

From 2004-2007, publishers produced 45 USMLE preparation resources (total n = 546) to every 1 NBDE preparation resource (total n = 12). Users' needs, institutional missions and goals, financial status, and official collection policies most often underlie decisions to collect or not collect examination preparation materials. Evaluating the quality of examination preparation materials can be problematic due to lack of published reviews, lack of usability testing by libraries, and librarians' and library users' unfamiliarity with the actual content of examinations. Libraries must integrate faculty and students into the purchase process to make sure examination preparation resources of the highest quality are purchased.

Comprehensive changes in the learning environment: subsequent step 1 scores of academically at-risk students

BACKGROUND

During the past 10 years at our institution, a number of changes have been instituted in the learning environment, including instructional techniques, assessment methods, academic support, and explicit board preparation.

METHOD

The authors studied the Step 1 performance of students with MCAT scores of 20 to 25 in our former and current curricula. Effect sizes were calculated for score improvement using adjusted means from ANCOVA with covariates of MCAT and age.

RESULTS

The overall effect size was 0.48, with larger effects seen for underrepresented minority students overall (d = 0.64) and African American students especially (d = 0.77), representing medium to large effects. Overall failure rates decreased by two thirds.

CONCLUSIONS

Comprehensive changes in the learning environment were followed by substantial improvement in Step 1 performance among academically at-risk students.

A compromise method to facilitate under-represented minority admissions to medical school

Ranked purely on the basis of the available reliable and valid measures, the demographic composition of incoming medical students fails to meet societal concerns regarding under-represented minorities (URM). Admissions offices are squeezed, administratively and legally, between honouring objectively measured individual merit and addressing societal needs. Until more successful recruitment and preparation of URM applicants closes that gap, compromise methods balancing those discordant ideals can be applied. Indeed, as progress over time decreases the relative societal need for affirmative action, compromise methods are mandated by no less a voice than that of the United States Supreme Court. This article proffers one such method, based upon the work of Hofstee, and demonstrates how it can be practically applied to a specific URM challenge. As perspective, past and present philosophical approaches to this discordance are categorized.

Competency-based assessment and cultural compression in medical education: lessons from educational anthropology

OBJECTIVE

This paper explores the thesis that medical education is the cultural transmission to learners of specific values, which are increasingly expressed as graduation competencies. As testing is a powerful way to transmit cultural values to learners in a brief period of time, competency-based assessments can be an instrument of cultural compression in medical education.

METHODS

The author reviewed medical literature to illustrate the concepts from educational anthropology, led the process one medical school used to develop its list of graduation competencies, and conducted a citation search about competency domains.

RESULTS

There is support in the literature for viewing medical education as an example of cultural transmission and compression and for the assertion that testing influences student behaviour. The graduation competency statements developed by the school reflect traditional and emergent values. The citation search data confirmed that some competency domains reflected traditional values, while others reflected more emergent values.

CONCLUSION

Concepts from educational anthropology are relevant to medical education and provide perspectives for understanding contemporary issues such as competency-based assessments.