Demographic disparities in dental school selection: An analysis of current UK practices

INTRODUCTION

Specific social groups remain under-represented within dentistry. While the University Clinical Aptitude Test (UCAT) aims to widen participation in under-represented social groups, there is no evidence in dental education that this aim is being met.

MATERIALS AND METHODS

Data over two admission cycles (2012 and 2013), including 3246 applicants to 10 UK dental schools, were analysed. Applicant and selected pools were compared to the UK population. Multiple logistic regression was used to investigate the association between demographic variables and UCAT and receiving an offer of a place at dental school.

RESULTS

Over-representation of Female, Asian, least deprived and grammar school groups were found in applicant and selected pools compared with the UK population. White ethnic applicants were significantly more selected than Black (OR 0.25), Asian (OR 0.57) and Mixed (OR 0.80) ethnicities, while least deprived applicants were significantly more selected than most deprived (OR 0.59). Grammar school education increased odds of selection by 1.8 when compared to state school. The addition of UCAT to the model for applicants reduced ethnic disparities but led to disparities between other groups.

CONCLUSION

Current widening participation practices focus on attracting applicants from lower socio-economic groups. However, this study showed that ethnicity, sex and educational background biases also affect demographic diversity in dentistry. The UCAT shows promise in levelling the playing field; however, widening access measures will only succeed if selection committees radically change selection processes to address the systemic biases, enabling the dentists of tomorrow to represent the society they serve.

The added value of free preparatory activities for widening access to medical education: a multi-cohort study

BACKGROUND

Medical schools are reported to be less accessible to students with non-traditional backgrounds. These students face barriers when applying for and transitioning to medical school, which may be reduced by offering free preparatory activities. By equalizing access to resources, these activities are expected to reduce disparities in selection outcomes and early academic performance. In the present study, four free institutionally-provided preparatory activities were evaluated by comparing the demographic composition of participating and non-participating applicants. Additionally, the association between participation and selection outcomes and early academic performance was investigated for subgroups (based on sex, migration background and parental education).

METHODS

Participants were applicants to a Dutch medical school in 2016-2019 (N = 3592). Free preparatory activities included Summer School (N = 595), Coaching Day (N = 1794), Pre-Academic Program (N = 217), and Junior Med School (N = 81), supplemented with data on participation in commercial coaching (N = 65). Demographic compositions of participants and non-participants were compared using chi-squared tests. Regression analyses were performed to compare selection outcomes (curriculum vitae [CV], selection test score, probability of enrolment) and early academic performance (first-course grade) between participants and non-participants of demographic subgroups, controlling for pre-university grades and participation in other activities.

RESULTS

Generally, no differences in sociodemographic compositions of participants and non-participants were found, but males participated less often in Summer School and Coaching Day. Applicants with a non-Western background participated less often in commercial coaching, but the overall participation rate was low and participation had negligible effects on selection outcomes. Participation in Summer School and Coaching Day were stronger related with selection outcomes. In some cases, this association was even stronger for males and candidates with a migration background. After controlling for pre-university grades, none of the preparatory activities were positively associated with early academic performance.

CONCLUSIONS

Free institutionally-provided preparatory activities may contribute to student diversity in medical education, because usage was similar across sociodemographic subgroups, and participation was positively associated with selection outcomes of underrepresented and non-traditional students. However, since participation was not associated with early academic performance, adjustments to activities and/or curricula are needed to ensure inclusion and retention after selection.

An assessment of the impact of formal preparation activities on performance in the University Clinical Aptitude Test (UCAT): a national study

BACKGROUND

Previous studies have shown performance in the University Clinical Aptitude Test (UCAT) to be associated with measures of candidate socio-economic advantage such as parental occupation and type of school attended. It is possible that access to preparation support and materials may in part explain these associations. In this paper we determine whether use of formal preparation resources is associated with higher UCAT scores and whether differences in use of preparation resources exist between socio-demographic groups.

METHODS

After completing the 2017 UCAT UK school-leaver candidates (n = 14,332) were asked to answer a questionnaire regarding their use of official UCAT and commercial resources, school-based support, and time spent preparing. Multiple linear and logistic regression models were used to evaluate the associations between preparedness, demographic characteristics and UCAT performance.

RESULTS

Five thousand, four hundred thirty-nine (38%) candidates responded to the questionnaire. Use of freely available UCAT official practice tests, paid commercial materials, attendance at school-based preparation courses and spending more time preparing were significantly associated with higher UCAT scores. Candidates who were from less deprived backgrounds and attending independent or grammar schools were significantly more likely to use paid commercial materials and spend longer preparing.

CONCLUSIONS

Reported use of preparation resources varies between candidates from different socio-demographic backgrounds and is associated independently with performance in the UCAT. Increasing the availability of freely available resources may mitigate some of these differences.

Predictive validity of admission criteria in predicting academic performance of medical students: A retrospective cohort study

Introduction

Admission to medical school is one of the most competitive entry points in higher education. Medical school admissions committees need accurate and precise screening tools to select among well-qualified applicants. This study explores data from a cohort of graduated medical students over 6 years to offer a critical perspective on predictive validity in medical school admissions.

Methods

A retrospective cohort study of 160 students was performed to identify the predictive validity of admission criteria for medical students to predict academic performance over 6 years for a cohort of all students enrolled in the medical program during the academic year 2013–2014.

Results

The main results indicated that there was a statistically significant positive correlation between the admission criteria and Students’ performance in Year 1, Year 4, B.Sc. (Bachelor of Medical Science) exam, and Medical Doctor (MD) exam across the 6 years of the medical program, except for the English Test, which showed that there was no significant correlation with average MD exam scores for students who enrolled directly in Year 1. The results related to students who were admitted to the Foundation Program showed that there was no significant correlation between high school Grade Point Average (HSGPA) and their academic performance in Year 1, Year 4, B.Sc. exam, and MD exam. The overall results related to all study samples indicated that all predictor variables correlate significantly with all outcome variables (academic performance), and the results showed that Science test scores demonstrated 27.7, 15.0, 19.7, and 12.6% of variation in Students’ performance in Year 1, Year 4, B.Sc. exam, and MD exam, respectively.

Conclusion

Science test scores were found to be more predictive of academic performance compared to other predictors. Not all the admission criteria used for student selection are good indicators of their achievement in the medical program. It is recommended that other valid and reliable admission tools, such as the multiple mini-interviews and the questionnaire for a candidate’s suitability to follow a problem-based learning curriculum, should be considered.

The change from UMAT to UCAT for undergraduate medical school applicants: impact on selection outcomes

OBJECTIVES

To assess whether the change from the Undergraduate Medical and Health Sciences Admissions Test (UMAT; 1991-2019) to the University Clinical Aptitude Test (UCAT) for the 2020 New South Wales undergraduate medical degree intake was associated with changes in the impact of sex, socio-economic status and remoteness of residence, and professional coaching upon selection for interview.

DESIGN, SETTING, PARTICIPANTS

Cross-sectional study of applicants for the three NSW undergraduate medical programs for entry in 2019 (4114 applicants) or 2020 (4270); 703 people applied for both intakes. Applicants selected for interview were surveyed about whether they had received professional coaching for the selection test.

MAIN OUTCOME MEASURES

Scores on the three sections of the UMAT (2019 entry cohort) and the five subtests of the UCAT (2020 entry); total UMAT and UCAT scores.

RESULTS

Mean scores for UMAT 1 and 3 and for all four UCAT cognitive subtests were higher for men than women; the differences were statistically significant after adjusting for age, socio-economic status, and remoteness. The effect size for sex was 0.24 (95% CI, 0.18-0.30) for UMAT total score, 0.38 (95% CI, 0.32-0.44) for UCAT total score. For the 2020 intake, 2303 of 4270 applicants (53.9%) and 476 of 1074 interviewees (44.3%) were women. The effect size for socio-economic status was 0.47 (95% CI, 0.39-0.54) for UMAT, 0.43 (95% CI, 0.35-0.50) for UCAT total score; the effect size for remoteness was 0.54 (95% CI, 0.45-0.63) for UMAT, 0.48 (95% CI, 0.39-0.58) for UCAT total score. The impact of professional coaching on UCAT performance was not statistically significant among those accepted for interview.

CONCLUSIONS

Women and people from areas outside major cities or of lower socio-economic status perform less well on the UCAT than other applicants. Reviewing the test and applicant quotas may be needed to achieve selection equity.

The economic costs of selecting medical students: An Australian case study

CONTEXT

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained.

METHODS

A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated.

RESULTS

A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%).

CONCLUSIONS

The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI

BACKGROUND

Sociodemographic subgroup differences in multiple mini-interview (MMI) performance have been extensively studied within the MMI research literature, but heterogeneous findings demand a closer look at how specific aspects of MMI design (such as station type) affect these differences. So far, it has not been investigated whether sociodemographic subgroup differences imply that an MMI is biased, particularly in terms of its predictive validity.

METHODS

Between 2010 and 2017, the University Medical Centre Hamburg-Eppendorf (UKE) tested 1438 candidates in an MMI who also provided sociodemographic data and agreed to participate in this study. Out of these, 400 candidates were admitted and underwent a first objective structured clinical examination (OSCE) after one and a half years, including one station assessing communication skills. First, we analyzed the relationship between gender, age, native language and medical family background and MMI station performance including interaction terms with MMI station type (simulation, interview, and group) in a hierarchical linear model. Second, we tested whether the prediction of OSCE overall and communication station performance in particular differed depending on sociodemographic background by adding interaction terms between MMI performance and gender, age and medical family background in a linear regression model.

RESULTS

Young female candidates performed better than young male candidates both at interview and simulation stations. The gender difference was smaller (simulation) or non-significant (interview) in older candidates. There were no gender or age effects in MMI group station performance. All effects were very small, with the overall model explaining only 0.6% of the variance. MMI performance was not related to OSCE overall performance but significantly predicted OSCE communication station performance with no differences in the prediction for sociodemographic subgroups.

CONCLUSIONS

The Hamburg MMI is fair in its prediction of OSCE communication scores. Differences in MMI station performance for gender and age and their interaction with MMI station type can be related to the dimensions assessed at different station types and thus support the validity of the MMI. Rather than being threats to fairness, these differences could be useful for decisions relating to the design and use of an MMI.

Twelve tips for constructing a multiple mini-interview

Health professions the world over value various competencies in their practitioners that are not easily captured by academic measures of performance. As a result, many programs have begun using multiple mini-interviews (MMIs) to facilitate the selection of candidates who are most likely to demonstrate and further develop such qualities. In this twelve-tips article, the authors offer evidence- and experience-based advice regarding how to construct an MMI that is fit for purpose. The tips are provided chronologically, offering guidance regarding how one might conceptualize their goals for creating an MMI, how to establish a database of stations that are context appropriate, and how to prepare both candidates and examiners for their task. While MMIs have been shown to have utility in many instances, the authors urge caution against over-generalization by stressing the importance of post-MMI considerations including data monitoring and integration between one's admissions philosophy and one's curricular efforts.

Retest effects in medical selection interviews

CONTEXT

Repetition of a cognitive ability test is known to increase scores, but almost no research has examined whether similar improvement occurs with repetition of interviews. Retest effects can change the rank order of candidates and reduce the test's criterion validity. Because interviews are widely used to select medical students and postgraduate trainees, and because applicants apply to multiple programmes and often reapply if unsuccessful, the potential for retest effects needs to be understood.

OBJECTIVES

This study was designed to identify if retest improvements occur when candidates undertake multiple interviews and, if so, whether the effect is attributable to general interview experience or specific experience and whether repeat testing affects criterion validity.

METHODS

We compared interview scores of applicants who were interviewed for one or more of three independent undergraduate medical programmes in two consecutive years and those who were interviewed in both years for the same programme. Correlations between initial and repeat interview scores and a written test of social understanding were compared.

RESULTS

General experience (being interviewed by multiple programmes) did not produce improvement in subsequent interview performance. There was no evidence of method effect (having prior experience of the multiple mini-interview process). Specific experience (being interviewed by the same programme across 2 years) resulted in a significant improvement in scores for which regression to the mean did not fully account. Criterion validity did not appear to be affected.

CONCLUSIONS

Unsuccessful candidates for medical school who reapply and are re-interviewed on a subsequent occasion at the same institution are likely to increase their scores. The results of this study suggest the increase is probably not attributable to improved ability.

2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions

Selection and recruitment into healthcare education and practice is a key area of interest for educators with significant developments in research, policy, and practice in recent years. This updated consensus statement, developed through a multi-stage process, examines future opportunities and challenges in selection and recruitment. There is both a gap in the literature around and a compelling case for further theoretical and empirical literature to underpin the development of overall selection philosophes and policies and their enactment. More consistent evidence has emerged regarding the quality of different selection methods. Approaches to selection are context-dependent, requiring the consideration of an institution's philosophy regarding what they are trying to achieve, the communities it purports to serve, along with the system within which they are used. Diversity and globalization issues continue to be critically important topics. Further research is required to explore differential attainment and explain why there are substantial differences in culturally acceptable ways of approaching diversity and widening access. More sophisticated evaluation approaches using multi-disciplinary theoretical frameworks are required to address the issues. Following a discussion of these areas, 10 recommendations are presented to guide future research and practice and to encourage debate between colleagues across the globe.

Medical School Applicant Characteristics Associated With Performance in Multiple Mini-Interviews Versus Traditional Interviews: A Multi-Institutional Study

PURPOSE

To examine applicant characteristics associated with multiple mini-interview (MMI) or traditional interview (TI) performance at five California medical schools.

METHOD

Of the five California Longitudinal Evaluation of Admission Practices consortium schools, three used TIs and two used MMIs. Schools provided retrospective data on 2011-2013 admissions cycle interviewees: age, gender, race/ethnicity (underrepresented in medicine [UIM] or not), disadvantaged (DA) status, undergraduate GPA, Medical College Admission Test (MCAT) score, and interview score (standardized as z score; mean = 0; SD = 1). Adjusted linear regression analyses, stratified by interview type, examined associations with interview performance.

RESULTS

The 4,993 applicants who completed 7,516 interviews included 931 (18.6%) UIM and 962 (19.3%) DA individuals; 3,226 (64.6%) had only 1 interview. Mean age was 24.4 (SD = 2.7); mean GPA and MCAT score were 3.72 (SD = 0.22) and 33.6 (SD = 3.7), respectively. Older age, female gender, and number of prior interviews were associated with better performance on both MMIs and TIs. Higher GPA was associated with lower MMI scores (z score, per unit GPA = -0.26; 95% CI = -0.45, -0.06) but unrelated to TI scores. DA applicants had higher TI scores (z score = 0.17; 95% CI = 0.07, 0.28) but lower MMI scores (z score = -0.18; 95% CI = -0.28, -0.08) than non-DA applicants. Neither UIM status nor MCAT score was associated with interview performance.

CONCLUSIONS

These findings have potentially important workforce implications, particularly regarding MMI performance of DA applicants, and illustrate the need for other multi-institutional studies.

A systematic review of stakeholder views of selection methods for medical schools admission

BACKGROUND

The purpose of this paper is to systematically review the literature with respect to stakeholder views of selection methods for medical school admissions.

METHODS

An electronic search of nine databases was conducted between January 2000-July 2014. Two reviewers independently assessed all titles (n = 1017) and retained abstracts (n = 233) for relevance. Methodological quality of quantitative papers was assessed using the MERSQI instrument. The overall quality of evidence in this field was low. Evidence was synthesised in a narrative review.

RESULTS

Applicants support interviews, and multiple mini interviews (MMIs). There is emerging evidence that situational judgement tests (SJTs) and selection centres (SCs) are also well regarded, but aptitude tests less so. Selectors endorse the use of interviews in general and in particular MMIs judging them to be fair, relevant and appropriate, with emerging evidence of similarly positive reactions to SCs. Aptitude tests and academic records were valued in decisions of whom to call to interview. Medical students prefer interviews based selection to cognitive aptitude tests. They are unconvinced about the transparency and veracity of written applications. Perceptions of organisational justice, which describe views of fairness in organisational processes, appear to be highly influential on stakeholders' views of the acceptability of selection methods. In particular procedural justice (perceived fairness of selection tools in terms of job relevance and characteristics of the test) and distributive justice (perceived fairness of selection outcomes in terms of equal opportunity and equity), appear to be important considerations when deciding on acceptability of selection methods. There were significant gaps with respect to both key stakeholder groups and the range of selection tools assessed.

CONCLUSIONS

Notwithstanding the observed limitations in the quality of research in this field, there appears to be broad concordance of views on the various selection methods, across the diverse stakeholders groups. This review highlights the need for better standards, more appropriate methodologies and for broadening the scope of stakeholder research.

Response distortion on personality tests in applicants: comparing high-stakes to low-stakes medical settings

The current study examined the degree to which applicants applying for medical internships distort their responses to personality tests and assessed whether this response distortion led to reduced predictive validity. The applicant sample (n = 530) completed the NEO Personality Inventory whilst applying for one of 60 positions as first-year post-graduate medical interns. Predictive validity was assessed using university grades, averaged over the entire medical degree. Applicant responses for the Big Five (i.e., neuroticism, extraversion, openness, conscientiousness, and agreeableness) and 30 facets of personality were compared to a range of normative samples where personality was measured in standard research settings including medical students, role model physicians, current interns, and standard young-adult test norms. Applicants had substantially higher scores on conscientiousness, openness, agreeableness, and extraversion and lower scores on neuroticism with an average absolute standardized difference of 1.03, when averaged over the normative samples. While current interns, medical students, and especially role model physicians do show a more socially desirable personality profile than standard test norms, applicants provided responses that were substantially more socially desirable. Of the Big Five, conscientiousness was the strongest predictor of academic performance in both applicants (r = .11) and medical students (r = .21). Findings suggest that applicants engage in substantial response distortion, and that the predictive validity of personality is modest and may be reduced in an applicant setting.

Predicting patterns of change and stability in student performance across a medical degree

CONTEXT

Evidence of predictive validity is essential for making robust selection decisions in high-stakes contexts such as medical student selection. Currently available evidence is limited to the prediction of academic performance at single points in time with little understanding of the factors that might undermine the predictive validity of tests of academic and non-academic qualities considered important for success. This study addressed these issues by predicting students' changing performance across a medical degree and assessing whether factors outside an institution's control (such as the uptake of commercial coaching) impact validity.

METHODS

Three cohorts of students (n = 301) enrolled in an undergraduate medical degree from 2007-2013 were used to identify trajectories of student academic performance using growth mixture modelling. Multinomial logistic regression assessed whether past academic performance, a test of cognitive ability and a multiple mini-interview could predict a student's likely trajectory and whether this predictive validity was different for those who undertook commercial coaching compared with those who didn't.

RESULTS

Among the medical students who successfully graduated (n = 268), four unique trajectories of academic performance were identified. In three trajectories, performance changed at the time when learning became more self-directed and focused on clinical specialties. Scores on all selection tests, with the exception of a test of abstract reasoning, significantly affected the odds of following a trajectory that was consistently below average. However, selection tests could not distinguish those whose performance improved across time from those whose performance declined after an average start. Commercial coaching increased the odds of being among the below-average performers, but did not alter the predictive validity of the selection tests.

CONCLUSION

Identifying distinct groups of students has important implications for selection, but also for educating medical students. Commercial coaching may result in selecting students who are less suited for coping with the rigours of medical studies.

The effect of personality traits on undergraduate dental students' performance in multiple mini interviews

BACKGROUND

Multiple mini interviews (MMI) is a structured student selection method where applicants rotate around a series of stations that do not require previously learned knowledge. The Big Five Inventory (BFI) is a tool often used to measure personality traits commonly used in applicant selection.

OBJECTIVE

The aim of the current study was to assess the effect of personality traits using the BFI on undergraduate dental students' performance in MMI.

METHODS

This research was conducted at The Faculty of Dentistry, King Abdulaziz University, Jeddah, Saudi Arabia. All undergraduates applying to the dental program at King Abdulaziz University, Faculty of Dentistry (KAUFD) in two academic years [(2014-2015) and (2015-2016)] undertook the MMI held at KAUFD and completed the BFI inventory with a total number of 350 students. The MMI consisted of five stations while the BFI contained forty-four items. Data were analysed by SPSS version 20, using independent-samples t-test, Mann-Whitney U test, Spearman's rho correlation coefficient and Chi-square test.

RESULTS

female students scored higher than their male counterparts but scores were only significant in the empathy station (p<0.001) in the first year, and in both empathetic skills and team work abilities (p<0.05) in the second. BFI results indicate that male students scored higher on agreeableness (p<0.003) and lower on neuroticism (p<0.001) in the first year while female students scored higher on agreeableness and conscientiousness (p<0.001) in the second year. Students of private schools had higher total MMI scores compared to those of public schools (p<0.05).

CONCLUSION

MMI combined with the Big Five Inventory can be a useful tool in the admission process.

The construct validity of HPAT-Ireland for the selection of medical students: unresolved issues and future research implications

Aptitude tests are widely used in selection. However, despite certain advantages their use remains controversial. This paper aims to critically appraise five sources of evidence for the construct validity of the Health Professions Admission Test (HPAT)-Ireland, an aptitude test used for selecting undergraduate medical students. The objectives are to identify gaps in the evidence, draw comparisons with other aptitude tests and outline future research directions. Our appraisal of the literature found that stakeholder feedback indicates that there is reasonable evidence for test content validity for two of the three sections of HPAT-Ireland. By contrast the Non-Verbal Reasoning section is widely criticised as having limited relevance to medical school performance and future clinical practice. In terms of concurrent validity there is a significant small to medium, negative correlation with school exit examinations, but not consistently so across all studies (r = -0.18, -0.28, 0.017). Likewise predictive validity studies vary, from negative to moderate strength correlations with examination performance during early years at medical school. Five studies indicate that HPAT-Ireland is supported in principle by the majority of stakeholders. While one consequence of its introduction is that successful applicants are now coming from more diverse academic backgrounds, there is no evidence that the socio-economic background of medical school entrants has been altered significantly. Negative perceptions of unfairness relating to gender, coaching and socio-economics remain. The evidence to date suggests that while there are slight gender differences, initially favouring males, these vary year on year. In conclusion, the attitudes towards, and performance of, HPAT-Ireland is not unlike that of other aptitude tests widely used internationally. The main justifications for its introduction have been achieved, in that Ireland no longer relies exclusively on a single measure of academic record for selection to medical school. However a number of areas require further research and exploration.

Candidate Evaluation Using Targeted Construct Assessment in the Multiple Mini-Interview: A Multifaceted Rasch Model Analysis

Construct: A 7-station multiple mini-interview (MMI) circuit was implemented and assessed for 214 candidates rated by 37 interviewers (N = 1,498 ratings). The MMI stations were designed to assess 6 specific constructs (adaptability, empathy, integrity, critical thinking, teamwork [receiving instruction], teamwork [giving instruction]) and one open station about the candidate's interest in the school.

BACKGROUND

Despite the apparent benefits of the MMI, construct-irrelevant variance continues to be a topic of study. Refining the MMI to more effectively measure candidate ability is critical to improving our ability to identify and select candidates that are equipped for success within health professions education and the workforce.

APPROACH

Each station assessed a single construct and was rated by a single interviewer who was provided only the name of the candidate and no additional information about the candidate's background, application, or prior academic performance. All interviewers received online and in-person training in the fall prior to the MMI and the morning of the MMI. A 3-facet multifaceted Rasch measurement analysis was completed to determine interviewer severity, candidate ability, and MMI station difficulty and examine how the model performed overall (e.g., rating scale).

RESULTS

Altogether, the Rasch measures explained 62.84% of the variance in the ratings. Differences in candidate ability explained 45.28% of the variance in the data, whereas differences in interviewer severity explained 16.09% of the variance in the data. None of the interviewers had Infit or Outfit mean-square scores greater than 1.7, and only 2 (5.4%) had mean-square scores less than 0.5.

CONCLUSIONS

The data demonstrated acceptable fit to the multifaceted Rasch measurement model. This work is the first of its kind in pharmacy and provides insight into the development of an MMI that provides useful and meaningful candidate assessment ratings for institutional decision making.

The Multiple Mini-Interview as an Admission Tool for a PharmD Program Satellite Campus

Objective. To assess the multiple mini-interview (MMI) as an admission tool for a satellite campus. Methods. In 2013, the MMI was implemented as part of a new admissions model at the UNC Eshelman School of Pharmacy. From fall 2013 to spring 2015, 73 candidates were interviewed by 15 raters on the satellite campus in Asheville, North Carolina. A many-facet Rasch measurement (MFRM) with three facets was used to determine the variance in candidate ratings attributable to rater severity, candidate ability, and station difficulty. Candidates were surveyed to explore their perceptions of the MMI. Results. Rasch measures accounted for 48.3% of total variance in candidate scores. Rater severity accounted for 9.1% of the variance, and candidate ability accounted for 36.2% of the variance. Eighty percent of survey respondents (strongly) agreed that interviewers got to know them based on questions they answered.Conclusion. This study suggests that the MMI is a useful and valid tool for candidate selection at a satellite campus.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

BACKGROUND

In the 11 years since its development at McMaster University Medical School, the multiple mini-interview (MMI) has become a popular selection tool. We aimed to systematically explore, analyze and synthesize the evidence regarding MMIs for selection to undergraduate health programs.

METHODS

The review protocol was peer-reviewed and prospectively registered with the Best Evidence Medical Education (BEME) collaboration. Thirteen databases were searched through 34 terms and their Boolean combinations. Seven key journals were hand-searched since 2004. The reference sections of all included studies were screened. Studies meeting the inclusion criteria were coded independently by two reviewers using a modified BEME coding sheet. Extracted data were synthesized through narrative synthesis.

RESULTS

A total of 4338 citations were identified and screened, resulting in 41 papers that met inclusion criteria. Thirty-two studies report data for selection to medicine, six for dentistry, three for veterinary medicine, one for pharmacy, one for nursing, one for rehabilitation, and one for health science. Five studies investigated selection to more than one profession. MMIs used for selection to undergraduate health programs appear to have reasonable feasibility, acceptability, validity, and reliability. Reliability is optimized by including 7-12 stations, each with one examiner. The evidence is stronger for face validity, with more research needed to explore content validity and predictive validity. In published studies, MMIs do not appear biased against applicants on the basis of age, gender, or socio-economic status. However, applicants of certain ethnic and social backgrounds did less well in a very small number of published studies. Performance on MMIs does not correlate strongly with other measures of noncognitive attributes, such as personality inventories and measures of emotional intelligence.

DISCUSSION

MMI does not automatically mean a more reliable selection process but it can do, if carefully designed. Effective MMIs require careful identification of the noncognitive attributes sought by the program and institution. Attention needs to be given to the number of stations, the blueprint and examiner training.

CONCLUSION

More work is required on MMIs as they may disadvantage groups of certain ethnic or social backgrounds. There is a compelling argument for multi-institutional studies to investigate areas such as the relationship of MMI content to curriculum domains, graduate outcomes, and social missions; relationships of applicants' performance on different MMIs; bias in selecting applicants of minority groups; and the long-term outcomes appropriate for studies of predictive validity.

How effective are selection methods in medical education? A systematic review

CONTEXT

Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non-academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.

METHODS

A formalised literature search was conducted for studies published between 1997 and 2015. A total of 194 articles met the inclusion criteria and were appraised in relation to: (i) selection method used; (ii) research question(s) addressed, and (iii) type of study design.

RESULTS

Eight selection methods were identified: (i) aptitude tests; (ii) academic records; (iii) personal statements; (iv) references; (v) situational judgement tests (SJTs); (vi) personality and emotional intelligence assessments; (vii) interviews and multiple mini-interviews (MMIs), and (viii) selection centres (SCs). The evidence relating to each method was reviewed against four evaluation criteria: effectiveness (reliability and validity); procedural issues; acceptability, and cost-effectiveness.

CONCLUSIONS

Evidence shows clearly that academic records, MMIs, aptitude tests, SJTs and SCs are more effective selection methods and are generally fairer than traditional interviews, references and personal statements. However, achievement in different selection methods may differentially predict performance at the various stages of medical education and clinical practice. Research into selection has been over-reliant on cross-sectional study designs and has tended to focus on reliability estimates rather than validity as an indicator of quality. A comprehensive framework of outcome criteria should be developed to allow researchers to interpret empirical evidence and compare selection methods fairly. This review highlights gaps in evidence for the combination of selection tools that is most effective and the weighting to be given to each tool.

PROGNOSTIC VALUE OF THE MULTIPLE MINI-INTERVIEW FOR EMERGENCY MEDICINE RESIDENCY PERFORMANCE

BACKGROUND

The Multiple Mini-Interview (MMI) uses short, structured contacts, and is known to predict medical school success better than traditional interviews and application materials. Its utility in Emergency Medicine residency selection is untested.

OBJECTIVES

We investigate whether it provides additional information regarding future first-year resident performance that can be useful in resident selection.

METHODS

From three Emergency Medicine residency programs, 71 interns in their first month completed an MMI developed to focus on desirable resident characteristics. Application data were reviewed. First-year resident performance assessments covering the American Council for Graduate Medical Education (ACGME) core competencies, along with professionalism and performance concerns, were obtained. Multiple logistic regressions were employed and MMI correlations were compared with program rank lists and typical selection factors.

RESULTS

An individual's score on the MMI correlated with overall performance (p < 0.05) in single logistic regression. MMI correlated with ACGME individual competencies patient care and procedural skills at a less robust level (p < 0.1), but not with any other outcomes. Rank list position correlated with the diagnostic skill competency (p < 0.05), but no others. Traditional selection factors correlated with overall performance, disciplinary action, patient care, medical knowledge, and diagnostic skills (p < 0.05). MMI was not correlated significantly with the outcomes when included in multiple ordinal logistic regression with other selection factors.

CONCLUSIONS

MMI scores correlate with overall performance, but are not statistically significant when other traditional selection factors were considered. The MMI process seems potentially superior to program rank list at correlating with first-year performance. The MMI may provide additional benefit when examined using a larger and more diverse sample.

Medical school admission test: advantages for students whose parents are medical doctors?

BACKGROUND

Admission candidates especially in medicine do not represent the socio-demographic proportions of the average population: children of parents with an academic background are highly overrepresented, and those with parents who are medical doctors represent quite a large and special group. At Göttingen University Medicine, a new admission procedure was established with the intention to broaden the base of applicants towards including candidates with previous medical training or lower final school grades. With a view to family background, we wished to know whether candidates differ in the test scores in our admission procedure.

METHODS

In February 2014 we asked all admission candidates of Göttingen University Medicine by questionnaire (nine closed, four open questions) about the academic background in their families, specifically, the medical background, school exam grades, and previous medical training as well as about how they prepared for the admission test. We also analysed data from admission scores of this group (semi-structured interview and four multiple mini-interviews). In addition to descriptive statistics, we used a Pearson correlation, means comparisons (t-test, analysis of variance), ANOVA, and a Scheffé test.

RESULTS

In February 2014 nearly half of the applicants (44%) at Göttingen University Medicine had a medical background, most frequently, their parents were physicians. This rate is much higher than reported in the literature. Other socio-demographic baseline data did not differ from the percentages given in the literature. Of all applicants, 20% had previous medical training. The group of applicants with parents who were medical doctors did not show any advantage in either test-scoring (MMI and interview), their individual preparation for the admission test, or in receiving or accepting a place at medical school. Candidates with parents who were medical doctors had scored slightly lower in school exam grades.

CONCLUSION

Our results suggest that there is a self-selection bias as well as a pre-selection for this particular group of applicants. This effect has to be observed during future admission procedures.

Medical student selection criteria and socio-demographic factors as predictors of ultimately working rurally after graduation

BACKGROUND

We have previously demonstrated that both coming from a rural background and spending a year-long clinical rotation in our Rural Clinical School (RCS) have independent and additive effects to increase the likelihood of medical students practicing rurally following graduation. The current study assesses the extent to which medical school selection criteria and/or the socio-demographic profile of medical students may further facilitate or hamper the selection of students ultimately destined for the rural medical workforce.

METHODS

The study comprised 729 students, admitted from secondary school since 1999 and having graduated by 2011, whose actual workplace location in 2014 was classified as either urban or rural using the Australian Health Practitioner Regulation Agency database. Selection factors on entry (score from a standardised interview, percentile scores for the 3 components of the Undergraduate Medicine and Health Sciences Admission Test (UMAT) and prior academic performance as assessed by the Australian Tertiary Admissions Rank) together with socio-demographic factors (age, gender, decile for the Index of Relative Socioeconomic Advantage and Disadvantage (IRSAD)), were examined in relation to ultimate rural destination of practice.

RESULTS

In logistic regression, those practicing in a rural location in 2014 were more likely to have come from the lower 6 IRSAD deciles (OR 2.75, 95% CI 1.44, 5.23, P = 0.002), to be older (OR 1.86, 95% CI 1.09, 3.18, p = 0.023) and to have a lower UMAT-3 (Non-verbal communication) score (OR 0.98, 95% CI 0.97, 0.99, P = 0.005). After further controlling for either rural background or RCS participation, only age and UMAT-3 remained as independent predictors of current rural practice.

CONCLUSIONS

In terms of the socio-demographic profiles of those selected for medical school entry from secondary school, only older age weakly augmented the selection of graduates likely to ultimately work in a rural destination. Among the selection factors, having achieved higher scores in UMAT-3 tended to mitigate this outcome. The major focus in attempts to grow the rural medical workforce should therefore remain on recruiting medical students from a rural background together with providing maximal opportunity for prolonged immersion in rural clinical environments during their training.

Evaluation of the effect of socio-economic status on performance in a Multiple Mini Interview for admission to medical school

INTRODUCTION

Widening participation in Medicine is a key policy priority as it helps promote a diverse and representative workforce and improves patient care. The selection process employed can influence the socio-economic composition of the student cohort and this study therefore evaluated whether Multiple Mini Interview (MMI) performance was influenced by school type or area-level Higher Education (HE) participation rates.

METHODS

MMI performance for all UK applicants was recorded and consent to link performance with socio-economic data was requested using an applicant questionnaire. Station-level and total MMI scores, and offer rates were compared between applicants from non-selective, non-fee-paying schools and from selective and/or fee-paying schools; and between applicants from each quintile of area-level HE participation.

RESULTS

793 applicants were included in the analysis. MMI performance and offer rates were slightly higher for applicants from non-selective, non-fee-paying schools and/or from lower HE participation quintiles, but the effects were small and not statistically significant.

CONCLUSIONS

The MMI did not favour applicants from selective and/or fee-paying schools, or from areas with high HE participation rates. Work to evaluate whether the relationship between MMI and medical school performance is the same for all sub-groups of students is now required.

Multiple mini-interviews: same concept, different approaches

OBJECTIVES

Increasing numbers of educational institutions in the medical field choose to replace their conventional admissions interviews with a multiple mini-interview (MMI) format because the latter has superior reliability values and reduces interviewer bias. As the MMI format can be adapted to the conditions of each institution, the question of under which circumstances an MMI is most expedient remains unresolved. This article systematically reviews the existing MMI literature to identify the aspects of MMI design that have impact on the reliability, validity and cost-efficiency of the format.

METHODS

Three electronic databases (OVID, PubMed, Web of Science) were searched for any publications in which MMIs and related approaches were discussed. Sixty-six publications were included in the analysis.

RESULTS

Forty studies reported reliability values. Generally, raising the number of stations has more impact on reliability than raising the number of raters per station. Other factors with positive influence include the exclusion of stations that are too easy, and the use of normative anchored rating scales or skills-based rater training. Data on criterion-related validities and analyses of dimensionality were found in 31 studies. Irrespective of design differences, the relationship between MMI results and academic measures is small to zero. The McMaster University MMI predicts in-programme and licensing examination performance. Construct validity analyses are mostly exploratory and their results are inconclusive. Seven publications gave information on required resources or provided suggestions on how to save costs. The most relevant cost factors that are additional to those of conventional interviews are the costs of station development and actor payments.

CONCLUSIONS

The MMI literature provides useful recommendations for reliable and cost-efficient MMI designs, but some important aspects have not yet been fully explored. More theory-driven research is needed concerning dimensionality and construct validity, the predictive validity of MMIs other than those of McMaster University, the comparison of station types, and a cost-efficient station development process.

Views of doctors of varying disciplines on HPAT-Ireland as a selection tool for medicine

BACKGROUND

Selection tools for medicine must achieve political validity and enjoy stakeholder acceptability. This qualitative study aimed to establish the perspectives of doctors, from various clinical specialities, on HPAT-Ireland, a new selection tool for undergraduate medical students.

METHODS

Fifteen doctors participated over three iterative cycles of recruitment, interviewing and analysis. Prior to interview, participants sat a practice HPAT-Ireland test. HPAT-Ireland has three sections: (1) Logical reasoning/problem solving; (2) Interpersonal understanding and (3): Non-verbal reasoning.

SUMMARY OF RESULTS

Three themes emerged: job relatedness; utility of HPAT-Ireland and diversity. Sections 1 and 2 were considered very job related however Section 3 was widely criticised for lacking clinical relevance. Doctors did not think that the test would reliably predict future performance. However, one-third felt it was acceptable as a selection tool in conjunction with academic record. Those who found it unacceptable were influenced by its perceived narrow focus, limited job relatedness, potential for socioeconomic bias, impact on gender and potential for negative influence on student diversity.

CONCLUSIONS

A selection tool that does not enjoy the confidence of the medical profession is unlikely to achieve political validity and may ultimately fail, regardless of other objective measures of its effectiveness such as predictive validity.

Multiple mini-interview test characteristics: 'tis better to ask candidates to recall than to imagine

CONTEXT

The multiple mini-interview (MMI), used to facilitate the selection of applicants in health professional programmes, has been shown to be capable of generating reliable data predictive of success. It is a process rather than a single instrument and therefore its psychometric properties can be expected to vary according to the stations generated, the alignment between the stations and the qualities an institution prioritises, and the outcomes used. The purpose of this study was to explore the MMI's test characteristics when station type is manipulated.

METHODS

A 12-station MMI was established in which four stations were presented in three different ways. These included: situational judgement (SJ) stations, in which applicants were asked to imagine what they would do in specific situations; behavioural interview (BI) stations, in which applicants were asked to recall what they did in experienced situations, and free form (FF) stations, which were unstructured in that the examiner was simply given a brief explanation of the intent of the station without further guidance on how to conduct the discussion. Four circuits of the 12 stations were run with one examiner within each station. Candidates and examiners were surveyed regarding their experience. The reliability of the scores derived from the assessment was analysed separately for each station type.

RESULTS

A total of 41 medical school candidates participated after completing the regular admission process. Although the score assigned did not differ across station type, BI stations more reliably differentiated between candidates (g = 0.77) than did the other station types (SJ, g = 0.69; FF, g = 0.66). The correlation between actual MMI scores and BI stations was also greatest (BI, r = 0.57; SJ, r = 0.45; FF, r = 0.42). Candidates' opinions indicated that FF stations were more anxiety-provoking, less clear, and more difficult than structured stations (SJ and BI stations). Examiner opinions indicated equivalence on these measures.

CONCLUSIONS

The results suggest that structuring stations has value, although that value was gained only through the use of BI stations, in which candidates were asked to recall and discuss a specific experience of relevance to the purpose of the interview station.

The Multiple Mini-Interview (MMI) for student selection in health professions training - a systematic review

BACKGROUND

The Multiple Mini-Interview (MMI) has been used increasingly for selection of students to health professions programmes.

OBJECTIVES

This paper reports on the evidence base for the feasibility, acceptability, reliability and validity of the MMI.

DATA SOURCES

CINAHL and MEDLINE STUDY ELIGIBILITY CRITERIA: All studies testing the MMI on applicants to health professions training.

STUDY APPRAISAL AND SYNTHESIS METHODS

Each paper was appraised by two reviewers. Narrative summary findings on feasibility, acceptability, reliability and validity are presented.

RESULTS

Of the 64 citations identified, 30 were selected for review. The modal MMI consisted of 10 stations, each lasting eight minutes and assessed by one interviewer. The MMI was feasible, i.e. did not require more examiners, did not cost more, and interviews were completed over a short period of time. It was acceptable, i.e. fair, transparent, free from gender, cultural and socio-economic bias, and did not favour applicants with previous coaching. Its reliability was reported to be moderate to high, with Cronbach's alpha = 0.69-0.98 and G = 0.55-0.72. MMI scores did not correlate to traditional admission tools scores, were not associated with pre-entry academic qualifications, were the best predictor for OSCE performance and statistically predictive of subsequent performance at medical council examinations.

CONCLUSIONS

The MMI is reliable, acceptable and feasible. The evidence base for its validity against future medical council exams is growing with reports from longitudinal investigations. However, further research is needed for its acceptability in different cultural context and validity against future clinical behaviours.

The impact of preparatory activities on medical school selection outcomes: a cross-sectional survey of applicants to the University of Adelaide Medical School in 2007

BACKGROUND

Selection into medical school is highly competitive with more applicants than places. Little is known about the preparation that applicants undertake for this high stakes process. The study aims to determine what preparatory activities applicants undertake and what difficulties they encounter for each stage of the application process to medical school and in particular what impact these have on the outcome.

METHODS

A cross-sectional survey of 1097 applicants who applied for a place in the University of Adelaide Medical School in 2007 and participated in the UMAT (Undergraduate Medicine and Health Sciences Admission Test) and oral assessment components of the selection process. The main outcome measures were an offer of an interview and offer of a place in the medical school and were analysed using logistic regression.

RESULTS

The odds of a successful outcome increased with each additional preparatory activity undertaken for the UMAT (odds ratio 1.22, 95% confidence interval 1.11 to 1.33; P < 0.001) and the oral assessment (1.36, 1.19 to 1.55; P < 0.001) stage of selection. The UMAT preparatory activities associated with the offer of an interview were attendance of a training course by a private organisation (1.75, 1.35 to 2.27: P < 0.001), use of online services of a private organisation (1.58, 1.23 to 2.04; P < 0.001), and familiarising oneself with the process (1.52, 1.15 to 2.00; p = 0.021). The oral assessment activities associated with an offer of a place included refining and learning a personal resume (9.73, 2.97 to 31.88; P < 0.001) and learning about the course structure (2.05, 1.29 to 3.26; P = 0.022).For the UMAT, applicants who found difficulties with learning for this type of test (0.47, 0.35 to 0.63: P < 0.001), with the timing of UMAT in terms of school exams (0.48, 0.5 to 0.66; P < 0.001) and with the inability to convey personal skills with the UMAT (0.67, 0.52 to 0.86; P = 0.026) were significantly less likely to be offered an interview.

CONCLUSIONS

Medical schools make an enormous effort to undertake a selection process that is fair and equitable and which selects students most appropriate for medical school and the course they provide. Our results indicate that performance in the selection processes can be improved by training. However, if these preparatory activities may be limited to those who can access them, the playing field is not even and increasing equity of access to medical schools will not be achieved.

Do differentials in the support and advice available at UK schools and colleges influence candidate performance in the medical school admissions interview? A survey of direct school leaver applicants to a UK medical school

BACKGROUND

To our knowledge, nothing is known about whether differentials in support and advice during preparation for the interview influence candidate performance and thereby contribute to bias in selection for medical school.

AIM

To assess if differences in advice and support with preparation for the medical school admissions interview given type of school last attended influence interview score achieved by direct school leaver applicants to study on an undergraduate UK medical degree course.

METHODS

Confidential self-completed on-line questionnaire survey.

RESULTS

Interview performance was positively related to whether a teacher, tutor or career advisors at the School or College last attended had advised a respondent to prepare for the interview, had advised about the various styles of medical interview used and the types of questions asked, and what resources were available to help in preparation. Respondents from Private/Independent schools were more likely than those from State schools to have received such advice and support.

CONCLUSIONS

Differentials in access to advice on and support with preparation for the medical school interview may advantage some candidates over others. This inequity would likely be ameliorated by the provision of an authoritative and comprehensive guide to applying to medical school outlining admission requirements and the preparation strategy applicants should use in order to best meet those requirements. The guide could be disseminated to the Principals of all UK schools and colleges and freely available electronic versions signposted in medical school prospectuses and the course descriptor on the Universities and Colleges Admissions Service.

Core personal competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admission process?

Assessing applicants' personal competencies in the admission process has proven difficult because there is not an agreed-on set of personal competencies for entering medical students. In addition, there are questions about the measurement properties and costs of currently available assessment tools. The Association of American Medical College's Innovation Lab Working Group (ILWG) and Admissions Initiative therefore engaged in a multistep, multiyear process to identify personal competencies important to entering students' success in medical school as well as ways to measure them early in the admission process. To identify core personal competencies, they conducted literature reviews, surveyed U.S and Canadian medical school admission officers, and solicited input from the admission community. To identify tools with the potential to provide data in time for pre-interview screening, they reviewed the higher education and employment literature and evaluated tools' psychometric properties, group differences, risk of coaching/faking, likely applicant and admission officer reactions, costs, and scalability. This process resulted in a list of nine core personal competencies rated by stakeholders as very or extremely important for entering medical students: ethical responsibility to self and others; reliability and dependability; service orientation; social skills; capacity for improvement; resilience and adaptability; cultural competence; oral communication; and teamwork. The ILWG's research suggests that some tools hold promise for assessing personal competencies, but the authors caution that none are perfect for all situations. They recommend that multiple tools be used to evaluate information about applicants' personal competencies in deciding whom to interview.

The effect of commercial coaching on selection test performance

AIM

To investigate whether commercial coaching affects performance in the Undergraduate Medical and Health Sciences Admission Test (UMAT).

METHOD

The effect of coaching on the UMAT scores of 402 final high school students applying for medicine was examined using t-tests and multiple regression analyses that controlled for academic performance, high school type and gender.

RESULTS

Over half (56.2%) of the participants had enrolled in commercial coaching. Coaching had no influence on UMAT Section 1 or 2, even after controlling for academic performance, gender and type of high school. UMAT Section 3 was significantly higher for coached students, but this was only the case for high ability students. Coaching had a slight negative effect for lower ability students. Those attending academically selective high school also had higher UMAT scores.

CONCLUSION

The results have implications for the fairness and validity of the UMAT. The differential effect of coaching possibly occurred because high academic ability students are able to learn solution rules for non-verbal tests. We propose that the effect of school type relates to a competitive culture that drives extended test practice in academically selective schools.

Preparation courses for a medical admissions test: effectiveness contrasts with opinion

OBJECTIVES

The Undergraduate Medicine and Health Sciences Admissions Test (UMAT) is used to rank applicants for admission to many medical schools. This study aimed to determine the effects of preparation courses on UMAT performance and on students' perceptions of their performance.

METHODS

We asked students who sat the UMAT twice across two consecutive years to complete an online survey. The survey was administered in 2010 and 2011 to gather information on preparation activities, costs of preparation activities and students' opinions regarding their expected performance. Survey responses were compared with student scores on the second taking of the UMAT, adjusted for prior UMAT scores and university performance.

RESULTS

The study (cohort: n = 263) was sufficiently powered to investigate two forms of preparation: courses offered by MedEntry (a UMAT preparation provider), and tutoring offered by the students' halls of residence. Neither was found to significantly affect UMAT score (p = 0.09 for MedEntry courses; p = 0.50 for halls of residence tutoring). There was no relationship between total time or money spent preparing and UMAT performance. However, students taking MedEntry courses and students spending more money on UMAT preparation were more likely to predict an improved score (p < 0.001 for both). A total of 85% of students improved their score on the second sitting, irrespective of preparation.

CONCLUSIONS

The use of either of two common forms of UMAT preparation does not translate to an improvement in UMAT score. However, in accordance with confirmation bias theory, the association between money spent on preparatory courses and self-assessed predicted score suggests that students' belief in the effectiveness of such courses may be confounded by their financial outlay.

How to select the doctors of the future

Although previous academic performance is acknowledged as the best predictor of achievement in medical school, no one has succeeded in finding a selection method which will choose students who will become doctors with the qualities the community expect. Australian medical schools use various selection methods. It could well be argued that the most important phase in the selection process is informed decision-making by potential applicants. More effort should be made by medical schools to achieve this.

Comparison of UMAT scores and GPA in prediction of performance in medical school: a national study

CONTEXT

Medical schools continue to seek robust ways to select students with the greatest aptitude for medical education, training and practice. Tests of general cognition are used in combination with markers of prior academic achievement and other tools, although their predictive validity is unknown. This study compared the predictive validity of the Undergraduate Medicine and Health Sciences Admission Test (UMAT), the admission grade point average (GPA), and a combination of both, on outcomes in all years of two medical programmes.

METHODS

Subjects were students (n = 1346) selected since 2003 using UMAT scores and attending either of New Zealand's two medical schools. Regression models incorporated demographic data, UMAT scores, admission GPA and performance on routine assessments.

RESULTS

Despite the different weightings of UMAT used in selection at the two institutions and minor variations in student demographics and programmes, results across institutions were similar. The net predictive power of admission GPA was highest for outcomes in Years 2 and 5 of the 6-year programme, accounting for 17-35% of the variance; UMAT score accounted for < 10%. The highest predictive power of the UMAT score was 9.9% for a Year 5 written examination. Combining UMAT score with admission GPA improved predictive power slightly across all outcomes. Neither UMAT score nor admission GPA predicted outcomes in the final trainee intern year well, although grading bands for this year were broad and numbers smaller.

CONCLUSIONS

The ability of the general cognitive test UMAT to predict outcomes in major assessments within medical programmes is relatively minor in comparison with that of the admission GPA, but the UMAT score adds a small amount of predictive power when it is used in combination with the GPA. However, UMAT scores may predict outcomes not studied here, which underscores the need for further validation studies in a range of settings.

Learning strategies of first year nursing and medical students: a comparative study

BACKGROUND

Interprofessional education (IPE), where two or more professions learn with, from, and about each other to improve collaboration and the quality of care, has been proposed as a curriculum strategy to promote mutual understanding between professions, thus helping to prepare health professionals to work in challenging contemporary health systems. Although there is support for IPE initiatives within health professional education, differences in student motivation and learning strategies are likely to contribute to the success of these initiatives.

OBJECTIVE

To explore self-regulated learning strategies used by first year medical and nursing students, and to determine if these strategies were different among nursing students who were high achievers.

DESIGN

A comparative survey design.

SETTING

Nursing and medical nursing schools in a large university in the western region of Sydney, Australia.

PARTICIPANTS

Six hundred and sixty-five first year nursing (n=565) and medical (n=100) students in a large university in the western region of Sydney were surveyed to assess motivational and learning strategies using The Motivated Strategies for Learning Questionnaire (MSLQ). Data relating to sociodemographic characteristics and academic performance were also collected.

RESULTS

Nursing students were significantly older than medical students (mean age: 24.4 years versus 19.4 years; p<0.001), and there were also more females in the nursing student group (82% versus 56%; p<0.001). Although nursing students had a higher mean score for extrinsic goal orientation compared to medical students (p<0.001), medical students had higher mean scores for the other four learning strategies measured: peer learning (p=0.003), help seeking (p=0.008), critical thinking (p=0.058), and time and study environment management (p<0.001). Similarly, the grade point average (GPA) of medical students at the end of their first year was significantly higher (4.5, S.D. 1.4 versus 3.6, S.D. 1.3; p<0.001) compared to that of nursing students.

CONCLUSION

While interprofessional education is seen to have many benefits for students, this study demonstrates differences in motivational and learning strategies between nursing and medical students that may impact on the success of interprofessional programs.