How effective are selection methods in medical education? A systematic review

How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal

BACKGROUND

Peer review represents a cornerstone of the scientific process, yet few studies have evaluated its association with scientific impact. The objective of this study is to assess the association of peer review scores with measures of impact for manuscripts submitted and ultimately published.

METHODS

3173 manuscripts submitted to Regional Anesthesia & Pain Medicine (RAPM) between August 2018 and October 2021 were analyzed, with those containing an abstract included. Articles were categorized by topic, type, acceptance status, author demographics and open-access status. Articles were scored based on means for the initial peer review where each reviewer's recommendation was assigned a number: 5 for 'accept', 3 for 'minor revision', 2 for 'major revision' and 0 for 'reject'. Articles were further classified by whether any reviewers recommended 'reject'. Rejected articles were analyzed to determine whether they were subsequently published in an indexed journal, and their citations were compared with those of accepted articles when the impact factor was <1.4 points lower than RAPM's 5.1 impact factor. The main outcome measure was the number of Clarivate citations within 2 years from publication. Secondary outcome measures were Google Scholar citations within 2 years and Altmetric score.

RESULTS

422 articles met inclusion criteria for analysis. There was no significant correlation between the number of Clarivate 2-year review citations and reviewer rating score (r=0.038, p=0.47), Google Scholar citations (r=0.053, p=0.31) or Altmetric score (p=0.38). There was no significant difference in 2-year Clarivate citations between accepted (median (IQR) 5 (2-10)) and rejected manuscripts published in journals with impact factors >3.7 (median 5 (2-7); p=0.39). Altmetric score was significantly higher for RAPM-published papers compared with RAPM-rejected ones (median 10 (5-17) vs 1 (0-2); p<0.001).

CONCLUSIONS

Peer review rating scores were not associated with citations, though the impact of peer review on quality and association with other metrics remains unclear.

New evidence on the validity of the selection methods for recruitment to general practice training: a cohort study

BACKGROUND

Selection into UK-based GP training has used the Multi-Specialty Recruitment Assessment (MSRA) and a face-to-face selection centre (SC). The MSRA comprises of a situational judgement test and clinical problem-solving test. The SC was suspended during the COVID-19 pandemic. Evidence is needed to guide national and international selection policy.

AIM

To evaluate the validity of GP training selection.

DESIGN & SETTING

A retrospective cohort study using data from UK-based national recruitment to GP training, from 2015-2021.

METHOD

Data were available for 32 215 GP training applicants. The ability of scores from the specialty selection process to predict subsequent performance in the Clinical Skills Assessment (CSA) of the Membership of the Royal College of General Practitioners examination was modelled using path analysis. The effect sizes for sex, professional family background, and world region of qualification were estimated.

RESULTS

All component scores of the selection process demonstrated statistically significant independent relationships with CSA performance (P<0.001), thus establishing their predictive validity. All were sensitive to demographic factors. The SC scores had the weakest relationship with future CSA performance. However, for candidates with MSRA scores below the lowest quartile, the relative contribution of the SC scores to predicting CSA performance was similar to that observed for MSRA components.

CONCLUSION

The MSRA has predictive validity in this context. Re-instituting an SC for those with relatively low MSRA scores should be considered. However, the relative costs and potential advantages and disadvantages should be carefully weighed.

Investigating the internal structure of multiple mini interviews-A perspective from Pakistan

BACKGROUND

Healthcare professionals require many personal attributes in addition to cognitive abilities and psychomotor skills for competent practice. Multiple Mini- Interviews are being employed globally to assess personality attributes of candidates for selection in health professions education at all level of entry; these attributes are namely, communication skills, critical thinking, honesty, responsibility, health advocacy, empathy and sanctity of life. Considering the high stakes involved for students, faculty, institutions and the society, rigorous quality assurance mechanisms similar to those used for student assessment must be employed for student selection, throughout the continuum of medical education. It is a difficult undertaking as these psychological constructs are difficult to define and measure. Though considered to yield reliable and valid scores, studies providing multiple evidences of internal structure especially dimensionality of Multiple Mini-Interviews are sparse giving rise to questions if they are measuring a single or multiple constructs and even if they are measuring what they are purported to be measuring.

OBJECTIVE

The main objective is to provide statistical support of the multi-dimensional nature of our Multiple Mini Interviews, hypothesized a-priori, through CFA. Another objective is to provide multiple evidences for the internal structure. Our study highlights the link between content and internal structure evidences of the constructs, thus establishing that our Multiple Mini Interviews measure what they were intended to measure.

METHOD

After securing permission from the Institutional review board, an a-priori seven factor-model was hypothesized based on the attributes considered most essential for the graduating student of the institution. After operationally defining the attributes through extensive literature search, scenarios were constructed to assess them. A 5-point rating scale was used to rate each item on the station. A total 259 students participated in the multiple mini interviews over a period of three days. A training workshop had been arranged for the participating faculty.

RESULTS

The reliability coefficient using Cronbach's alpha were calculated (range from 0.73 to 0.94), Standard Error of Measurement (ranged from 0.80 to1.64), and item to station-total correlation ranged from 0.43-0.50 to 0.75-0.83. Inter-station correlation was also determined. Confirmatory factor analysis endorsed the results of Exploratory factor analysis in the study revealing a seven model fit with multiple indices of Goodness-of-fit statistics such as Root mean square error of approximation (RMSEA) value 0.05, Standardized root mean square residual (SRMR) value with less than 0.08. All these indices showed that model fit is good. The Confirmatory factor analysis confirmed the multi-dimensional nature of our MMIs and also confirmed that our stations measured the attributes that they were supposed to measure.

CONCLUSION

This study adds to the validity evidence of Multiple Mini-Interviews, in selection of candidates, with required personality traits for healthcare profession. It provides the evidence for the multi-dimensional structure of Multiple Mini interviews administered with multiple evidences for its internal structure and demonstrates the independence of different constructs being measured.

Effect of Performance-Based Nonfinancial Incentives on Data Quality in Individual Medical Records of Institutional Births: Quasi-Experimental Study

BACKGROUND

Despite the potential of routine health information systems in tackling persistent maternal deaths stemming from poor service quality at health facilities during and around childbirth, research has demonstrated their suboptimal performance, evident from the incomplete and inaccurate data unfit for practical use. There is a consensus that nonfinancial incentives can enhance health care providers' commitment toward achieving the desired health care quality. However, there is limited evidence regarding the effectiveness of nonfinancial incentives in improving the data quality of institutional birth services in Ethiopia.

OBJECTIVE

This study aimed to evaluate the effect of performance-based nonfinancial incentives on the completeness and consistency of data in the individual medical records of women who availed institutional birth services in northwest Ethiopia.

METHODS

We used a quasi-experimental design with a comparator group in the pre-post period, using a sample of 1969 women's medical records. The study was conducted in the "Wegera" and "Tach-armacheho" districts, which served as the intervention and comparator districts, respectively. The intervention comprised a multicomponent nonfinancial incentive, including smartphones, flash disks, power banks, certificates, and scholarships. Personal records of women who gave birth within 6 months before (April to September 2020) and after (February to July 2021) the intervention were included. Three distinct women's birth records were examined: the integrated card, integrated individual folder, and delivery register. The completeness of the data was determined by examining the presence of data elements, whereas the consistency check involved evaluating the agreement of data elements among women's birth records. The average treatment effect on the treated (ATET), with 95% CIs, was computed using a difference-in-differences model.

RESULTS

In the intervention district, data completeness in women's personal records was nearly 4 times higher (ATET 3.8, 95% CI 2.2-5.5; P=.02), and consistency was approximately 12 times more likely (ATET 11.6, 95% CI 4.18-19; P=.03) than in the comparator district.

CONCLUSIONS

This study indicates that performance-based nonfinancial incentives enhance data quality in the personal records of institutional births. Health care planners can adapt these incentives to improve the data quality of comparable medical records, particularly pregnancy-related data within health care facilities. Future research is needed to assess the effectiveness of nonfinancial incentives across diverse contexts to support successful scale-up.

Detecting Artificial Intelligence-Generated Personal Statements in Professional Physical Therapist Education Program Applications: A Lexical Analysis

OBJECTIVE

The objective of this study was to compare the lexical sophistication of personal statements submitted by professional physical therapist education program applicants with those generated by OpenAI's Chat Generative Pretrained Transformer (ChatGPT).

METHODS

Personal statements from 152 applicants and 20 generated by ChatGPT were collected, all in response to a standardized prompt. These statements were coded numerically, then analyzed with recurrence quantification analyses (RQAs). RQA indices including recurrence, determinism, max line, mean line, and entropy were compared with t-tests. A receiver operating characteristic curve analysis was used to examine discriminative validity of RQA indices to distinguish between ChatGPT and human-generated personal statements.

RESULTS

ChatGPT-generated personal statements exhibited higher recurrence, determinism, mean line, and entropy values than did human-generated personal statements. The strongest discriminator was a 13.04% determinism rate, which differentiated ChatGPT from human-generated writing samples with 70% sensitivity and 91.4% specificity (positive likelihood ratio = 8.14). Personal statements with determinism rates exceeding 13% were 8 times more likely to have been ChatGPT than human generated.

CONCLUSION

Although RQA can distinguish artificial intelligence (AI)-generated text from human-generated text, it is not absolute. Thus, AI introduces additional challenges to the authenticity and utility of personal statements. Admissions committees along with organizations providing guidelines in professional physical therapist education program admissions should reevaluate the role of personal statements in applications.

IMPACT

As AI-driven chatbots like ChatGPT complicate the evaluation of personal statements, RQA emerges as a potential tool for admissions committees to detect AI-generated statements.

Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes

BACKGROUND

Computer-based assessment for sampling personal characteristics (Casper), an online situational judgement test, is a broad measure of personal and professional qualities. We examined the impact of Casper in the residency selection process on professionalism concerns, learning interventions and resource utilization at an institution.

METHODS

In 2022, admissions data and information in the files of residents in difficulty (over three years pre- and post- Casper implementation) was used to determine the number of residents in difficulty, CanMEDS roles requiring a learning intervention, types of learning interventions (informal learning plans vs. formal remediation or probation), and impact on the utilization of institutional resource (costs and time). Professionalism concerns were mapped to the 4I domains of a professionalism framework, and their severity was considered in mild, moderate, and major categories. Descriptive statistics and between group comparisons were used for quantitative data.

RESULTS

In the pre- and post- Casper cohorts the number of residents in difficulty (16 vs. 15) and the number of learning interventions (18 vs. 16) were similar. Professionalism concerns as an outcome measure decreased by 35% from 12/16 to 6/15 (p < 0.05), were reduced in all 4I domains (involvement, integrity, interaction, introspection) and in their severity. Formal learning interventions (15 vs. 5) and informal learning plans (3 vs. 11) were significantly different in the pre- and post-Casper cohorts respectively (p < 0.05). This reduction in formal learning interventions was associated with a 96% reduction in costs f(rom hundreds to tens of thousands of dollars and a reduction in time for learning interventions (from years to months).

CONCLUSIONS

Justifiable from multiple stakeholder perspectives, use of an SJT (Casper) improves a clinical performance measure (professionalism concerns) and permits the institution to redirect its limited resources (cost savings and time) to enhance institutional endeavors and improve learner well-being and quality of programs.

How professional and academic pre-qualifications relate to success in medical education: Results of a multicentre study in Germany

OBJECTIVE

Every year, many applicants want to study medicine. Appropriate selection procedures are needed to identify suitable candidates for the demanding curriculum. Although research on medical school admissions has shown good predictive validity for cognitive selection methods (undergraduate GPA, aptitude tests), the literature on applicants with professional and/or academic experience prior to entering medical school remains slim. In our study, we therefore aimed to examine the association between academic success in medical school and having previously completed vocational training in the medical field, voluntary service (≥11 months) or an academic degree.

METHODS

Data were collected in a multicentre, cross-sectional study at five medical schools in Germany (Baden-Wuerttemberg) from students during medical school (i.e. 3rd-, 6th-, and 10th-semester and final-year students). Academic success was assessed according to scores on the first and second state examinations, the total number of examinations repeated and the number of semesters beyond the standard period of study. For the analysis we calculated ordinal logistic regression models for each outcome variable of academic success.

RESULTS

A total of N = 2,370 participants (response rate: RR = 47%) participated in the study. Having completed vocational training was associated with a higher amount of repeated examinations (small effect), while having an academic degree was associated with worse scores on the second state examination (medium effect). No significant association emerged between voluntary service and academic success.

CONCLUSION

The results indicate that professional and academic pre-qualifications pose no advantage for academic success. Possible associations with the financing of study and living conditions of students with pre-qualifications were analysed and discussed in an exploratory manner. However, the operationalisation of academic success from objective and cognitive data should be critically discussed, as the benefits of prior experience may be captured by personal qualities rather than examination results.

Follow the policy: An actor network theory study of widening participation to medicine in two countries

INTRODUCTION

The slow pace of change in respect of increasing the diversity of medical students suggests powerful actors are reproducing practices to support the status quo. Opening up medicine to embrace diversity thus requires the deconstruction of entrenched processes and practices. The first step in doing so is to understand how the actor-network of widening participation and access to medicine (WP/WA) is constructed. Thus, here we examine how the connections among actors in WP/WA in two different networks are assembled.

METHODS

A comparative case study using documents (n = 7) and interviews with staff and students (n = 45) from two medical schools, one United Kingdom and one Australian, was used. We used Callon's moments of translation (problematisation, interessement/operationalisation, enrolment, mobilisation) to map the network of actors as they are assembled in relation to one another. Our main actant was institutional WP to medicine policy (actor-as-policy).

RESULTS

Our actor-as-policy introduced five other actors: the medical school, medical profession, high schools, applicants and medical school staff. In terms of problematisation, academic excellence holds firm as the obligatory passage point and focal challenge for all actors in both countries. The networks are operationalised via activities such as outreach and admissions policy (e.g., affirmative action is apparent in Australia but not the UK). High schools play (at best) a passive role, but directed by the policy, the medical schools and applicants work hard to achieve WP/WA to medicine. In both contexts, staff are key mobilisers of WP/WA, but with little guidance in how to enact policy. In Australia, policy drivers plus associated entry structures mean the medical profession exerts significant influence.

CONCLUSIONS

Keeping academic excellence as the obligatory passage point to medical school shapes the whole network of WP/WA and perpetuates inequality. Only by addressing this can the network reconfigure.

Reframing selection as a learning experience: Insights from a residency selection assessment

PURPOSE

Selection for postgraduate medical training is high-stakes and complex. The learning value of assessments for selection has, thus far, been underexplored, limiting their uptake as potentially meaningful learning experiences. The aim of this study was to explore the learning value residency applicants derive from an intelligence, personality, motivation and competency selection assessment and what factors influence the experienced learning value.

METHODS

In Autumn 2020 and Spring 2021, we conducted individual semi-structured interviews with sixteen applicants for pediatric residency training. Selection outcomes were unknown at the time of the interview. Interviews were transcribed verbatim and thematically analyzed.

RESULTS

Participants reported that the assessment was valuable in fostering self-reflection and self-awareness, embracing self-acceptance, pursuing development goals, assessing professional fit, and harnessing motivational drivers in work. The experienced learning value was influenced by applicants' ability to interpret its results, their focus on the high-stakes selection process and concerns regarding the acceptability and credibility of the selection tool.

CONCLUSIONS

While the selection assessment showed learning potential, its learning value was impeded by a preoccupation with the high-stakes nature of the selection procedure. Intentional integration of the selection assessment in the learning curriculum may play a pivotal role in realizing its learning potential.

'Selection into training will always be an inexact process': A survey of Directors of Physician Education on selection into Basic Physician Training in Australia and New Zealand

BACKGROUND

Despite being one of the largest medical specialty training programmes in Australasia, there is no standardised method for selection into Basic Physician Training (BPT), and limited data exist regarding current practices.

AIMS

To address existing knowledge gaps, we aimed to create a 'snapshot' of current BPT selection practices and explore the perspectives of Directors of Physician Education (DPEs) regarding trainee selection.

METHODS

An electronic survey of DPEs from adult and paediatric medicine BPT sites in Australia and New Zealand was undertaken in January-February 2022. A combination of free text, multiple-choice and yes/no answers were analysed using descriptive statistics and qualitative content analysis.

RESULTS

A total of 70 responses were received, achieving a response rate of 35% (70/198). Selection practices were found to be heterogenous across BPT sites. Respondents had varying opinions regarding the utility of selection tools and desirable candidate attributes. A heavy reliance upon interviews and the reported use of subjective assessments raise concerns for selection process bias.

CONCLUSION

BPT sites should critically evaluate their selection methods, and more research in this field is needed to establish best practice.

'Click, I Guess I'm Done': Applicants' and Assessors' Experiences Transitioning to a Virtual Multiple Mini Interview Format

Introduction

During the COVID-19 pandemic, medical schools were forced to suspend in-person interviews and transition to a virtual Multiple Mini Interview (vMMI) format. MMIs typically comprise multiple short assessments overseen by assessors, with the aim of measuring a wide range of non-cognitive competencies. The adaptation to vMMI required medical schools to make swift changes to their MMI structure and delivery. In this paper, we focus on two specific groups greatly impacted by the decision to transition to vMMIs: medical school applicants and MMI assessors.

Methods

We conducted an interpretive qualitative study to explore medical school applicants' and assessors' experiences transitioning to an asynchronous vMMI format. Ten assessors and five medical students from one Canadian medical school participated in semi-structured interviews. Data was analyzed using a thematic analysis framework.

Results

Both applicants and assessors shared a mutual feeling of longing and nostalgia for an interview experience that, due to the pandemic, was understandably adapted. The most obvious forms of loss experienced - albeit in different ways - were: 1) human connection and 2) missed opportunity. Applicants and assessors described several factors that amplified their grief/loss response. These were: 1) resource availability, 2) technological concerns, and 3) the virtual interview environment.

Discussion

While virtual interviewing has obvious advantages, we cannot overlook that asynchronous vMMIs do not lend themselves to the same caliber of interaction and camaraderie as experienced in in-person interviews. We outline several recommendations medical schools can implement to enhance the vMMI experience for applicants and assessors.

Residency Selection in Otolaryngology: Past, Present, & Future

OBJECTIVES

To examine the otolaryngology residency selection process, including past experiences based on the medical literature and survey analysis of our present practices to generate recommendations for future selection system design.

METHODS

A mixed-methods study, including a scoping review and a cross-sectional survey, was completed. Four databases were assessed for articles on otolaryngology residency selection published from January 1, 2016 through December 31, 2020. A 36-question survey was developed and distributed to 114 otolaryngology program directors. Descriptive and thematic analysis was performed.

RESULTS

Ultimately, 67 of 168 articles underwent data abstraction and assessment. Three themes surfaced during the analysis: effectiveness, efficiency, and equity. Regarding the survey, there were 62 participants (54.4% response rate). The three most important goals for the selection process were: (1) to fit the program culture, (2) to make good colleagues, and (3) to contribute to the program's diversity. The three biggest 'pain points' were as follows: (1) Large volume of applications, (2) Lack of reliable information about personal characteristics, and (3) Lack of reliable information about a genuine interest in the program.

CONCLUSIONS

Within this study, the depth and breadth of the literature on otolaryngology residency selection have been synthesized. Additionally, baseline data on selection practices within our specialty has been captured. With an informed understanding of our past and present, we can look to the future. Built upon the principles of person-environment fit theory, our proposed framework can guide research and policy discussions regarding the design of selection systems in otolaryngology, as we work to achieve more effective, efficient, and equitable outcomes.

LEVEL OF EVIDENCE

N/A Laryngoscope, 133:2929-2941, 2023.

A Comparison of the Academic Achievement at the End of the Medicine Undergraduate Degree Program Between Students Who Only Used the University Admission Test and Those Who Used the University Admission Test Plus Marks from the High School National Exam (ENEM) at a Single Brazilian Center

Purpose

The role of marks in the University Admission Test (UAT) plus the marks from pre-university academic records in predicting academic achievement at the end of the Medicine undergraduate degree program is not completely known. This study was undertaken to compare the performance of marks in the UAT alone with those of the UAT plus marks from the National High School Exam (ENEM in Brazil) regarding students' outcomes at the end of the Medicine undergraduate degree program.

Methods

Fifty-one (51) students from the last semester (12th) of our Medicine undergraduate degree program were included in the study. They were divided into a group of those who used the marks obtained in the UAT plus the marks obtained in the ENEM (ENEM group, n=9), and those who only used the marks in the UAT (non-ENEM group, n=42). We compared the academic achievement of the non-ENEM group with that of the ENEM group regarding the mean marks obtained in the clerkship, in the Progress Test (PT), and in the Objective Structured Clinical Examination (OSCE).

Results

The mean scores obtained in the disciplines of the clerkship were higher in the non-ENEM group compared to the ENEM group (7.32 ± 0.41 vs 6.98 ± 0.31, p= 0.01). Both groups obtained similar mean marks in the OSCE and in the PT. A moderate correlation was observed between the marks in the clerkship with those of the UAT from the non-ENEM group (p=0.00006; r=0.45).

Conclusion

Marks of the UAT alone appear to be associated with a higher academic achievement in the clerkship than marks of the UAT plus scores obtained from the ENEM at the end of the Medicine undergraduate degree program.

Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153

Selection is the first assessment of medical education and training. Medical schools must select from a pool of academically successful applicants and ensure that the way in which they choose future clinicians is robust, defensible, fair to all who apply and cost-effective. However, there is no comprehensive and evidence-informed guide to help those tasked with setting up or rejuvenating their local selection process. To address this gap, our guide draws on the latest research, international case studies and consideration of common dilemmas to provide practical guidance for designing, implementing and evaluating an effective medical school selection system. We draw on a model from the field of instructional design to frame the many different activities involved in doing so: the ADDIE model. ADDIE provides a systematic framework of Analysis (of the outcomes to be achieved by the selection process, and the barriers and facilitators to achieving these), Design (what tools and content are needed so the goals of selection are achieved), Development (what materials and resources are needed and available), Implementation (plan [including piloting], do study and adjust) and Evaluation (quality assurance is embedded throughout but the last step involves extensive evaluation of the entire process and its outcomes).HIGHLIGHTSRobust, defensible and fair selection into medical school is essential. This guide systematically covers the processes required to achieve this, from needs analysis through design, development and implementation, to evaluation of the success of a selection process.

Selection tools and student diversity in health professions education: a multi-site study

Student diversity in health professions education (HPE) can be affected by selection procedures. Little is known about how different selection tools impact student diversity across programs using different combinations of traditional and broadened selection criteria. The present multi-site study examined the chances in selection of subgroups of applicants to HPE undergraduate programs with distinctive selection procedures, and their performance on corresponding selection tools. Probability of selection of subgroups (based on gender, migration background, prior education, parental education) of applicants (N = 1935) to five selection procedures of corresponding Dutch HPE undergraduate programs was estimated using multilevel logistic regression. Multilevel linear regression was used to analyze performance on four tools: prior-education grade point average (pe-GPA), biomedical knowledge test, curriculum-sampling test, and curriculum vitae (CV). First-generation Western immigrants and applicants with a foreign education background were significantly less likely to be selected than applicants without a migration background and with pre-university education. These effects did not vary across programs. More variability in effects was found between different selection tools. Compared to women, men performed significantly poorer on CVs, while they had higher scores on biomedical knowledge tests. Applicants with a non-Western migration background scored lower on curriculum-sampling tests. First-generation Western immigrants had lower CV-scores. First-generation university applicants had significantly lower pe-GPAs. There was a variety in effects for applicants with different alternative forms of prior education. For curriculum-sampling tests and CVs, effects varied across programs. Our findings highlight the need for continuous evaluation, identifying best practices within existing tools, and applying alternative tools.

Effects of candidates' demographics and evaluation of the virtual Multiple Mini Interview (vMMI) as a tool for selection into paediatric training in Queensland

INTRODUCTION

The Queensland Basic Paediatric Training Network (QBPTN) is responsible for the selection of candidates into paediatric training in Queensland. The COVID-19 pandemic necessitated interviews to be conducted 'virtually' as virtual Multiple-Mini-Interviews (vMMI). The study aimed to describe the demographic characteristics of candidates applying for selection into paediatric training in Queensland, and to explore their perspectives and experiences with the vMMI selection tool.

METHODOLOGY

The demographic characteristics of candidates and their vMMI outcomes were collected and analysed with a mixed methods approach. The qualitative component was comprised of seven semi-structured interviews with consenting candidates.

RESULTS

Seventy-one shortlisted candidates took part in vMMI and 41 were offered training positions. The demographic characteristics of candidates at various stages of selection were similar. The mean vMMI scores were not statistically different between candidates from the Modified Monash Model 1 (MMM1) location and others [mean (SD): 43.5 (5.1) versus 41.7 (6.7), respectively, p = 0.26]. However, there was a statistically significant difference (p value 0.03) between being offered and not offered a training position for candidates from MMM2 and above. The analysis of the semi-structured interviews suggested that candidate experiences of the vMMI were influenced by the quality of the management of the technology used. Flexibility, convenience, and reduced stress were the main factors that influenced candidates' acceptance of vMMI. Perceptions of the vMMI process focused on the need to build rapport and facilitate communication with the interviewers.

DISCUSSION

vMMI is a viable alternative to face-to-face (FTF) MMI. The vMMI experience can be improved by facilitating enhanced interviewer training, by making provision for adequate candidate preparation and by having contingency plans in place for unexpected technical challenges. Given government priorities in Australia, the impact of candidates' geographical location on the vMMI outcome for candidates from MMM >1 location needs to be further explored.

Effects of the alternative medical curriculum at the Hannover Medical School on length of study and academic success

Objective

The model curriculum HannibaL (Hannoversche integrierter berufsorientierter und adaptiver Lehrplan) differs significantly from other medical study programs in Germany in terms of its structure with which, among other factors, the Hannover Medical School (MHH) saw an opportunity to positively influence the length of study. We investigate how the length of medical study is influenced by the curriculum's structure and whether this has any impact on academic success.

Methods

We use data from over 2,500 students who studied medicine at MHH between 2011 and 2021. We measure study time as the number of years which pass until completion of the respective study phases and academic success as the grades achieved on final exams.

Results

Since they more often fail or postpone exams, students admitted based on special quotas (VQ) or a waiting list (WQ) need significantly more time to complete the first study phase (M1) compared to students who were admitted based on a selection process (AdH) or who belong to the "best school graduates" quota (AQ) because they earned the highest scores on the final secondary school exam. Yet, students from all admission groups reach the written state exam (M2) almost simultaneously. In HannibaL, WQ and VQ manage to catch up on delays from M1 with no negative impact on success in M2. In general, however, VQ and WQ achieve lower grades and drop out more often than students from AQ and AdH.

Discussion

In the regular curriculum, students can only proceed with their studies once M1 has been entirely completed. HannibaL, on the other hand, allows for the catching up of delays from the first two years of study by integrating both study phases. The curricular structure thus accommodates students with lower academic performance who accumulate delays early on in their studies. By contrast, delays in the AQ and AdH groups arise during the second phase of study (M2).

Impact of medical students' socioeconomic backgrounds on medical school application, admission and migration in Japan: a web-based survey

OBJECTIVES

The aim of this study was to clarify and compare the socioeconomic backgrounds of medical students with those of other health professions and non-health faculty students in an era of increasing inequity in Japanese society.

DESIGN

This was a quantitative nationwide study. Data were collected by a cross-sectional web-based anonymous questionnaire.

SETTING

Data from years 3 and 4 medical, health professions and non-health faculty students across Japan were collected in 2021.

PARTICIPANTS

Participants were 1991 students from medical schools, 224 from dental, 419 from pharmacy, 326 from nursing, 144 from other health professions and 207 from non-health faculties.

RESULTS

The proportion of high-income families (>18 million yen: ca. US$140 000) among medical students was 25.6%, higher than that of pharmacy (8.7%) and nursing students (4.1%) (p<0.01). One-third of medical students had a physician parent, more common than in non-medical students (p<0.01). Students who only applied to public medical schools and a regional quota 'Chiiki-waku' students with scholarship had lower family income and physician parents compared with those who applied to private medical schools (p<0.01), but they still had higher physician parents compared with non-medical students (p<0.01). Logistic regression revealed that having a physician parent (p<0.01), aspiring to the present profession during elementary school (p<0.01) and private upper secondary school graduation (p<0.01) predicted the likelihood of studying medicine. There were regional differences of backgrounds among medical students, and 80% of medical students with urban backgrounds intended to work in urban localities after graduation.

CONCLUSIONS

This study provides evidence that medical students in Japan hail from urban and higher income classes and physicians' families. This finding has implications for the health workforce maldistribution in Japan. Widening the diversity of medical students is essential for solving physician workforce issues and meeting broad healthcare needs.

Does medical education reform change who is selected? A national cross-sectional survey from China

OBJECTIVES

Approximately 10 years ago, China introduced an education plan to improve the overall quality of medical education and to better serve the population's health needs. Many medical schools were then recognised and financed by China's Ministry of Education to develop and operationalise new pilot programmes (PPs) aligned with this plan. These ran in parallel with the traditional programmes (TPs). One way to achieve the plan's first aim, improving the quality of medical education, is to select academically stronger candidates. We, thus, examined and compared who were selected into PPs and TPs.

DESIGN

Cross-sectional study.

SETTING

Data were collected from 123 medical schools across China via the 2021 China Medical Student Survey.

PARTICIPANTS

Participants were undergraduate clinical medicine students across all year groups.

PRIMARY AND SECONDARY OUTCOME MEASURES

Medical school selection was via the National College Entrance Examination (NCEE). Medical students' NCEE performance and their sociodemographics were used as the primary and secondary outcome measures. Mann-Whitney or χ2 tests were used to compare the means between educational programmes (PPs vs TPs) and various selection outcomes. Multilevel mixed-effects regressions were employed to account for school idiosyncratic selection results.

RESULTS

Of the 204 817 respondents, 194 163 (94.8%) were in a TP and 10 654 (5.2%) a PP. PP respondents (median=75.2, IQR=69.5-78.8) had significantly higher NCEE scores than their TP counterparts (median=73.9, IQR=68.5-78.7). Holding constant their NCEE score, PP respondents were significantly more likely to come from urban areas, not be first-generation college students, and have parents with higher occupational status and income.

CONCLUSIONS

Assuming quality can be indicated by prior academic achievement at the point of selection, PPs achieved this mission. However, doing so limited medical students' diversity. This may be unhelpful in achieving the Education Plan's goal to better serve China's health needs.

The association between how medical students were selected and their perceived stress levels in Year-1 of medical school

BACKGROUND

The prevalence of medical students' mental distress is high. While schools apply various methods to select a well-performing and diverse student population, little is known about the association between different selection methods and the well-being of these students during medical school. The present retrospective multi-cohort study assessed whether students selected by high grades, assessment, or weighted lottery showed different stress perception levels in Year-1 of medical school.

METHODS

Of 1144 Dutch Year-1 medical students, 650 (57%) of the cohorts 2013, 2014, and 2018 who were selected by high grades, assessment, or weighted lottery completed a stress perception questionnaire (PSS-14). A multilevel regression analysis assessed the association between selection method (independent variable) and stress perception levels (dependent variable) while controlling for gender and cohort. In a post-hoc analysis, academic performance (optimal vs. non-optimal) was included in the multilevel model.

RESULTS

Students selected by assessment (B = 2.25, p < .01, effect size (ES) = small) or weighted lottery (B = 3.95, p < .01, ES = medium) had higher stress perception levels than students selected by high grades. Extending the regression model with optimal academic performance (B=-4.38, p < .001, ES = medium), eliminated the statistically significant difference in stress perception between assessment and high grades and reduced the difference between weighted lottery and high grades from 3.95 to 2.45 (B = 2.45, p < .05, ES = small).

CONCLUSIONS

Selection methods intended to create a diverse student population - assessment and lottery - are associated with higher stress perception levels in Year-1 of medical school. These findings offer medical schools insights into fulfilling their responsibility to take care of their students' well-being.

Admission into healthcare education in France: Half-baked reform that further complicates the system

France has undertaken the most ambitious reform of healthcare education in more than 10 years. It has resulted in a hybrid system with multiple pathways, granting admission into the healthcare professions after competitive exams. The reform continues the trend to increase the quotas limiting the number of second year healthcare students, and also creates new local access options to healthcare education. However, the heterogeneity in implementation has led, in conjunction with the difficulties caused by the Covid-19 pandemic, to great dismay among students and parents. This article seeks to outline the historical underpinnings of the reform program(s) and argues that the core question - selecting students from the very high number of candidates in a fair and effective manner - remains largely unresolved.

What does shared decision making ask from doctors? Uncovering suppressed qualities that could improve person-centered care

OBJECTIVE

Shared decision making (SDM) is infrequently seen in clinical practice despite four decades of efforts. We propose a need to explore what SDM asks from doctors in terms of enabling competencies and necessary, underlying qualities, and how these can be nurtured or suppressed in medical training.

DISCUSSION

Key SDM tasks call for doctors to understand communication and decision mechanisms to carry them out well, including reflecting on what they know and do not know, considering what to say and how, and listening unprejudiced to patients. Different doctor qualities can support accomplishing these tasks; humility, flexibility, honesty, fairness, self-regulation, curiosity, compassion, judgment, creativity, and courage, all relevant to deliberation and decision making. Patient deference to doctors, lack of supervised training opportunities with professional feedback, and high demands in the work environment may all inflate the risk of only superficially involving patients.

CONCLUSIONS

We have identified ten professional qualities and related competencies required for SDM, with each to be selected based on the specific situation. The competencies and qualities need to be preserved and nurtured during doctor identity building, to bridge the gap between knowledge, technical skills, and authentic efforts to achieve SDM.

What is the evidence for biology as the 'heart of eligibility' to study medicine? A retrospective analysis

The first, sometimes the only, selection tool for entry into undergraduate medicine is prior educational attainment (PEA). This is often further specified to include certain subjects, for example, biology is a prerequisite for entry into medicine in many Asian countries. However, there seems no clear evidence base for this prerequisite. Our aim, therefore, was to carry out a retrospective quantitative study comparing the performances of five cohorts of students (2015-2019 entry; n = 588) with and without biology PEA in Years 1 and 2 Bachelor of Medicine and Bachelor of Surgery (MBBS) integrated written assessments (n = 3) and anatomy practical examinations (APE) (n = 5). The study was conducted at one of Singapore's three medical schools. Data were analyzed using independent t-tests and Mann-Whitney U with p values of less than 0.05 were considered significant. There were no significant differences in performance on any Years 1 or 2 integrated written assessments. Similarly, in one of the APE, a significant difference was found for one cohort (academic year [AY] 2015-2016) out of five assessments. These results suggest that having a prior biology qualification does not make a difference in assessment performance in the early years of medical school. This information may help stakeholders and admissions committees decide whether biology is required for medical school entrance.

CARDA: Guiding document analyses in health professions education research

INTRODUCTION

Documents, from policies and procedures to curriculum maps and examination papers, structure the everyday experiences of health professions education (HPE), and as such can provide a wealth of empirical information. Document analysis (DA) is an umbrella term for a range of systematic research procedures that use documents as data.

METHODS

A meta-study review was conducted with the aims of describing the current state of DA in HPE, guiding researchers engaging in DA and improving methodological, analytical and reporting rigour. Structured searches were conducted, returns were filtered for inclusion and the 115 remaining articles were critically analysed for their use of DA methods and methodologies.

RESULTS

There was a significant increase in the number of articles reporting the use of DA over time. Sixty-three articles were single method (DA only), while the others were mixed methods research (MMR). Overall, there were major lacunae in terms of why documents were used, how documents were identified, what the authors did and what they found from the documents. This was particularly apparent in MMR where DA reporting was typically poorer than the reporting of other methods in the same paper.

DISCUSSION

Given these many lacunae, a framework for reporting on DA research was developed to facilitate rigorous DA research and transparent, complete and accurate reporting of the same, to help readers assess the trustworthiness of the findings from document use and analysis in HPE and, potentially, other domains. It was also noted that there are gaps in HPE knowledge that could be addressed through DA, particularly where documents are conceptualised as more than passive holders of information. Scholars are encouraged to reflect more deeply on the applications and practices of DA, with the ultimate aim of ensuring more substantive and more rigorous use of documents for understanding and constructing meaning in our field.

[Multimodal selection of medical students: The predictive power of individual process components in the two-stage selection process at Witten/Herdecke University (UW/H)]

OBJECTIVE

According to the legislator's ideas, the selection of medical students in Germany should no longer be based on the Abitur grade alone. This approach has already been implemented in the two-stage selection process at Witten/Herdecke University (UW/H) using several criteria. On the one hand, the present study aims to determine the prognostic value of the procedural components for the overall performance on the selection day. On the other hand, the different strategies of the applicants in processing the written task (phase 1) will be examined with regard to their application success.

METHODOLOGY

Data on applications for the summer semester 2020 (N = 819 phase 1; N = 233 phase 2) were available retrospectively. A stepwise regression analysis was conducted to determine the predictive power of each procedural component. Using a summary content analysis, the four essays from the applicants' motivation letters were structured and categories were identified, and an extreme group comparison (Group 1: Not invited; Group 2: University acceptance; N = 60 essays) was conducted.

RESULTS

As the stepwise regression analysis shows, the individual biographical interview emerged as the strongest predictor in terms of overall performance, followed by lecture, group interview, and multiple mini interviews. Content analysis extracted content and scaling categories for the individual essays, as well as an additional meta-category (Impression Management, IM). Successful applicants demonstrated, among other things, better judgment skills, more sophisticated reasoning skills, and an internalized role model as a physician. In addition, they used defensive IM strategies, e.g., subjectification and self-deprecation, more frequently.

CONCLUSION

Biographical interview is considered the strongest predictor of overall performance. The dimensions of impression management, reasoning quality and judgment proved to be reliable predictors of successful performance in the selection process. In addition, role image as a physician and professional commitment had a favorable effect on the selection decision.

Does doctors' personality differ from those of patients, the highly educated and other caring professions? An observational study using two nationally representative Australian surveys

OBJECTIVES

Personality differences between doctors and patients can affect treatment outcomes. We examine these trait disparities, as well as differences across medical specialities.

DESIGN

Retrospective, observational statistical analysis of secondary data.

SETTING

Data from two data sets that are nationally representative of doctors and the general population in Australia.

PARTICIPANTS

We include 23 358 individuals from a representative survey of the general Australian population (with subgroups of 18 705 patients, 1261 highly educated individuals and 5814 working in caring professions) as well as 19 351 doctors from a representative survey of doctors in Australia (with subgroups of 5844 general practitioners, 1776 person-oriented specialists and 3245 technique-oriented specialists).

MAIN OUTCOME MEASURES

Big Five personality traits and locus of control. Measures are standardised by gender, age and being born overseas and weighted to be representative of their population.

RESULTS

Doctors are significantly more agreeable (a: standardised score -0.12, 95% CIs -0.18 to -0.06), conscientious (c: -0.27 to -0.33 to -0.20), extroverted (e: 0.11, 0.04 to 0.17) and neurotic (n: 0.14, CI 0.08 to 0.20) than the general population (a: -0.38 to -0.42 to -0.34, c: -0.96 to -1.00 to -0.91, e: -0.22 to -0.26 to -0.19, n: -1.01 to -1.03 to -0.98) or patients (a: -0.77 to -0.85 to -0.69, c: -1.27 to -1.36 to -1.19, e: -0.24 to -0.31 to -0.18, n: -0.71 to -0.76 to -0.66). Patients (-0.03 to -0.10 to 0.05) are more open than doctors (-0.30 to -0.36 to -0.23). Doctors have a significantly more external locus of control (0.06, 0.00 to 0.13) than the general population (-0.10 to -0.13 to -0.06) but do not differ from patients (-0.04 to -0.11 to 0.03). There are minor differences in personality traits among doctors with different specialities.

CONCLUSIONS

Several personality traits differ between doctors, the population and patients. Awareness about differences can improve doctor-patient communication and allow patients to understand and comply with treatment recommendations.

The relationship between required physician letters of recommendation and decreasing diversity in osteopathic medical school admissions

CONTEXT

Some racial and ethnic groups are underrepresented in the medical field because they face unique barriers to admission to medical school. One admission requirement that can present a barrier for applicants is the physician letter of recommendation (PLOR). Undergraduate students report confusion with the application process and lack of mentorship to be two of their biggest challenges to becoming a doctor. It is especially challenging to those who already have limited access to practicing physicians. Therefore, we hypothesized that in the presence of a PLOR requirement, the diversity of students who apply and matriculate into medical school will be decreased.

OBJECTIVES

This study aims to determine if a relationship exists between a PLOR requirement for the medical school application and the proportion of underrepresented in medicine (URM) students applying and matriculating to that school.

METHODS

A retrospective study was conducted utilizing data published by the American Association of Colleges of Osteopathic Medicine Application Services (AACOMAS) on the race and ethnicity of applicants and matriculants to osteopathic medical schools during the years 2009-2019. In total, 35 osteopathic schools with 44 campuses were included in the study. Schools were grouped based on whether they required a PLOR. For each group of schools, descriptive statistics were performed for the following variables: number of total applicants, class size, application rate per ethnicity, matriculation rate per ethnicity, number of applicants per ethnicity, number of matriculants per ethnicity, and percentage of student body per ethnicity. The Wilcoxon rank-sum test was utilized to detect differences between the two groups. Statistical significance was assessed at the α=0.05 level.

RESULTS

Schools that required a PLOR showed decreases in the number of applicants across all races and ethnicities. Black students showed the greatest difference between groups and were the only ethnicity to show significant reductions across all outcomes in the presence of a PLOR requirement. On average, schools that required a PLOR have 37.3% (185 vs. 295; p<0.0001) fewer Black applicants and 51.2% (4 vs. 8.2; p<0.0001) fewer Black matriculants.

CONCLUSIONS

This study strongly suggests a relationship between requiring a PLOR's and decreasing racial and ethnic diversity in medical school matriculants, specifically the Black applicants. Based on this result, it is recommended that the requirement of a PLOR be discontinued for osteopathic medical schools.

Clinical experience can compensate for inferior academic achievements in an undergraduate objective structured clinical examination

BACKGROUND

Practical and non-cognitive skills are essential to medical professions; yet, success in medical studies is primarily assessed with cognitive criteria. We show that practical exams can benefit students who have only average high school final grades, but working experience in medical professions.

METHODS

With a cross-sectional study, we compare the performance of undergraduate medical students with working experience in adjacent health-care professions (and below-average school leaving-grades) with students who entered medical school directly based on their excellent school records in an Objective Structured Clinical Examination (OSCE). For a sample of more than 1,200 students, we use information on OSCE scores in medical and practical skills, doctor-patient communication/interaction, performance in MC-exams, and core sociodemographic variables.

RESULTS

Waiting list students outperformed their classmates in the demonstration of practical skills. Students admitted via their excellent school grades scored best overall. This difference vanishes once we control for school-leaving grade and age, the two main factors separating the analysed groups. Students from the waiting list have a significantly smaller overall chance to reach excellent grades in the first two years of study.

CONCLUSIONS

Students who gathered experiences in health-care professions before enrolling at medical school can benefit from an expanded role of practical elements in medical studies. Student selection instruments should take these different starting positions and qualities of applicants into account, for example with a quota for the professionally experienced.

Investigating Students’ Pre-University Admission Requirements and Their Correlation with Academic Performance for Medical Students: An Educational Data Mining Approach

Medical education is one of the most sought-after disciplines for its prestigious and noble status. Institutions endeavor to identify admissions criteria to register bright students who can handle the complexity of medical training and become competent clinicians. This study aims to apply statistical and educational data mining approaches to study the relationship between pre-admission criteria and student performance in medical programs at a public university in Saudi Arabia. The present study is a retrospective cohort study conducted at the College of Computer Science, King Khalid University, Abha, Kingdom of Saudi Arabia between February and November 2022. The current pre-admission criterion is the admission score taken as the weighted average of high school percentage (HSP), general aptitude test (GAT) and standard achievement admission test (SAAT), with respective weights of 0.3, 0.3 and 0.4. Regression and optimization techniques have been applied to identify weightages that better fit the data. Five classification techniques—Decision Tree, Neural Network, Random Forest, Naïve Bayes and K-Nearest Neighbors—are employed to develop models to predict student performance. The regression and optimization analyses show that optimized weights of HSP, GAT and SAAT are 0.3, 0.2 and 0.5, respectively. The results depict that the performance of the models improves with admission scores based on optimized weightages. Further, the Neural Network and Naïve Bayes techniques outperform other techniques. Firstly, this study proposes to revise the weights of HSP, GAT and SAAT to 0.3, 0.2 and 0.5, respectively. Secondly, as the evaluation metrics of models remain less than 0.75, this study proposes to identify additional student features for calculating admission scores to select ideal candidates for medical programs.

A mixed methods evaluation of Multiple Mini Interviews for entry into the Bachelor of Midwifery

BACKGROUND

Multiple-mini interviews (MMI) are increasingly used as part of the admission process into healthcare degrees. Evaluations have found MMIs to be a fair assessment tool in terms of reliability and validity and viewed positively by those involved in the MMI process. The use of MMIs in midwifery is novel and evaluation is lacking.

AIM

To evaluate the use of MMIs as part of the admission process for the Bachelor of Midwifery in one Australian university.

METHODS

A basic convergent mixed methods study design was utilised. Data included linked data sets, Likert scale responses to survey questions, focus groups and open-ended survey questions. Integration took place at the interpretation and reporting stage.

FINDINGS

Participants viewed the MMI experience positively. The study confirmed the reliability of the MMIs as an assessment tool. Most variance in MMI scores was attributed to the candidate at 31.4 % with the interviewer and the interview station having less influence on the MMI score at 11 % and 6.4 % variance. Older applicants on average achieved higher MMI scores, and those who spoke a language other than English at home or were first in family to attend university had lower on average MMI scores. Being born overseas did not impact an applicant's MMI score. The overall experience was seen as fair, offering further opportunity to gain entry into the Bachelor of Midwifery.

CONCLUSION

MMIs were viewed positively and findings support the use of MMIs as part of an admissions process for the Bachelor of Midwifery.

Evaluating situational judgment test use and diversity in admissions at a southern US medical school

INTRODUCTION

Situational judgment tests have been adopted by medical schools to assess decision-making and ethical characteristics of applicants. These tests are hypothesized to positively affect diversity in admissions by serving as a noncognitive metric of evaluation. The purpose of this study was to evaluate the performance of the Computer-based Assessment for Sampling Personal Characteristics (CASPer) scores in relation to admissions interview evaluations.

METHODS

This was a cohort study of applicants interviewing at a public school of medicine in the southeastern United States in 2018 and 2019. Applicants took the CASPer test prior to their interview day. In-person interviews consisted of a traditional interview and multiple-mini-interview (MMI) stations. Between subjects, analyses were used to compare scores from traditional interviews, MMIs, and CASPer across race, ethnicity, and gender.

RESULTS

1,237 applicants were interviewed (2018: n = 608; 2019: n = 629). Fifty-seven percent identified as female. Self-identified race/ethnicity included 758 White, 118 Black or African-American, 296 Asian, 20 Native American or Alaskan Native, 1 Native Hawaiian or Other Pacific Islander, and 44 No response; 87 applicants identified as Hispanic. Black or African-American, Native American or Alaskan Native, and Hispanic applicants had significantly lower CASPer scores than other applicants. Statistically significant differences in CASPer percentiles were identified for gender and race; however, between subjects, comparisons were not significant.

CONCLUSIONS

The CASPer test showed disparate scores across racial and ethnic groups in this cohort study and may not contribute to minimizing bias in medical school admissions.

Relationship between selection criteria and trainee performance in medical specialty training: A retrospective longitudinal study

BACKGROUND

Investigation of tools used for candidate selection as predictors of future performance in medical education has been primarily undertaken within the undergraduate setting, but little is known about the selection to medical specialist programs. This retrospective longitudinal study aims to explore correlations between selection tools and the performance of trainees enrolled in the Australasian College of Dermatologists (ACD), the accredited specialist medical college for training in dermatology in Australia.

METHODS

Data were collected from consecutive cohorts of ACD trainees commencing from 2007 to 2015. Predictive variables were trainee demographics and selection tools (prior academic qualification; research experience; clinical experience; curriculum vitae [CV] rating; interview scores). The outcome variables were exam performance. First Year ([Pharmacology and Clinical Sciences] and Fellowship Exam [Written, Clinical, and Overall Score]). Statistical analyses included: descriptive statistics summaries; bivariate correlation of selection criteria and demographic data with exam performance; and linear regression analysis to identify predictors of exam performance.

RESULTS

Demographic analysis (N = 172) showed that 64% of trainees were female, the average age was 30.5 years (± 3.47) and trainees living in high socioeconomic status (SES) areas were over-represented. Using Pearson correlation analysis, interview scores were significantly positively correlated with First Year Pharmacology results (p = 0.018), Fellowship Written results (p = 0.002), and Fellowship Overall Scores (p = 0.006). First Year Pharmacology Exam performance was most highly correlated with Fellowship Exam performance (p = 0.000). No association was identified between exam performance and gender, SES, prior academic qualification, research experience or CV rating. Linear regression analysis showed that interview score (p = 0.012), entry age (p = 0.026) and First Year Pharmacology score (p = 0.002) were predictors of Fellowship exam performance.

CONCLUSION

These results suggest that the comprehensive selection tools used by ACD are effective, with interviews being a highly valid tool linked to exam performance. These findings have important practical implications for assessing the selection process in specialist dermatology training.

Doctors in distress: The personality profile of derailing doctors

OBJECTIVE

This study looked at the bright and dark-side personality profile of distressed and potentially derailing doctors (82% male). The derailing doctors were on average 48.75 years old, and from many specialities, in particular, general practice and surgery.

METHOD

In all, a group of 77 derailing British doctors, and a control group of 357 doctors completed a valid multi-dimensional test of bright-side (normal) personality (NEO-PI-R) and one of dysfunctional interpersonal themes (sub-clinical personality disorders) (HDS: Hogan Development Survey).

RESULTS

Controlling for sex and age, the derailing doctors were more Neurotic (less resilient), and less Agreeable, Conscientious, Extraverted and Open-to- Experience. They were also more Excitable (Borderline), Sceptical (Paranoid), Cautious (Avoidant), Reserved (Schizoid), Leisurely (Passive Aggressive) and Bold (Narcissistic). Discriminant analysis showed age, Neuroticism, Extraversion, Leisurely and Excitable were, in that order, the greatest personality discriminators between those who did and did not derail.

CONCLUSION

More research needs to be done on doctor derailment to inform the selection and training of doctors.

Validity of constructed-response situational judgment tests in training programs for the health professions: A systematic review and meta-analysis protocol

BACKGROUND

Situational judgments tests have been increasingly used to help training programs for the health professions incorporate professionalism attributes into their admissions process. While such tests have strong psychometric properties for testing professional attributes and are feasible to implement in high-volume, high-stakes selection, little is known about constructed-response situational judgment tests and their validity.

METHODS

We will conduct a systematic review of primary published or unpublished studies reporting on the association between scores on constructed-response situational judgment tests and scores on other tests that measure personal, interpersonal, or professional attributes in training programs for the health professions. In addition to searching electronic databases, we will contact academics and researchers and undertake backward and forward searching. Two reviewers will independently screen the papers and decide on their inclusion, first based on the titles and abstracts of all citations, and then according to the full texts. Data extraction will be done independently by two reviewers using a data extraction form to chart study details and key findings. Studies will be assessed for the risk of bias and quality by two reviewers using the "Quality In Prognosis Studies" tool. To synthesize evidence, we will test the statistical heterogeneity and conduct a psychometric meta-analysis using a random-effects model. If adequate data are available, we will explore whether the meta-analytic correlation varies across different subgroups (e.g., race, gender).

DISCUSSION

The findings of this study will inform best practices for admission and selection of applicants for training programs for the health professions and encourage further research on constructed-response situational judgment tests, in particular their validity.

TRIAL REGISTRATION

The protocol for this systematic review has been registered in PROSPERO [CRD42022314561]. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022314561.

Can Different Admissions to Medical School Predict Performance of Non-Technical Skill Performance in Simulated Clinical Settings?

Non-technical skills (NTS) in medical care are essential to ensure patient safety. Focussing on applicants' NTS during medical school admission could be a promising approach to ensure that future physicians master NTS at a high level. Next to pre-university educational attainment, many selection tests have been developed worldwide to facilitate and standardise the selection process of medical students. The predictive validity of these tests regarding NTS performance in clinical settings has not been investigated (yet). Therefore, we explored the predictive validities and prognosis of the Hamburg MMI (HAM-Int), HAM-Nat, PEA, and waiting as well as other quota (as example) designated by the Federal Armed Forces) for NTS performance in clinical emergency medicine training of medical students. During 2017 and 2020, N = 729 second, third, and fourth year students were enrolled within the study. The mean age of participants was 26.68 years (SD 3.96) and 49% were female students. NTS of these students were assessed during simulation scenarios of emergency training with a validated rating tool. Students admitted via waiting quota and designated by the Armed Forces performed significantly better than students admitted by excellent PEA (p = 0.026). Non-EU students performed significantly inferior (p = 0.003). Our findings provide further insight to explain how and if admission to medical school could predict NTS performance of further physicians.

A preliminary study of the probitive value of personality assessment in medical school admissions within the United States

BACKGROUND

Allopathic medicine faces a daunting challenge of selecting the best applicants because of the very high applicant / matriculant ratio. The quality of graduates ultimately reflects the quality of medical practice. Alarming recent trends in physician burnout, misconduct and suicide raise questions of whether we are selecting the right candidates. The United States (US) lags far behind the United Kingdom (UK) and Europe in the study of non-cognitive tests in medical school admissions. Although more recently, medical schools in both the UK, Europe and the US have begun to use situational judgement tests such as the Computer-Based Assessment for Sampling Personal Characteristics (CASPer) and the situational judgement test (SJT), recently developed by the Association of American Medical Colleges (AAMC) and that these tests are, in a sense non-cognitive in nature, direct personality tests per se have not been utilized. We have historically used, in the admissions process within the US, knowledge, reasoning and exam performance, all of which are largely influenced by intelligence and also improved with practice. Personality, though also undoubtedly influenced by intelligence, is fundamentally different and subject to different kinds of measurements.

METHODS

A popular personality measurement used over the past two decades within the US in business and industry, but not medical school has been the Neo Personality Inventory - Revised (NEO-PI-R) Test. This test has not been utilized regularly in allopathic medicine probably because of the paucity of exploratory retrospective and validating prospective studies. The hypothesis which we tested was whether NEO-PI-R traits exhibited consistency between two institutions and whether their measurements showed probative value in predicting academic performance.

RESULTS

Our retrospective findings indicated both interinstitutional consistencies and both positive and negative predictive values for certain traits whose correlative strengths exceeded traditional premed metrics: medical college admission test (MCAT) scores, grade point average (GPA), etc. for early academic performance.

CONCLUSIONS

Our exploratory studies should catalyze larger and more detailed confirmatory studies designed to validate the importance of personality traits not only in predicting early medical school performance but also later performance in one's overall medical career.

'I'd have to fight for my life there': a multicentre qualitative interview study of how socioeconomic background influences medical school choice

Students from lower socio-economic backgrounds who were educated in state funded schools are underrepresented in medicine in the UK. Widening access to medical students from these backgrounds has become a key political and research priority. It is known that medical schools vary in the number of applicants attracted and accepted from non-traditional backgrounds but the reasons for this are poorly understood. This study aims to explore what applicants value when choosing medical schools to apply to and how this relates to their socioeconomic background. We conducted a multicentre qualitative interview study, purposively sampling applicants and recent entrants based on socioeconomic background, stage of application and medical school of application. We recruited participants from eight UK medical schools. Participants attended semi-structured interviews. We performed a framework analysis, identifying codes inductively from the data. Sixty-six individuals participated: 35 applicants and 31 first year medical students. Seven main themes were identified; course style, proximity to home, prestige, medical school culture, geographical area, university resources, and fitting in. These were prioritised differently depending on participants' background. Participants from lower socioeconomic backgrounds described proximity to home as a higher priority. This was typically as they intended to be living at home for at least part of the course. Those from higher socioeconomic backgrounds were more concerned with the perceived prestige of medical schools. Since medicine is a highly selective course, only offered at a minority of UK higher education institutions, these differences in priorities may help explain observed differential patterns of medical school applications and success rates by applicant social background.

Oral Health Literacy, Sense of Coherence and Associations With Poor School Performance

BACKGROUND

This study aimed to investigate the associations of oral health literacy (OHL), sense of coherence (SOC) and other covariates with self-reported school performance (SRSP) in adolescents enrolled in public schools in Brazil.

METHODS

This was a cross-sectional study conducted with adolescents aged from 14 to 18 years enrolled in 6 public schools in Piracicaba, São Paulo, Brazil. Researchers carried out data collection at schools using a self-administered structured questionnaire. The dependent variable was SRSP. The explanatory factors included were sociodemographic characteristics, OHL, SOC, self-rated oral health, health behaviors, and experience of bullying. Analyses included simple and multiple logistic regression models.

RESULTS

The final sample consisted of 385 adolescents with a mean age of 16 years. Poor school performance was associated (95%CI) with being male, with more than 1 sibling, individual whose last visit to the dentist was due to dental pain, with a lower level of OHL and SOC.

CONCLUSIONS

OHL and SOC were important factors associated with SRSP performance together with other sociodemographic and oral health-related variables in high school students. Our findings reinforce the importance of intersectoral actions between the health and education sectors to promote better school performance among adolescents, including actions to improve OHL and SOC.

An assessment of the impact of formal preparation activities on performance in the University Clinical Aptitude Test (UCAT): a national study

BACKGROUND

Previous studies have shown performance in the University Clinical Aptitude Test (UCAT) to be associated with measures of candidate socio-economic advantage such as parental occupation and type of school attended. It is possible that access to preparation support and materials may in part explain these associations. In this paper we determine whether use of formal preparation resources is associated with higher UCAT scores and whether differences in use of preparation resources exist between socio-demographic groups.

METHODS

After completing the 2017 UCAT UK school-leaver candidates (n = 14,332) were asked to answer a questionnaire regarding their use of official UCAT and commercial resources, school-based support, and time spent preparing. Multiple linear and logistic regression models were used to evaluate the associations between preparedness, demographic characteristics and UCAT performance.

RESULTS

Five thousand, four hundred thirty-nine (38%) candidates responded to the questionnaire. Use of freely available UCAT official practice tests, paid commercial materials, attendance at school-based preparation courses and spending more time preparing were significantly associated with higher UCAT scores. Candidates who were from less deprived backgrounds and attending independent or grammar schools were significantly more likely to use paid commercial materials and spend longer preparing.

CONCLUSIONS

Reported use of preparation resources varies between candidates from different socio-demographic backgrounds and is associated independently with performance in the UCAT. Increasing the availability of freely available resources may mitigate some of these differences.

Exploring variation in surgical practice: does surgeon personality influence anastomotic decision-making?

BACKGROUND

Decision-making under uncertainty may be influenced by an individual's personality. The primary aim was to explore associations between surgeon personality traits and colorectal anastomotic decision-making.

METHODS

Colorectal surgeons worldwide participated in a two-part online survey. Part 1 evaluated surgeon characteristics using the Big Five Inventory to measure personality (five domains: agreeableness; conscientiousness; extraversion; emotional stability; openness) in response to scenarios presented in Part 2 involving anastomotic decisions (i.e. rejoining the bowel with/without temporary stomas, or permanent diversion with end colostomy). Anastomotic decisions were compared using repeated-measure ANOVA. Mean scores of traits domains were compared with normative data using two-tailed t tests.

RESULTS

In total, 186 surgeons participated, with 127 surgeons completing both parts of the survey (68.3 per cent). One hundred and thirty-one surgeons were male (70.4 per cent) and 144 were based in Europe (77.4 per cent). Forty-one per cent (77 surgeons) had begun independent practice within the last 5 years. Surgeon personality differed from the general population, with statistically significantly higher levels of emotional stability (3.25 versus 2.97 respectively), lower levels of agreeableness (3.03 versus 3.74), extraversion (2.81 versus 3.38) and openness (3.19 versus 3.67), and similar levels of conscientiousness (3.42 versus 3.40 (all P <0.001)). Female surgeons had significantly lower levels of openness (P <0.001) than males (3.06 versus 3.25). Personality was associated with anastomotic decision-making in specific scenarios.

CONCLUSION

Colorectal surgeons have different personality traits from the general population. Certain traits seem to be associated with anastomotic decision-making but only in specific scenarios. Further exploration of the association of personality, risk-taking, and decision-making in surgery is necessary.

Best practices for interviewing applicants for medical school admissions: a systematic review

INTRODUCTION

Interviews are commonly used to select applicants for medical school, residency, and fellowship. However, interview techniques vary in acceptability, feasibility, reliability, and validity. This systematic review investigated the effectiveness of different interview methods in selecting the best qualified applicants for admission to medical school and developed a logic model to implement best practices for interviewing.

METHODS

Five electronic literature databases were searched for comparative studies related to interviewing in medical schools from inception through February 1, 2021. Inclusion criteria included publications in English that compared different methods of conducting a selection interview in medical schools with a controlled trial design. General study characteristics, measurement methodologies, and outcomes were reviewed. Quality appraisal was performed using the Medical Education Research Study Quality Instrument (MERSQI) and the Oxford Risk of Bias Scale. Based on these findings, a logic model was constructed using content analysis.

RESULTS

Thirteen studies were included. The multiple mini-interview (MMI) was reliable, unbiased, and predicted clinical and academic performance; the virtual MMI increased reliability and lowered costs. For unstructured interviews, blinding interviewers to academic scores reduced bias towards higher scorers; student and faculty interviewers rated applicants similarly. Applicants preferred structured over unstructured interviews. Study quality was above average per the MERSQI, risk of bias was high per the Oxford scale, and between-study heterogeneity was substantial.

DISCUSSION

There were few high-quality studies on interviewing applicants for admission to medical school; the MMI appears to offer a reliable method of interviewing. A logic model can provide a conceptual framework for conducting evidence-based admissions interviews.

Institutional choice among medical applicants: a profile paper for The United Kingdom Medical Applicant Cohort Study (UKMACS) prospective longitudinal cohort study

PURPOSE

To generate a large cohort of those in 2019 seriously considering applying to study Medicine, collecting data on a range of socioeconomic and other demographic factors that influence choice of medical schools and to link to other datasets to form a longitudinal study of progress through medical school and careers in medicine.

DESIGN

Cross-sectional questionnaire studies, part of the longitudinal UK Medical Applicant Cohort Study (UKMACS).

SETTING

UK medical school admissions in 2020.

PARTICIPANTS

UK residents aged 16+ and seriously considering applying to study Medicine. The cohort was primarily drawn from those registering in 2019 for the U(K)CAT (University Clinical Aptitude Test (formerly the UK Clinical Aptitude Test)) with additional potential applicants responding to an open call. Participants consented to their data being linked within the UK Medical Education Database.

FINDINGS TO DATE

UKMACS Wave 1 questionnaire respondents consisted of 6391 consenting respondents from across the UK. In 2019, 14 980 of the 17 470 UK-domiciled medicine applicants were first-time applicants. The questionnaires show that many of these applicants have a need for more help and guidance to make informed choices, with less advantaged groups reporting themselves as being at a disadvantage when applying due to limited understanding of information and limited access to guidance to enable informed and effective decision-making.

FUTURE PLANS

To link the cohort with successive Universities and Colleges Admissions Service and other datasets to analyse outcomes of applications and establish national longitudinal evidence to understand how medical choices are made and how they impact on educational, career and workforce outcomes.

BMAT's predictive validity for medical school performance: A retrospective cohort study

BACKGROUND

Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools.

METHODS

This was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students.

RESULTS

When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades.

DISCUSSION

Schools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.

Can selection interviews predict OSCE performance? Evidence from Hannover Medical School

OBJECTIVES

We analyze whether the student selection process at Hannover Medical School (MHH), which combined a semi-structured interview with school leaving grades, can predict performances in an Objective Structured Clinical Examination (OSCE). We also check whether there are differences between assessments of clinical knowledge, practical skills, and communication abilities.

METHODS

We use data from 525 medical students who were admitted after a successful selection process and who completed the OSCE in the years 2015-2019. We employ multivariate regressions and a mediation analysis approach to learn whether study success after admission and prior to the OSCE mediates the outcome of the latter.

RESULTS

A better performance in the MHH's selection interview is unrelated to success in the OSCE. However, there is a small but significant influence of school grades on OSCE results in each part except for the assessment of communication skills. The impact of the school grade is partially mediated by performances in written and oral exams preceding the OSCE.

DISCUSSION

School grades matter for the OSCE outcome, albeit to different degrees for more learning-based vs. practical parts of the examination. The interview at MHH was purely informative and unrelated to study success, also in the assessment of communication skills. Better structured interview tools may yield better results.

CONCLUSION

Students' cognitive abilities predict study success in an undergraduate OSCE. Performances in a semi-structured selection interview have no impact, not even the assessment of communication skills.

Quality Assurance of Undergraduate Medical Education in Israel by Continuous Monitoring and Prioritization of the Accreditation Standards

External accreditation reviews of undergraduate medical curricula play an important role in their quality assurance. However, these reviews occur only at 4-10-year intervals and are not optimal for the immediate identification of problems related to teaching. Therefore, the Standards of Medical Education in Israel require medical schools to engage in continuous, ongoing monitoring of their teaching programs for compliance with accreditation standards. In this paper, we propose the following: (1) this monitoring be assigned to independent medical education units (MEUs), rather than to an infrastructure of the dean's office, and such MEUs to be part of the school governance and draw their authority from university institutions; and (2) the differences in the importance of the accreditation standards be addressed by discerning between the "most important" standards that have been shown to improve student well-being and/or patient health outcomes; "important" standards associated with student learning and/or performance; "possibly important" standards with face validity or conflicting evidence for validity; and "least important" standards that may lead to undesirable consequences. According to this proposal, MEUs will evolve into entities dedicated to ongoing monitoring of the education program for compliance with accreditation standards, with an authority to implement interventions. Hopefully, this will provide MEUs and faculty with the common purpose of meeting accreditation requirements, and an agreed-upon prioritization of accreditation standards will improve their communication and recommendations to faculty.

Associations between communication OSCE performance and admission interviews in medical education

OBJECTIVE

To investigate associations between medical students' communication and other clinical skills assessed in OSCEs and MMIs performance upon admission by developing an assessment strategy based on an internationally acknowledged curriculum framework.

METHODS

Between July 2019 and March 2020, 365 medical students in the 4th and 5th semester took two OSCEs containing 10 5-minute stations examining communication and other clinical skills. We used a European conceptual framework to determine the content validity of the communication score and calculated scores of communication and other clinical skills. We assessed students' OSCE performance and estimated multiple regressions to predict its association with MMIs.

RESULTS

The multiple linear regressions showed that students' MMI performance upon admission is significantly associated with the communication score (b = 0.32, p = 0.006) but not the clinical score (b = 0.19, p = 0.121), when controlling for gender and cognitive criteria.

CONCLUSIONS

Our assessment strategy designed to distinguish between different areas of competence provides a more thorough description of the positive relationship between OSCE performance and MMIs.

PRACTICE IMPLICATIONS

We developed a communication skills assessment strategy that can be easily applied by medical schools that use OSCEs as a training or assessment method.

Recruitment and retention of volunteer multiple mini interview interviewers: Understanding their motivations

BACKGROUND

Without volunteer interviewers, many universities would not be able to run multiple mini interviews (MMIs) due to the prohibitive cost of paying interviewers. Despite the opportunity cost borne by volunteers, many interviewers participate in multiple MMI sessions per year and volunteer year after year. There is surprisingly little research into what motivates interviewers to volunteer as MMI interviewers. This research aims to explore both what motivates individuals to volunteer to interview in MMIs for undergraduate medical selection and what adds and detracts value from their participation.

METHODS

We applied a qualitative sequential two-phase design consisting of open-ended survey questions, followed by semi-structured interviews to explore interviewers' motivators in more depth. The survey data on motivators and the six functions from the volunteer functions inventory (VFI) informed interview data collection and provided a lens through which to examine MMI interviewer motivations. Content analysis was used to analyse the survey data. Framework analysis was used to analyse the interview data.

RESULTS

The survey was completed by 108 interviewers (50% response rate), and 19 semi-structured interviews were conducted (54% response rate). From the content analysis, the time commitment of involvement was the biggest detractor identified by participants. Through the framework analysis, five overarching motivators were developed: (i) acting on values, (ii) gaining understanding, (iii) gaining personal satisfaction and gratification, (iv) shaping the future workforce and (v) having social interaction. These mirrored five of the six functions proposed in the VFI.

CONCLUSIONS

There are a range of motivating factors that influenced the participants' decision to volunteer as an interviewer for MMIs. Some motivations were for the benefit of others, some were self-serving, and some a combination of both. Universities should utilise these motivating factors to aid in targeted recruitment of volunteer interviewers.

Personal statement versus psychological test as admission to the nursing degree: an evaluation

BACKGROUND

A review of admissions to nursing in Northern Ireland was prompted by the growing number of applications and a desire to ensure that the applicants had the right values for a career in nursing. Concerns regarding authorship, plagiarism and reliability of personal statements used to select applicants to interview was the focus of this research. This study evaluates the psychometric properties of a Personal Statement (PS) as a method for admission to a nursing programme and a values-based psychological screening tool, Nurse Match (NM).

METHODS

A self-selecting, purposive sample (n = 228; 9.7%) was drawn from applicants to Schools of Nursing in the United Kingdom (n = 2350). Participants all of whom had completed a Personal Statement were asked to complete a psychological tool and the scoring outcomes and psychometric properties of both tests were investigated. Statistical analysis was conducted using Minitab 17.

RESULTS

Applicants from 18 schools and five colleges responded. The majority (72.4%) were aged 18-19. Findings provide practical, theoretical, statistical, and qualitative reasons for concluding that the Personal Statement has substantial limitations as a measure of suitability. It does not compare well with international test standards for psychometric tests. In contrast, NM is a valid and reliable measure with good discriminatory power, standardised administration and consistent marking.

CONCLUSION

NM is a viable alternative to the PS for shortlisting applicants for nursing interviews.