2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions

Professional judgement: a social practice perspective on a multiple mini-interview for specialty training selection

BACKGROUND

Interviewers' judgements play a critical role in competency-based assessments for selection such as the multiple-mini-interview (MMI). Much of the published research focuses on the psychometrics of selection and the impact of rater subjectivity. Within the context of selecting for entry into specialty postgraduate training, we used an interpretivist and socio-constructivist approach to explore how and why interviewers make judgments in high stakes selection settings whilst taking part in an MMI.

METHODS

We explored MMI interviewers' work processes through an institutional observational approach, based on the notion that interviewers' judgements are socially constructed and mediated by multiple factors. We gathered data through document analysis, and observations of interviewer training, candidate interactions with interviewers, and interviewer meetings. Interviews included informal encounters in a large selection centre. Data analysis balanced description and explicit interpretation of the meanings and functions of the interviewers' actions and behaviours.

RESULTS

Three themes were developed from the data showing how interviewers make professional judgements, specifically by; 'Balancing the interplay of rules and agency,' 'Participating in moderation and shared meaning making; and 'A culture of reflexivity and professional growth.' Interviewers balanced the following of institutional rules with making judgment choices based on personal expertise and knowledge. They engaged in dialogue, moderation, and shared meaning with fellow interviewers which enabled their consideration of multiple perspectives of the candidate's performance. Interviewers engaged in self-evaluation and reflection throughout, with professional learning and growth as primary care physicians and supervisors being an emergent outcome.

CONCLUSION

This study offers insights into the judgment-making processes of interviewers in high-stakes MMI contexts, highlighting the balance between structured protocols and personal expertise within a socially constructed framework. By linking MMI practices to the broader work-based assessment literature, we contribute to advancing the design and implementation of more valid and fair selection tools for postgraduate training. Additionally, the study underscores the dual benefit of MMIs-not only as a selection tool but also as a platform for interviewers' professional growth. These insights offer practical implications for refining future MMI practices and improving the fairness of high-stakes selection processes.

Medical school service regions in Canada: exploring graduate retention rates across the medical education training continuum and into professional practice

PURPOSE

To create medical school service regions and examine national in-region graduate retention patterns across the medical education continuum and into professional practice as one approach to advancing social accountability in medical education.

METHODS

Medical school service regions were created in Canada using publicly available data and mapped using Geographic Information System (GIS) software. Population size and density for each service region were calculated using census data. Retrospective data of medical graduates who completed their medical degrees between 2001-2015 (n = 19,971) were obtained from a centralized data repository and used to analyze in-region retention rates by medical specialty across the training continuum and five years into professional practice.

RESULTS

Spatial inequities were observed across medical school service regions. Graduate retention patterns also varied across service region groups and medical specialties. Quebec (86.5%) and Ontario (80.4%) had above-average retention rates across the medical education continuum. Family medicine had the highest retention rates from undergraduate to postgraduate training (81.9%), while psychiatry had the highest retention rate across the training continuum and into professional practice (71.2%). The Alberta and British Columbia service region group demonstrated high retention rates across the training continuum and into professional practice and medical specialties, except for retention from undergraduate to postgraduate medical education.

CONCLUSION

This study highlights the importance of considering both medical specialty and practice location of graduates when planning and retaining the physician workforce. The observed retention patterns among graduates are a critical aspect of addressing societal needs and represent an intermediate step towards achieving health equity. Furthermore, graduate retention patterns serve as an outcome measure for schools to demonstrate their commitment to social accountability. Tracking and monitoring graduate outcomes may lead schools to actively collaborate with government agencies responsible for healthcare policy, which may ultimately improve physician workforce planning and promote more equitable healthcare access.

Developing institution-specific admission competency criteria for prospective health sciences students

BACKGROUND

Given the critical role of health professionals in societal health, the development of robust and effective selection methods is of fundamental concern for educational institutions within the field of health sciences education. Conventionally, admission competencies have been determined by institutional authorities. Developing institution-specific competency criteria enables an admission process that is mission- and value-aware, evidence-based, and strategically adaptable. However, few schools have established their admission competency criteria, although the majority possess their own models of graduation competencies. This study reports the process of developing and validating an institution-specific admission competency model that addresses the need for evidence-based and mission-aligned selection processes that are distinct from standardized models.

METHODS

This study was conducted in two phases, using both qualitative and quantitative analyses. Phase I involved constructing an admission competency model through a qualitative approach facilitated by workshops with 17 faculty members and 92 first-year pre-doctoral students of a dental school. Through constant comparative analysis, this phase focused on the extraction and refinement of competencies for entering dental students. In Phase II, a questionnaire developed from the workshops asked respondents to rate the importance of 47 attributes across 10 constructs on a 5-point Likert scale. A total of 301 individuals participated in the survey. Exploratory Factor Analysis (EFA) identified the factor structure, and Confirmatory Factor Analysis (CFA) examined construct validity and assessed the model fit with the data.

RESULTS

The EFA of the 47 attributes identified 10 factors, and the CFA results indicated a good-to-acceptable level of fit for the ten-factor model. Aligned with the American Association of Medical Colleges Premed competencies, this study identified unique attributes specific to the institution, such as confidence, leadership, and entrepreneurship. These findings highlight the importance of developing tailored competencies reflecting the unique needs of institutions and their fields.

CONCLUSIONS

This study demonstrates the feasibility and value of creating institution-specific admission competency models, offering a methodology that aligns with evidence-based mission-driven selection processes. The distinct competencies identified emphasize the need for educational institutions to consider unique institutional and field-specific requirements and move beyond standardized models to enhance the selection of medical students.

Can all roads lead to competency? School levels effects in Licensing examinations scores

At the foundation of research concerned with professional training is the idea of an assumed causal chain between the policies and practices of education and the eventual behaviours of those that graduate these programs. In medicine, given the social accountability to ensure that teaching and learning gives way to a health human resource that is willing and able to provide the healthcare that patients and communities need, it is of critical importance to generate evidence regarding this causal relationship. One question that medical education scholars ask regularly is the degree to which the unique features of training programs and learning environments impact trainee achievement of the intended learning outcomes. To date, this evidence has been difficult to generate because data pertaining to learners is only rarely systematically brought together across institutions or periods of training. We describe new research which leverages an inter-institutional data-driven approach to investigate the influence of school-level factors on the licensing outcomes of medical students. Specifically, we bring together sociodemographic, admissions, and in-training assessment variables pertaining to medical trainee graduates at each of the six medical schools in Ontario, Canada into multilevel stepwise regression models that determine the degree of association between these variables and graduate performances on the Medical Council of Canada Qualifying Examinations (Part 1, n = 1097 observations; Part 2, n = 616 observations), established predictors of downstream physician performance. As part of this analysis, we include an anonymized school-level (School 1, School 2) independent variable in each of these models. Our results demonstrate that the largest variable associated with performance on both the first and second parts of the licensing examinations is prior academic achievement, notably clerkship performance. Ratings of biomedical knowledge were also significantly associated with the first examination, while clerkship OSCE scores and enrollment in a family medicine residency were significantly associated with the Part 2. Small significant school effects were realized in both models accounting for 4% and 2% of the variance realized in the first and second examinations, respectively. These findings highlight that school enrollment plays a minor role relative to individual student performance in influencing examination outcomes.

Translating government policy into practice: How new UK medical schools enact widening participation

INTRODUCTION

Increasing the diversity of medical students, or widening participation (WP), is critical for social justice and healthcare delivery, and many governments are setting policies to encourage WP. However, establishing policy is only the first step in an educational change process: we also need to know "how" policy is enacted or how medical schools interpret and put into practice WP policy. Addressing this gap, the aim of this study was to examine policy enactment in six new UK medical schools with an explicit WP mandate.

METHODS

This qualitative study, underpinned by social constructivism, used semi-structured interviews to explore the experiences of key actors (6 Deans and 14 Admissions staff) of putting policy into practice when setting up a new medical school. Data coding and analysis were initially inductive, using thematic analysis. We then applied Ball's theory of policy enactment to organise the data into four contextual dimensions of 'situation', 'professional', 'material' and 'external'.

RESULTS

On the surface, there were many similarities across the six schools in terms of the four dimensions. However, how these dimensions interacted illuminated tensions and differences. For example, some schools found themselves increasingly subjected to local and extra-local rule systems, including pressure to follow host university norms and hosts struggling to accept that medical schools are heavily regulated. There were also tensions between the new medical schools and the medical education "establishment", including lack of power and being judged by overly narrow outcomes.

DISCUSSION

Different contexts seem to influence the enactment of WP to medicine in different places, even in the same country, even in medical schools established at the same time. That policy enactment is a complex, non-linear process of enactment is important to acknowledge: context is critical. Our findings will inform future policies and practices that aim to increase WP in medicine.

From policy to practice: Measuring success in widening participation

.@A_V_Simpson & Hope discuss the relevance of intersectionality in widening participation indicators while considering the importance of measuring policy success in a granular way.

Improving selection procedures in health professions education from the applicant perspective: an interview study

INTRODUCTION

Applicant perceptions of selection impact motivation and performance during selection, and student diversity. However, in-depth insight into which values underly these perceptions is lacking, creating challenges for aligning selection procedures with applicant perceptions. This qualitative interview study aimed to identify values applicants believe should underlie selection, and how, according to applicants, these values should be used to make specific improvements to selection procedures in undergraduate health professions education (HPE).

METHODS

Thirty-one applicants to five undergraduate HPE programs in the Netherlands participated in semi-structured interviews using Appreciative Inquiry, an approach that focuses on what goes well to create vision for improvement, to guide the interviews. Transcriptions were analyzed using thematic analysis, adopting a constructivist approach.

RESULTS

Applicants' values related to the aims of selection, the content of selection, and the treatment of applicants. Applicants believed that selection procedures should aim to identify students who best fit the training and profession, and generate diverse student populations to fulfill societal needs. According to applicants, the content of selection should be relevant for the curriculum and profession, assess a comprehensive set of attributes, be of high quality, allow applicants to show who they are, and be adapted to applicants' current developmental state. Regarding treatment, applicants believed that selection should be a two-way process that fosters reflection on study choice, be transparent about what applicants can expect, safeguard applicants' well-being, treat all applicants equally, and employ an equitable approach by taking personal circumstances into account. Applicants mentioned specific improvements regarding each value.

DISCUSSION

Applicants' values offer novel insights into what they consider important preconditions for the design of selection procedures. Their suggested improvements can support selection committees in better meeting applicants' needs.

Follow the policy: An actor network theory study of widening participation to medicine in two countries

INTRODUCTION

The slow pace of change in respect of increasing the diversity of medical students suggests powerful actors are reproducing practices to support the status quo. Opening up medicine to embrace diversity thus requires the deconstruction of entrenched processes and practices. The first step in doing so is to understand how the actor-network of widening participation and access to medicine (WP/WA) is constructed. Thus, here we examine how the connections among actors in WP/WA in two different networks are assembled.

METHODS

A comparative case study using documents (n = 7) and interviews with staff and students (n = 45) from two medical schools, one United Kingdom and one Australian, was used. We used Callon's moments of translation (problematisation, interessement/operationalisation, enrolment, mobilisation) to map the network of actors as they are assembled in relation to one another. Our main actant was institutional WP to medicine policy (actor-as-policy).

RESULTS

Our actor-as-policy introduced five other actors: the medical school, medical profession, high schools, applicants and medical school staff. In terms of problematisation, academic excellence holds firm as the obligatory passage point and focal challenge for all actors in both countries. The networks are operationalised via activities such as outreach and admissions policy (e.g., affirmative action is apparent in Australia but not the UK). High schools play (at best) a passive role, but directed by the policy, the medical schools and applicants work hard to achieve WP/WA to medicine. In both contexts, staff are key mobilisers of WP/WA, but with little guidance in how to enact policy. In Australia, policy drivers plus associated entry structures mean the medical profession exerts significant influence.

CONCLUSIONS

Keeping academic excellence as the obligatory passage point to medical school shapes the whole network of WP/WA and perpetuates inequality. Only by addressing this can the network reconfigure.

[Integration of natural sciences and basic medical subjects in the integrated dentistry program (iMED DENT) at the University of Hamburg]

In October 2019, an integrated dentistry program (iMED DENT) was implemented at the University of Hamburg and was the first of its kind in Germany. This model curriculum builds on didactic concepts that have been applied successfully for many years in curricula for human medicine, including interdisciplinary teaching, early clinical experience, and scientific education. The first year focuses on the healthy situation ("normal function") and aims to integrate the natural sciences (biology, chemistry, physics) and the basic medical subjects (anatomy, biochemistry, physiology, medical terminology) in the context of dental health. Further, basic practical and clinical tasks are assigned to the students during the first year.From the experience of the first four cohorts, initial conclusions can be drawn about this stage of study. Generally, its modular structure results in a condensation of learning content, which students judge as demanding. However, its interdisciplinary approach is well accepted. For instance, presenting the basics of the natural sciences in the context of their dental relevance is much better evaluated in the new compared to the previous curriculum, in which this content was taught without specific references to dental health. Teaching the basics of medicine within clinical context and the inclusion of early clinical practice are similarly appreciated. Presently, the interdisciplinary approach is limited by the focus on practical competencies of the dentistry curriculum, as some practical courses offer only few opportunities for other disciplines to interconnect their teaching. The continuous evaluation of the curriculum and exchange of experiences between the disciplines will further improve the integrative concept of the curriculum.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Cross-sectional evaluation of an asynchronous multiple mini-interview (MMI) in selection to health professions training programmes with 10 principles for fairness built-in

OBJECTIVES

We aimed to explore the psychometric properties of the first known online asynchronous multiple mini-interview (MMI) designed for fairness with subgroup analyses by key characteristics, usability and acceptability.

DESIGN

Cross-discipline multimethod evaluation.

SETTING

One UK University.

PARTICIPANTS

Applicants to nursing, midwifery and paramedic science undergraduate programmes during 2021-2022.

PRIMARY, SECONDARY OUTCOME MEASURES

Psychometric properties (internal consistency, construct validity, dimensionality) were assessed using Cronbach's alpha (α), parallel analysis (PA), Schmid-Leiman transformation and ordinal confirmatory factor analysis (CFA). Usability and acceptability were evaluated using descriptive statistics and conventional content analysis.

METHODS

The system was configured in a seven question 4 min MMI. Applicants' videorecorded their answers which were later assessed by interviewers and scores summed. Applicants and interviewers completed online evaluation questionnaires.

RESULTS

Performance data from 712 applicants determined good-excellent reliability for the asynchronous MMI (mean α 0.72) with similar results across subgroups (gender, age, disability/support needs, UK/non-UK). PA and factor analysis results suggested there were seven factors relating to the MMI questions with an underlying general factor that explained the variance in observed candidate responses. A CFA testing a seven-factor hierarchical model showed an excellent fit to the data (Confirmatory Fit Index=0.99), Tucker Lewis Index=0.99, root mean square error (RMSE) =0.034). Applicants (n=210) viewed the flexibility, relaxed environment and cost savings advantageous. Interviewers (n=65) reported the system to be intuitive, flexible with >70% time saved compared with face-to-face interviews. Reduced personal communication was cited as the principal disadvantage.

CONCLUSIONS

We found that the asynchronous MMI was reliable, time-efficient, fair and acceptable and building fairness in was lost-cost. These novel, insights are applicable across health professions selection internationally informing the future configuration of online interviews to ensure workforces represent the societies they serve.

Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153

Selection is the first assessment of medical education and training. Medical schools must select from a pool of academically successful applicants and ensure that the way in which they choose future clinicians is robust, defensible, fair to all who apply and cost-effective. However, there is no comprehensive and evidence-informed guide to help those tasked with setting up or rejuvenating their local selection process. To address this gap, our guide draws on the latest research, international case studies and consideration of common dilemmas to provide practical guidance for designing, implementing and evaluating an effective medical school selection system. We draw on a model from the field of instructional design to frame the many different activities involved in doing so: the ADDIE model. ADDIE provides a systematic framework of Analysis (of the outcomes to be achieved by the selection process, and the barriers and facilitators to achieving these), Design (what tools and content are needed so the goals of selection are achieved), Development (what materials and resources are needed and available), Implementation (plan [including piloting], do study and adjust) and Evaluation (quality assurance is embedded throughout but the last step involves extensive evaluation of the entire process and its outcomes).HIGHLIGHTSRobust, defensible and fair selection into medical school is essential. This guide systematically covers the processes required to achieve this, from needs analysis through design, development and implementation, to evaluation of the success of a selection process.

Commentary on robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153

The AMEE Guide to Selection for medical school is a welcome addition that provides much sound advice and guidance. It employs a comprehensive framework and a number of innovations, international case studies, for instance. There are also some omissions that a future revision could usefully address. The key ones concern the evidence base for assessment of personal attributes by questionnaire or interview; conflation of two separable stages in selection, meeting minimum requirements for suitability, and discriminating between suitable candidates; how best to provide feedback to candidates; and the question of what counts as a fair, equitable approach to selection. Nevertheless, the new AMEE Guide (No 153) is well-aligned with the most recent Ottawa consensus statement on selection, and will make a good contribution to the development or revision of selection systems in medical schools.

Global Perceptions on Social Accountability and Outcomes: A Survey of Medical Schools

Phenomenon: Social accountability has become a universal component in medical education. However, medical schools have little guidance for operationalizing and applying this concept in practice. This study explored institutional practices and administrative perceptions of social accountability in medical education. Approach: An online survey was distributed to a purposeful sample of English-speaking undergraduate medical school deans and program directors/leads from 245 institutions in 14 countries. The survey comprised of 38-items related to program mission statements, admission processes, curricular content, and educational outcomes. Survey items were developed using previous literature and categorized using a context-input-process-products (CIPP) evaluation model. Exploratory Factor Analysis (EFA) was used to assess the inter-relationship among survey items. Reliability and internal consistency of items were evaluated using McDonald's Omega. Findings: Results from 81 medical schools in 14 countries collected between February and June 2020 are presented. Institutional commonalities of social accountability were observed. However, our findings suggest programs focus predominately on educational inputs and processes, and not necessarily on outcomes. Findings from our EFA demonstrated excellent internal consistency and reliability. Four-factors were extracted: (1) selection and recruitment; (2) institutional mandates; (3) institutional activities; and (4) community awareness, accounting for 71% of the variance. McDonald's Omega reliability estimates for subscales ranged from 0.80-0.87. Insights: This study identified common practices of social accountability. While many medical schools expressed an institutional commitment to social accountability, their effects on the community remain unknown and not evaluated. Overall, this paper offers programs and educators a psychometrically supported tool to aid in the operationalization and reliability of evaluating social accountability.

Synthesis and perspectives from the Ottawa 2022 conference on the assessment of competence

INTRODUCTION

The Ottawa Conference on the Assessment of Competence in Medicine and the Healthcare Professions was first convened in 1985 in Ottawa. Since then, what has become known as the Ottawa conference has been held in various locations around the world every 2 years. It has become an important conference for the community of assessment - including researchers, educators, administrators and leaders - to share contemporary knowledge and develop international standards for assessment in medical and health professions education.

METHODS

The Ottawa 2022 conference was held in Lyon, France, in conjunction with the AMEE 2022 conference. A diverse group of international assessment experts were invited to present a symposium at the AMEE conference to summarise key concepts from the Ottawa conference. This paper was developed from that symposium.

RESULTS AND DISCUSSION

This paper summarises key themes and issues that emerged from the Ottawa 2022 conference. It highlights the importance of the consensus statements and discusses challenges for assessment such as issues of equity, diversity, and inclusion, shifts in emphasis to systems of assessment, implications of 'big data' and analytics, and challenges to ensure published research and practice are based on contemporary theories and concepts.

Does medical education reform change who is selected? A national cross-sectional survey from China

OBJECTIVES

Approximately 10 years ago, China introduced an education plan to improve the overall quality of medical education and to better serve the population's health needs. Many medical schools were then recognised and financed by China's Ministry of Education to develop and operationalise new pilot programmes (PPs) aligned with this plan. These ran in parallel with the traditional programmes (TPs). One way to achieve the plan's first aim, improving the quality of medical education, is to select academically stronger candidates. We, thus, examined and compared who were selected into PPs and TPs.

DESIGN

Cross-sectional study.

SETTING

Data were collected from 123 medical schools across China via the 2021 China Medical Student Survey.

PARTICIPANTS

Participants were undergraduate clinical medicine students across all year groups.

PRIMARY AND SECONDARY OUTCOME MEASURES

Medical school selection was via the National College Entrance Examination (NCEE). Medical students' NCEE performance and their sociodemographics were used as the primary and secondary outcome measures. Mann-Whitney or χ2 tests were used to compare the means between educational programmes (PPs vs TPs) and various selection outcomes. Multilevel mixed-effects regressions were employed to account for school idiosyncratic selection results.

RESULTS

Of the 204 817 respondents, 194 163 (94.8%) were in a TP and 10 654 (5.2%) a PP. PP respondents (median=75.2, IQR=69.5-78.8) had significantly higher NCEE scores than their TP counterparts (median=73.9, IQR=68.5-78.7). Holding constant their NCEE score, PP respondents were significantly more likely to come from urban areas, not be first-generation college students, and have parents with higher occupational status and income.

CONCLUSIONS

Assuming quality can be indicated by prior academic achievement at the point of selection, PPs achieved this mission. However, doing so limited medical students' diversity. This may be unhelpful in achieving the Education Plan's goal to better serve China's health needs.

What is the evidence for biology as the 'heart of eligibility' to study medicine? A retrospective analysis

The first, sometimes the only, selection tool for entry into undergraduate medicine is prior educational attainment (PEA). This is often further specified to include certain subjects, for example, biology is a prerequisite for entry into medicine in many Asian countries. However, there seems no clear evidence base for this prerequisite. Our aim, therefore, was to carry out a retrospective quantitative study comparing the performances of five cohorts of students (2015-2019 entry; n = 588) with and without biology PEA in Years 1 and 2 Bachelor of Medicine and Bachelor of Surgery (MBBS) integrated written assessments (n = 3) and anatomy practical examinations (APE) (n = 5). The study was conducted at one of Singapore's three medical schools. Data were analyzed using independent t-tests and Mann-Whitney U with p values of less than 0.05 were considered significant. There were no significant differences in performance on any Years 1 or 2 integrated written assessments. Similarly, in one of the APE, a significant difference was found for one cohort (academic year [AY] 2015-2016) out of five assessments. These results suggest that having a prior biology qualification does not make a difference in assessment performance in the early years of medical school. This information may help stakeholders and admissions committees decide whether biology is required for medical school entrance.

Does doctors' personality differ from those of patients, the highly educated and other caring professions? An observational study using two nationally representative Australian surveys

OBJECTIVES

Personality differences between doctors and patients can affect treatment outcomes. We examine these trait disparities, as well as differences across medical specialities.

DESIGN

Retrospective, observational statistical analysis of secondary data.

SETTING

Data from two data sets that are nationally representative of doctors and the general population in Australia.

PARTICIPANTS

We include 23 358 individuals from a representative survey of the general Australian population (with subgroups of 18 705 patients, 1261 highly educated individuals and 5814 working in caring professions) as well as 19 351 doctors from a representative survey of doctors in Australia (with subgroups of 5844 general practitioners, 1776 person-oriented specialists and 3245 technique-oriented specialists).

MAIN OUTCOME MEASURES

Big Five personality traits and locus of control. Measures are standardised by gender, age and being born overseas and weighted to be representative of their population.

RESULTS

Doctors are significantly more agreeable (a: standardised score -0.12, 95% CIs -0.18 to -0.06), conscientious (c: -0.27 to -0.33 to -0.20), extroverted (e: 0.11, 0.04 to 0.17) and neurotic (n: 0.14, CI 0.08 to 0.20) than the general population (a: -0.38 to -0.42 to -0.34, c: -0.96 to -1.00 to -0.91, e: -0.22 to -0.26 to -0.19, n: -1.01 to -1.03 to -0.98) or patients (a: -0.77 to -0.85 to -0.69, c: -1.27 to -1.36 to -1.19, e: -0.24 to -0.31 to -0.18, n: -0.71 to -0.76 to -0.66). Patients (-0.03 to -0.10 to 0.05) are more open than doctors (-0.30 to -0.36 to -0.23). Doctors have a significantly more external locus of control (0.06, 0.00 to 0.13) than the general population (-0.10 to -0.13 to -0.06) but do not differ from patients (-0.04 to -0.11 to 0.03). There are minor differences in personality traits among doctors with different specialities.

CONCLUSIONS

Several personality traits differ between doctors, the population and patients. Awareness about differences can improve doctor-patient communication and allow patients to understand and comply with treatment recommendations.

Helping healthcare teams to debrief effectively: associations of debriefers' actions and participants' reflections during team debriefings

BACKGROUND

Debriefings help teams learn quickly and treat patients safely. However, many clinicians and educators report to struggle with leading debriefings. Little empirical knowledge on optimal debriefing processes is available. The aim of the study was to evaluate the potential of specific types of debriefer communication to trigger participants' reflection in debriefings.

METHODS

In this prospective observational, microanalytic interaction analysis study, we observed clinicians while they participated in healthcare team debriefings following three high-risk anaesthetic scenarios during simulation-based team training. Using the video-recorded debriefings and INTERACT coding software, we applied timed, event-based coding with DE-CODE, a coding scheme for assessing debriefing interactions. We used lag sequential analysis to explore the relationship between what debriefers and participants said. We hypothesised that combining advocacy (ie, stating an observation followed by an opinion) with an open-ended question would be associated with participants' verbalisation of a mental model as a particular form of reflection.

RESULTS

The 50 debriefings with overall 114 participants had a mean duration of 49.35 min (SD=8.89 min) and included 18 486 behavioural transitions. We detected significant behavioural linkages from debriefers' observation to debriefers' opinion (z=9.85, p<0.001), from opinion to debriefers' open-ended question (z=9.52, p<0.001) and from open-ended question to participants' mental model (z=7.41, p<0.001), supporting our hypothesis. Furthermore, participants shared mental models after debriefers paraphrased their statements and asked specific questions but not after debriefers appreciated their actions without asking any follow-up questions. Participants also triggered reflection among themselves, particularly by sharing personal anecdotes.

CONCLUSION

When debriefers pair their observations and opinions with open-ended questions, paraphrase participants' statements and ask specific questions, they help participants reflect during debriefings.

Enhancing authenticity, diagnosticity and equivalence (AD-Equiv) in multicentre OSCE exams in health professionals education: protocol for a complex intervention study

INTRODUCTION

Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners' judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees' focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners' judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students' performances.

METHODS AND ANALYSIS

The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students' scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data.

ETHICS AND DISSEMINATION

The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.

BMAT's predictive validity for medical school performance: A retrospective cohort study

BACKGROUND

Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools.

METHODS

This was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students.

RESULTS

When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades.

DISCUSSION

Schools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.

Selection for health professions education leads to increased inequality of opportunity and decreased student diversity in The Netherlands, but lottery is no solution: A retrospective multi-cohort study

BACKGROUND

Concerns exist about the role of selection in the lack of diversity in health professions education (HPE). In The Netherlands, the gradual transition from weighted lottery to selection allowed for investigating the variables associated with HPE admission, and whether the representativeness of HPE students has changed.

METHOD

We designed a retrospective multi-cohort study using Statistics Netherlands microdata of all 16-year-olds on 1 October 2008, 2012, and 2015 (age cohorts, N > 600,000) and investigated whether they were eligible students for HPE programs (n > 62,000), had applied (n > 14,000), and were HPE students at age 19 (n > 7500). We used multivariable logistic regression to investigate which background variables were associated with becoming an HPE student.

RESULTS

HPE students with ≥1 healthcare professional (HP) parent, ≥1 top-10% income/wealth parent, and women are overrepresented compared to all age cohorts. During hybrid lottery/selection (cohort-2008), applicants with ≥1 top-10% wealth parent and women had higher odds of admission. During 100% selection (cohort-2015) this remained the case. Additionally, applicants with ≥1 HP parent had higher odds, those with a migration background had lower odds.

CONCLUSIONS

Odds of admission are increasingly influenced by applicants' backgrounds. Targeted recruitment and equitable admissions procedures are required to increase matriculation of underrepresented students.

Insights on selection of undergraduate dental students

INTRODUCTION

The goal of selection to dental education was to find those with the greatest aptitude for dentistry. Recently, schools have introduced a variety of tools; however, these have often been adopted without appropriate evaluation regarding existing evidence for fairness, reliability or validity. We explore dental admissions staff beliefs about the quality of different selection tools, with the objective of exploring their decision-making in implementing selection practices.

METHODS

This qualitative study is underpinned by a social constructionist epistemology, in which our principal concern is "explicating the processes by which people come to describe, explain or otherwise account for the world (including themselves) in which they live." We conducted individual interviews with 15 of the 16 UK dental admission leads to elicit their views around admissions processes and aims. Data coding and analysis were initially inductive, using thematic analysis. After the themes emerged, we applied a deductive framework of affordances to group themes and then examined these for heuristics.

RESULTS AND DISCUSSION

We identified three main themes; "Selection Tool Use," "Widening Participation Practices," and "Professionalising the Admission Lead Role." Admission leads spoke favourably of tools that allowed a "holistic" view of the applicants "potential". Selection tools were favoured if they enabled "Gut feeling". Leads spoke of evaluating candidates, making sure they were "rounded", and "know what dentistry is all about." In justifying the use of elements of their procedure, the use of heuristics was prominent.

CONCLUSION

In order to minimise the potential consequences of poor selection decisions, it is important to acknowledge that dental admission leads are at risk of depending on sub-optimal heuristics to make judgements about effective selection (shaped by previous practices) rather than using more rational decision-making processes based on the extant evidence (regarding the quality of different selection tools). Future research may be usefully informed by the knowledge translation literature to offer solutions for improving selection practices in dental education.

Personal statement versus psychological test as admission to the nursing degree: an evaluation

BACKGROUND

A review of admissions to nursing in Northern Ireland was prompted by the growing number of applications and a desire to ensure that the applicants had the right values for a career in nursing. Concerns regarding authorship, plagiarism and reliability of personal statements used to select applicants to interview was the focus of this research. This study evaluates the psychometric properties of a Personal Statement (PS) as a method for admission to a nursing programme and a values-based psychological screening tool, Nurse Match (NM).

METHODS

A self-selecting, purposive sample (n = 228; 9.7%) was drawn from applicants to Schools of Nursing in the United Kingdom (n = 2350). Participants all of whom had completed a Personal Statement were asked to complete a psychological tool and the scoring outcomes and psychometric properties of both tests were investigated. Statistical analysis was conducted using Minitab 17.

RESULTS

Applicants from 18 schools and five colleges responded. The majority (72.4%) were aged 18-19. Findings provide practical, theoretical, statistical, and qualitative reasons for concluding that the Personal Statement has substantial limitations as a measure of suitability. It does not compare well with international test standards for psychometric tests. In contrast, NM is a valid and reliable measure with good discriminatory power, standardised administration and consistent marking.

CONCLUSION

NM is a viable alternative to the PS for shortlisting applicants for nursing interviews.

Working intentions of medical students in response to healthcare workplace violence and descending resources reform in China

BACKGROUND

In order to curb healthcare workplace violence (WPV) and better allocate healthcare resources, China launched the descending resources reform in 2013 and tightened the anti-violence legal environment simultaneously. Medical students are expected to reconsider their working intentions of entering the medical market (inter-market effect) and choosing high- or low-level hospitals (intra-market effect) in response to the evolving WPV. The goal of this study was to explore the link between the perceived WPV incidence and medical students' willingness to practice medicine in the context of China's descending resources reform.

METHOD

Medical students were selected with cluster sampling from 8 medical colleges in Zhejiang Province, China, and 1497 valid questionnaires were collected by using a five-point unbalanced scale, to perform cross-sectional empirical research using the ordered logit model (OLM).

RESULTS

The perceived WPV incidence negatively correlate with the willingness of medical students to practice medicine but positively correlate with their willingness to practice in low-level hospitals, indicating the existence of inter- and intra-market effects. The anti-violence legal environment has no direct link with working intention but contributes to the perceived decline in the incidence of violence. Descending resources reform has simultaneous opposite effects on medical students, with the coexistence of prudent motives driven by reform costs and optimistic expectations of sharing external benefits.

CONCLUSIONS

Safety needs and risk aversion motive play an important role in medical students' career choice when facing severe WPV. Tightening of the anti-violence legal environment and the descending resources reform could drive medical students to low-level hospitals.

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues

BACKGROUND

Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile.

METHODS

Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508).

RESULTS

The SJT demonstrated good internal consistency (Cronbach's alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity.

CONCLUSIONS

This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

UCAT and dental student selection in the UK - what has changed?

Introduction The University Clinical Aptitude Test (UCAT) has been used since 2006 by a consortium of UK medical and dental schools to assist in undergraduate selection. In 2019, UCAT was used by 30 universities (14 dental schools).

Aim To report how UCAT use has changed in undergraduate student selection in the UK.

Methods UCAT use was categorised and trends identified from annual telephone interviews with dental school admission tutors; this process started in 2011.

Results Dental schools using UCAT rose from 8 (2006) to 14 (2020). The most significant use of the test to select applicants for interview was as a weighted factor; at offer stage, UCAT was most used to discriminate between applicants at borderlines. A growing number of dental schools are using the Situational Judgement Test (SJT) in selection (2019, n = 6). In 2019, eight schools adjusted selection processes for widening access applicants. Multiple mini interviews are now used by the majority (n = 10) of dental schools.

Conclusions UCAT represents a significant factor in selection to UK undergraduate dental programmes and is used by all but two dental schools. In most schools, UCAT contributes in a substantial way to selection outcomes and strength in test use has grown over time.

A description of how UCAT is used in dental student selection in the UK.

An insight into a unique data set specific to dental student selection in the UK.

An update on how UCAT use has changed over the years in dental student selection in the UK.

Feasibility of an automated interview grounded in multiple mini interview (MMI) methodology for selection into the health professions: an international multimethod evaluation

OBJECTIVES

Global, COVID-driven restrictions around face-to-face interviews for healthcare student selection have forced admission staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first automated interview grounded in multiple mini-interview (MMI) methodology. This study aimed to explore test-retest reliability, acceptability and usability of the system.

DESIGN, SETTING AND PARTICIPANTS

Multimethod feasibility study in Physician Associate programmes from two UK and one US university during 2019-2020.

PRIMARY, SECONDARY OUTCOMES

Feasibility measures (test-retest reliability, acceptability and usability) were assessed using intraclass correlation (ICC), descriptive statistics, thematic and content analysis.

METHODS

Volunteers took (T1), then repeated (T2), the automated MMI, with a 7-day interval (±2) then completed an evaluation questionnaire. Admission staff participated in focus group discussions.

RESULTS

Sixty-two students and seven admission staff participated; 34 students and 4 staff from UK and 28 students and 3 staff from US universities. Good-excellent test-retest reliability was observed at two sites (US and UK2) with T1 and T2 ICC between 0.65 and 0.81 (p<0.001) when assessed by individual total scores (range 80.6-119), station total scores 0.6-0.91, p<0.005 and individual site (≥0.79 p<0.001). Mean test re-test ICC across all three sites was 0.82 p<0.001 (95% CI 0.7 to 0.9). Admission staff reported potential to reduce resource costs and bias through a more objective screening tool for preselection or to replace some MMI stations in a 'hybrid model'. Maintaining human interaction through 'touch points' was considered essential. Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.

CONCLUSION

These preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection.

Does performance at the intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examination vary according to UK medical school and course type? A retrospective cohort study

OBJECTIVES

The knowledge, skills and behaviours required of new UK medical graduates are the same but how these are achieved differs given medical schools vary in their mission, curricula and pedagogy. Medical school differences seem to influence performance on postgraduate assessments. To date, the relationship between medical schools, course types and performance at the Membership of the Royal Colleges of Surgeons examination (MRCS) has not been investigated. Understanding this relationship is vital to achieving alignment across undergraduate and postgraduate training, learning and assessment values.

DESIGN AND PARTICIPANTS

A retrospective longitudinal cohort study of UK medical graduates who attempted MRCS Part A (n=9730) and MRCS Part B (n=4645) between 2007 and 2017, using individual-level linked sociodemographic and prior academic attainment data from the UK Medical Education Database.

METHODS

We studied MRCS performance across all UK medical schools and examined relationships between potential predictors and MRCS performance using χ2 analysis. Multivariate logistic regression models identified independent predictors of MRCS success at first attempt.

RESULTS

MRCS pass rates differed significantly between individual medical schools (p<0.001) but not after adjusting for prior A-Level performance. Candidates from courses other than those described as problem-based learning (PBL) were 53% more likely to pass MRCS Part A (OR 1.53 (95% CI 1.25 to 1.87) and 54% more likely to pass Part B (OR 1.54 (1.05 to 2.25)) at first attempt after adjusting for prior academic performance. Attending a Standard-Entry 5-year medicine programme, having no prior degree and attending a Russell Group university were independent predictors of MRCS success in regression models (p<0.05).

CONCLUSIONS

There are significant differences in MRCS performance between medical schools. However, this variation is largely due to individual factors such as academic ability, rather than medical school factors. This study also highlights group level attainment differences that warrant further investigation to ensure equity within medical training.

An investigation of the predictive validity of selection tools on performance in physiotherapy training in Australia

OBJECTIVES

Despite a large body of research on selection in medical education, very little is conducted in other health professions. This study investigated the predictive validity of multiple selection tools on academic and clinical performance outcomes of undergraduate physiotherapy students.

DESIGN

A retrospective observational study.

SETTING

Undergraduate physiotherapy program in Australia.

PARTICIPANTS

497 undergraduate physiotherapy students across seven entry cohorts. Including students directly from secondary school (n=381) and with prior tertiary study (n=116).

MAIN OUTCOME MEASURES

Academic performance as measured by written examinations. Clinical performance, measured by Objective Structured Clinical Examinations (OSCEs) during on-campus units and the Assessment of Physiotherapy Practice (APP) for off-campus clinical placements. Predictor variables included selection tools (academic achievement, interview, aptitude test) and demographic variables (age, gender).

RESULTS

Selection interview was a positive predictor of OSCEs and final year clinical performance in direct school leaver participants. Academic achievement scores from selection positively predicted written examinations scores.

CONCLUSION

Clinical and academic performance were predicted by tools measuring different domains at selection. Assessing broadly across academic and non-academic domains at selection can be valuable in identifying applicants who will be able to meet the range of outcomes for course completion and subsequent registration in the physiotherapy profession.

Increasing doctors working in specific rural regions through selection from and training in the same region: national evidence from Australia

BACKGROUND

'Grow your own' strategies are considered important for developing rural workforce capacity. They involve selecting health students from specific rural regions and training them for extended periods in the same regions, to improve local retention. However, most research about these strategies is limited to single institution studies that lack granularity as to whether the specific regions of origin, training and work are related. This national study aims to explore whether doctors working in specific rural regions also entered medicine from that region and/or trained in the same region, compared with those without these connections to the region. A secondary aim is to explore these associations with duration of rural training.

METHODS

Utilising a cross-sectional survey of Australian doctors in 2017 (n = 6627), rural region of work was defined as the doctor's main work location geocoded to one of 42 rural regions. This was matched to both (1) Rural region of undergraduate training (< 12 weeks, 3-12 months, > 1 university year) and (2) Rural region of childhood origin (6+ years), to test association with returning to work in communities of the same rural region.

RESULTS

Multinomial logistic regression, which adjusted for specialty, career stage and gender, showed those with > 1 year (RRR 5.2, 4.0-6.9) and 3-12 month rural training (RRR 1.4, 1.1-1.9) were more likely to work in the same rural region compared with < 12 week rural training. Those selected from a specific region and having > 1-year rural training there related to 17.4 times increased chance of working in the same rural region compared with < 12 week rural training and metropolitan origin.

CONCLUSION

This study provides the first national-scale empirical evidence supporting that 'grow your own' may be a key workforce capacity building strategy. It supports underserviced rural areas selecting and training more doctors, which may be preferable over policies that select from or train doctors in 'any' rural location. Longer training in the same region enhances these outcomes. Reorienting medical training to selecting and training in specific rural regions where doctors are needed is likely to be an efficient means to correcting healthcare access inequalities.

Accepting randomness in medical school admissions: The case for a lottery

McMaster University's medical school, faced with the need to socially distance during the COVID-19 pandemic, recently replaced their structured admission interview process with a partial lottery. At first, it may seem that leaving medical school admissions partly to chance could erode autonomy and meritocracy. Yet our current system for selecting medical students is strained by a limited predictive ability. In the search for good doctors, we lack meaningful, quantifiable, and comparable criteria. Partial or weighted admissions lotteries can offer us an escape. They have the potential to reduce mental and financial burdens on both applicants and medical schools, avoiding an overemphasis on marginal differences between applicants. Lotteries are also a simple way to address persistent admissions disparities by being truly non-discriminatory. At the very least, lotteries represent a useful benchmark against which we can rigorously compare current and future selection methods.

Postgraduate ethics training programs: a systematic scoping review

BACKGROUND

Molding competent clinicians capable of applying ethics principles in their practice is a challenging task, compounded by wide variations in the teaching and assessment of ethics in the postgraduate setting. Despite these differences, ethics training programs should recognise that the transition from medical students to healthcare professionals entails a longitudinal process where ethics knowledge, skills and identity continue to build and deepen over time with clinical exposure. A systematic scoping review is proposed to analyse current postgraduate medical ethics training and assessment programs in peer-reviewed literature to guide the development of a local physician training curriculum.

METHODS

With a constructivist perspective and relativist lens, this systematic scoping review on postgraduate medical ethics training and assessment will adopt the Systematic Evidence Based Approach (SEBA) to create a transparent and reproducible review.

RESULTS

The first search involving the teaching of ethics yielded 7669 abstracts with 573 full text articles evaluated and 66 articles included. The second search involving the assessment of ethics identified 9919 abstracts with 333 full text articles reviewed and 29 articles included. The themes identified from the two searches were the goals and objectives, content, pedagogy, enabling and limiting factors of teaching ethics and assessment modalities used. Despite inherent disparities in ethics training programs, they provide a platform for learners to apply knowledge, translating it to skill and eventually becoming part of the identity of the learner. Illustrating the longitudinal nature of ethics training, the spiral curriculum seamlessly integrates and fortifies prevailing ethical knowledge acquired in medical school with the layering of new specialty, clinical and research specific content in professional practice. Various assessment methods are employed with special mention of portfolios as a longitudinal assessment modality that showcase the impact of ethics training on the development of professional identity formation (PIF).

CONCLUSIONS

Our systematic scoping review has elicited key learning points in the teaching and assessment of ethics in the postgraduate setting. However, more research needs to be done on establishing Entrustable Professional Activities (EPA)s in ethics, with further exploration of the use of portfolios and key factors influencing its design, implementation and assessment of PIF and micro-credentialling in ethics practice.

Multiple Mini-Interview Utilization in United States Physician Assistant Program Admission Processes

PURPOSE

Health care program admission processes utilize multi-faceted approaches to evaluate cognitive and noncognitive attributes of applicants. The multiple mini-interview (MMI) was developed in response to the need for a reliable and validated tool to assess noncognitive factors and has been increasingly incorporated into the admissions process by physician assistant (PA) programs. The study's purpose was to explore the current implementation and utilization of the MMI within PA programs.

METHODS

The study used a mixed-methods exploratory approach including a telephone survey and semi-structured interview of 11 PA programs using the MMI in their admissions process. Quantitative data collected included demographic information, MMI implementation characteristics, station structure, scoring, feasibility, satisfaction with MMI utilization, and MMI evaluation methods.

RESULTS

During the 2015-2016 admissions cycle, the participating programs used from 5 to 10 stations, averaging 7 minutes per station, requiring 8 faculty, 2 staff, and 7 students per interview session. Despite variation in program size, number of applicants, and years of MMI utilization, all participating programs reported that they were satisfied with the format and would continue to utilize the MMI in the admissions process.

CONCLUSIONS

While there is substantial literature describing the use of the MMI within health care programs globally, this study represents the first characterization of its use within PA programs on a national level. Although there was variation among PA program implementation of the MMI, our results are comparable to studies within other health care professions. Additional studies are necessary to further describe the MMI and its correlation with PA program educational outcomes and the impact on diversity.

Graduate rural work outcomes of the first 8 years of a medical school: What can we learn about student selection and clinical school training pathways?

OBJECTIVE

To determine the student characteristics and clinical school training pathways that are associated with postgraduate rural practice.

DESIGN

A cross-sectional observational study.

SETTING

Deakin University's School of Medicine, encompassing 3 rural and 2 metropolitan clinical schools, including a rural longitudinal integrated clerkship (LIC) year.

PARTICIPANTS

A total of 948 School of Medicine medical graduates (2011-2018) who had an Australian Health Practitioners Regulation Agency registered work location in Australia (September 2019).

MAIN OUTCOME MEASURE(S)

Graduates' work locations were linked with admission and clinical school training data and analysed for significant associations.

RESULTS

Graduates completing a rural LIC year and a regional rural clinical school year were 7 times more likely to be working rurally than metropolitan trainees. A small number of rural background (RB) students completed a rural LIC and rural clinical school year, 66.7% of whom were practising rurally. Further associations with rural practice were attending a rural clinical school for 2 years or a rural LIC for 1 year, student having a RB, rurally bonded course placement and being an international student.

CONCLUSION

The significant influence of a rural LIC year on graduate rural work location is noteworthy, especially when undertaken by students from a RB in combination with a second regional rural clinical school year. These findings support the strategic alignment of rural student selection with rural clinical school training pathways to achieve the greatest impact on rural graduate workforce outcomes.

An exploration of the predictive validity of selection criteria on progress outcomes for pre-registration nursing programmes-A systematic review

AIM

To identify the selection methods currently being used for pre-registration nursing programmes and to assess the predictive power that these methods have on students' success.

BACKGROUND

Research into selection methods in nursing education is beginning to emerge, yet it is unclear which methods are most predictive of students' success.

DESIGN

A systematic review of the literature.

METHODS

A systematic search of ten electronic databases: CINAHL, MEDLINE Ovid, EMBASE, PROQUEST Health and Medical, PROQUEST Education, COCHRANE Library, Web of Science, ASSIA, SCOPUS and PROSPERO was conducted. The results were expanded by the handsearching of journals, reference lists and grey literature. The PRISMA statement guided the review. Studies published in English between January 2008-March 2020 were eligible for inclusion, and quality assessment was undertaken using the CASP Checklist for Cohort Studies.

RESULTS

Twenty-five studies met the criteria for inclusion. A range of selection methods was identified including prior academic achievement, admissions tests, interviews, emotional intelligence tests, personal statements and previous healthcare experience. Prior academic achievement and admissions tests appear to be the selection methods most predictive of student success. The evidence surrounding other selection methods such as interviews and personal statements is less conclusive.

CONCLUSION

Selecting individuals with the appropriate knowledge, interpersonal skills and personal qualities needed to complete an undergraduate nursing programme is an important part of the role of nurse educators. This review shows that a wide variety of selection methods are used across different institutions, some of which are more effective than others in predicting student outcomes.

RELEVANCE TO CLINICAL PRACTICE

Further research is required to justify the continued use of some commonly used selection methods for undergraduate nursing programmes. Selection models that combine various types of selection criteria with predictive power appear to increase the probability of selecting students that will have successful outcomes.

Predictors of getting a residency interview: Differences by medical specialty

OBJECTIVES

Gaining medical residency interviews has become more competitive and costly for medical students. Although limited evidence from residency programme directors indicates predictors for successfully matching into a programme, past research has not sufficiently explored the application components necessary to receive an interview offer. The present study will identify which application components are most helpful in obtaining interview offers for different medical specialties.

METHODS

Data were sourced from the Texas Seeking Transparency in Application to Residency (STAR), a survey of recently matched fourth-year American medical students who self-reported information on their residency application components and interview offers. Multi-level logistic regression analyses were employed to predict the odds of interview offer according to applicants' academic, research and extracurricular characteristics. Sub-analyses were conducted for each medical specialty.

RESULTS

Nearly 10 000 students reported information on over 419 010 applications submitted, which resulted in 164 696 interview offers. Across the sample, applicants had greater odds of receiving an interview offer if they had a geographic connection to the programme (odds ratio [OR] = 4.10, 95% confidence interval [CI] 4.00-4.20), had completed an away rotation at the programme (OR = 16.00, 95% CI 14.92-17.15), were Alpha Omega Alpha Honor Medical Society members (OR = 1.49, 95% CI 1.36-1.64), or had been inducted into the Gold Humanism Honor Society (OR = 1.50, 95% CI 1.39-1.62). Applicants had reduced odds of getting an interview if they had been required to remediate a course in medical school (OR = 0.73, 95% CI 0.64-0.83) or had failed the US Medical Licensing Examination Step 1 or Step 2 examination on their first attempt (OR = 0.40, 95% CI 0.33-0.47). Predictors of obtaining an interview varied by specialty.

CONCLUSIONS

The present findings can assist senior medical students as they prepare residency applications and identify programmes to which they will apply. Knowledge of the significant factors can help applicants more efficiently use resources to maximise their number of interview offers. Completing away rotations and selecting programmes with which applicants have geographic connections may increase their odds of receiving interview offers.

Social Accountability Frameworks and Their Implications for Medical Education and Program Evaluation: A Narrative Review

PURPOSE

Medical schools face growing pressures to produce stronger evidence of their social accountability, but measuring social accountability remains a global challenge. This narrative review aimed to identify and document common themes and indicators across large-scale social accountability frameworks to facilitate development of initial operational constructs to evaluate social accountability in medical education.

METHOD

The authors searched 5 electronic databases and platforms and the World Wide Web to identify social accountability frameworks applicable to medical education, with a focus on medical schools. English-language, peer-reviewed documents published between 1990 and March 2019 were eligible for inclusion. Primary source social accountability frameworks that represented foundational values, principles, and parameters and were cited in subsequent papers to conceptualize social accountability were included in the analysis. Thematic synthesis was used to describe common elements across included frameworks. Descriptive themes were characterized using the context-input-process-product (CIPP) evaluation model as an organizational framework.

RESULTS

From the initial sample of 33 documents, 4 key social accountability frameworks were selected and analyzed. Six themes (with subthemes) emerged across frameworks, including shared values (core social values of relevance, quality, effectiveness, and equity; professionalism; academic freedom and clinical autonomy) and 5 indicators related to the CIPP model: context (mission statements, community partnerships, active contributions to health care policy); inputs (diversity/equity in recruitment/selection, community population health profiles); processes (curricular activities, community-based clinical training opportunities/learning exposures); products (physician resource planning, quality assurance, program evaluation and accreditation); and impacts (overall improvement in community health outcomes, reduction/prevention of health risks, morbidity/mortality of community diseases).

CONCLUSIONS

As more emphasis is placed on social accountability of medical schools, it is imperative to shift focus from educational inputs and processes to educational products and impacts. A way to begin to establish links between inputs, products, and impacts is by using the CIPP evaluation model.

Situational judgement test validity for selection: A systematic review and meta-analysis

CONTEXT

Situational judgement tests (SJTs) are widely used to evaluate 'non-academic' abilities in medical applicants. However, there is a lack of understanding of how their predictive validity may vary across contexts. We conducted a systematic review and meta-analysis to synthesise existing evidence relating to the validity of such tools for predicting outcomes relevant to interpersonal workplace performance.

METHODS

Searches were conducted in relevant databases to June 2019. Study quality and risk of bias were assessed using the Quality In Prognosis Studies (QUIPS) tool. Results were pooled using random effects meta-analysis and meta-regressions.

RESULTS

Initially, 470 articles were identified, 218 title or abstracts were reviewed, and 44 full text articles were assessed with 30 studies meeting the final inclusion criteria and were judged, overall, to be at moderate risk of bias. Of these, 26 reported correlation coefficients relating to validity, with a pooled estimate of 0.32 (95% confidence interval 0.26 to 0.39, P < .0001). Considerable heterogeneity was observed (I²  = 96.5%) with the largest validity coefficients tending to be observed for postgraduate, rather than undergraduate, selection studies (β = 0.23, 0.11 to 0.36, P < .001). The correction of validity coefficients for attenuation was also independently associated with larger effects (β = 0.13, 0.03 to 0.23, P = .01). No significant associations with test medium (video vs text format), cross-sectional study design, or period of assessment (one-off vs longer-term) were observed. Where reported, the scores generally demonstrated incremental predictive validity, over and above tests of knowledge and cognitive ability.

CONCLUSIONS

The use of SJTs in medical selection is supported by the evidence. The observed trend relating to training stage requires investigation. Further research should focus on developing robust criterion-relevant outcome measures that, ideally, capture interpersonal aspects of typical workplace performance. This will facilitate additional work identifying the optimal place of SJTs within particular selection contexts and further enhancing their effectiveness.

UKCAT and medical student selection in the UK - what has changed since 2006?

BACKGROUND

The United Kingdom Clinical Aptitude Test (UKCAT) is an aptitude test used since 2006 within selection processes of a consortium of UK medical and dental schools. Since 2006, student numbers have increased in medical training and schools now have an increased focus on widening access. A growing evidence base has emerged around medical student selection (Patterson et al., Med Educ 50:36-60, 2016) leading to changes in practice. However, whilst some papers describe local selection processes, there has been no overview of trends in selection processes over time across Universities. This study reports on how the use of the UKCAT in medical student selection has changed and comments on other changes in selection processes.

METHODS

Telephone interviews were conducted annually with UKCAT Consortium medical schools. Use of the UKCAT was categorised and data analysed to identify trends over time.

RESULTS

The number of schools using the UKCAT to select applicants for interview has risen, with cognitive test results contributing significantly to outcomes at this stage at many universities. Where schools use different weighted criteria (Factor Method), the UKCAT has largely replaced the use of personal statements. Use of the test at offer stage has also increased; the most significant use being to discriminate between applicants at a decision borderline. A growing number of schools are using the UKCAT Situational Judgement Test (SJT) in selection. In 2018, all but seven (out of 26) schools made some adjustment to selection processes for widening access applicants. Multiple Mini Interviews (MMIs) are now used by the majority of schools. Whilst medical student numbers have increased over this time, the ratio of applicants to places has fallen. The probability of applicants being invited to interview or receiving an offer has increased.

CONCLUSIONS

More medical schools are using the UKCAT in undergraduate selection processes in an increasing number of ways and with increasing weight compared with 2007. It has replaced the use of personal statements in all but a few Consortium medical schools. An increased focus on academic attainment and the UKCAT across medical schools may be leading to the need for schools to interview and make offers to more applicants.

Admission policies and methods at crossroads: a review of medical school admission policies and methods in seven Asian countries

Selecting the right applicants is an important part of medical student admission. While one universally accepted selection criterion is academic capacity, there are other criteria such as communication skills and local criteria (e.g., socio-cultural values) that are no less important. This article reviews the policies and methods of selection to medical schools in seven countries with varying socio-economic conditions and healthcare systems. Senior academics involved in medical education in Indonesia, Japan, Malaysia, the Philippines, Singapore, Sri Lanka, and Taiwan completed a pre-agreed pro-forma per each country to describe the country's admission policies and methods. The details were then compared and contrasted. This review identifies tension between many of the policies and methods used in medical school admissions, such as between the need to assess non-cognitive abilities and widen access, and between the need for more medical professionals and the requirement to set high entry standards. Finding the right balance requires careful consideration of all variables, including the country's human resource needs; socio-economic status; graduates' expected competencies; and the school's vision, mission, and availability of resources.

Outcomes-Based Selection Into Medical School: Predicting Excellence in Multiple Competencies During the Clinical Years

PURPOSE

Medical school selection committees aim to identify the best possible students and, ultimately, the best future doctors from a large, well-qualified, generally homogeneous pool of applicants. Constructive alignment of medical school selection, curricula, and assessment with the ultimate outcomes (e.g., CanMEDS roles) has been proposed as means to attain this goal. Whether this approach is effective has not yet been established. The authors addressed this gap by assessing the relationship between performance in an outcomes-based selection procedure and performance during the clinical years of medical school.

METHOD

Two groups of students were compared: (1) those admitted into Maastricht University Medical School via an outcomes-based selection procedure and (2) those rejected through this procedure who were admitted into the program through a national, grade-point-average-based lottery. The authors compared performance scores of students from the 2 groups on all 7 CanMEDS roles, using assessment data gathered during clinical rotations. The authors examined data from 3 cohorts (2011-2013).

RESULTS

Students admitted through the local, outcomes-based selection procedure significantly outperformed the initially rejected but lottery-admitted students in all years, and the differences between groups increased over time. The selected students performed significantly better in the CanMEDS roles of Communicator, Collaborator, and Professional in the first year of clinical rotations; in these 3 roles-plus Organizer-in the second year; and in 2 additional roles (Advocate and Scholar-all except Medical Expert) at the end of their clinical training.

CONCLUSIONS

A constructively aligned selection procedure has increasing predictive value across the clinical years of medical school compared with a GPA-based lottery procedure. The data reported here suggest that constructive alignment of selection, curricula, and assessment to ultimate outcomes is effective in creating a selection procedure predictive of clinical performance.

Proficiency testing for identifying underperforming students before postgraduate education: a longitudinal study

BACKGROUND

Efficient selection of medical students in GP training plays an important role in improving healthcare quality. The aim of this study was to collect quantitative and qualitative validity evidence of a multicomponent proficiency-test for identifying underperforming students in cognitive and non-cognitive competencies, prior to entering postgraduate GP Training. From 2016 to 2018, 894 medical GP students in four Flemish universities in Belgium registered to take a multicomponent proficiency-test before admission to postgraduate GP Training. Data on students were obtained from the proficiency-test as a test-score and from traineeship mentors' narrative reports.

RESULTS

In total, 849 students took the multicomponent proficiency-test during 2016-2018. Test scores were normally distributed. Five different descriptive labels were extracted from mentors' narrative reports based on thematic analysis, considering both cognitive and non-cognitive competences. Chi-square tests and odds ratio showed a significant association between students scoring low on the proficiency-test and having gaps in cognitive and non-cognitive competencies during GP traineeship.

CONCLUSION

A multicomponent proficiency-test could detect underperforming students prior to postgraduate GP Training. Students that ranked in the lowest score quartile had a higher likelihood of being labelled as underperforming than students in the highest score quartile. Therefore, a low score in the multicomponent proficiency-test could indicate the need for closer guidance and early remediating actions focusing on both cognitive and non-cognitive competencies.

The economic costs of selecting medical students: An Australian case study

CONTEXT

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained.

METHODS

A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated.

RESULTS

A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%).

CONCLUSIONS

The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape

Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.

Nurturing professionalism in medical schools. A systematic scoping review of training curricula between 1990-2019

Introduction: Professionalism is an evolving, socioculturally informed multidimensional construct that influences doctor-patient relationships, patient satisfaction and care outcomes. However, despite its clinical significance there is little consistency in how professionalism is nurtured amongst medical students. To address this gap a systemic scoping review of nurturing professionalism in medical schools, is proposed.Methods: Levac's framework and the PRISMA-P 2015 checklist underpinned a 6-stage systematic review protocol. Concurrent use of Braun and Clarke's approach to thematic analysis and directed content analysis was used to identify the key elements in nurturing professionalism.Results: 13921 abstracts were identified from six databases, 854 full-text articles reviewed, and 162 full-text included articles were included. The 4 themes identified through thematic analysis are consistent with findings of the directed content analysis. These were the definition of professionalism, the approaches, content, barriers and enablers to teaching professionalism.Conclusion: Informed by a viable definition of professionalism and clear milestones nurturing professionalism nurturing professionalism begins with culturally appropriate training in clinical competence, humanistic qualities and reflective capacity. This process requires effective evaluations of professional identity formation, and the impact of the learning environment underlining the need for longitudinal assessments of the training process.