How effective are selection methods in medical education? A systematic review

LEGO®-based communication assessment in virtual general surgery residency interviews

Purpose

Effective communication skills are a critical quality and skill that is highly sought after for surgeons which largely impacts patient outcomes. Residency programs design their interview processes to select the best candidates. LEGO®-based activities have been frequently used to enhance communication skills and team building. This study investigates the effectiveness and reliability of a novel LEGO®-based communication assessment in interviews for surgical residencies and the feasibility of implementing it in a virtual setting.

Methods

This study conducted a retrospective analysis of a LEGO®-based communication assessment at the program's 2020/2021 residency interviews. Each applicant was assessed on a different model. The total scores were analyzed for consistency among raters and correlated to faculty interviews. Furthermore, the impact of the assessment structure, scoring criteria, and range of models' difficulties on the total scores were explored.

Results

A total of 54 categorical and 55 preliminary applicants interviewed on 2 days. The assessment on different models and had no impact on applicants' total scores for either categorical and preliminary groups (p = 0.791 and 0.709, respectively). The communication components of the assessment showed high consistency between the raters. The two applicant groups displayed a statistically significant difference (p = 0.004) in the communication evaluation and model accuracy components. Total scores did not correlate with the faculty interviews of standardized questions in either group.

Conclusion

This novel LEGO®-based communication assessment showed high reliability and promising results as a tool to assess communication and problem solving for residency interviews that can be readily implemented in a virtual setting.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00021-4.

Does Personality Matter? Perceptions and Experiences of Introverts and Extraverts as General Surgeons

PHENOMENON

Medical educators increasingly recognize both the challenges introverts, compared to extraverts, may face in medical training and the unique strengths they bring to practice. However, few researchers have examined in-depth how introverts and extraverts truly experience training and practice, particularly in specialties like surgery that tend to value qualities (e.g., dominance and assertiveness) typically associated with extraverts. This study aimed to explore the perceptions and experiences of individuals with both personalities within the field of general surgery.

APPROACH

Using a constructivist grounded theory approach, six general surgeons and 10 general surgery residents who identified as introverted, extraverted, or ambiverted were recruited from two Canadian tertiary care hospitals to participate in semi-structured interviews. Data collection and analysis occurred iteratively, and data were analyzed using open, selective, and thematic coding. Constant comparison allowed us to make sense of the similar and dissimilar views that emerged from each interview.

FINDINGS

Irrespective of their personalities, participants voiced two general patterns of responses. Some participants believed that "personality doesn't matter": that both introverted and extraverted practice styles were equally viable and neither introverts nor extraverts would find surgical training more challenging than the other (culture of equality). However, others believed that "personality matters," emphasizing that surgeons should be dominant and aggressive leaders. Only those who believed "personality matters" felt that introverts sometimes needed to act more extraverted in order to succeed in surgical training (culture of hierarchy). Similar numbers of introverts and extraverts adhered to each viewpoint.

INSIGHTS

Our qualitative approach allowed us to draw meaning from the complex subjective experiences of our research participants. Our findings suggest that two competing cultures (equality and hierarchy) co-exist within the field of surgery and that trainees, depending on which culture they adhere most to, will or will not "adapt" their personalities to the workplace. These findings deepen our understanding of the nuances of surgical culture and have important implications for how we select candidates based on personality.

Academic and non-academic predictors of academic performance in medical school: an exploratory cohort study

BACKGROUND

Medical schools should also evaluate applicants' non-academic characteristics in the search for successful students and future physicians, but ideal non-academic criteria have not yet been found. We followed two successive generations of medical students at the University of Split School of Medicine (USSM) to assess both academic and non-academic constructs as predictors of academic performance, defined as medical school grade point average (GPA). We also interviewed some of the participants to gain additional insight for future studies.

METHODS

We measured study GPA in first and last year, as well as attitudes towards science, motivation, emotional intelligence, self-esteem, and perceived personal incompetence in first year. We also obtained their scores on existing medical school enrollment criteria, the State Graduation Exam (SGE) and high-school GPA. Regression models were constructed for predictors of GPA in the last year of medical school. Four structured pilot interviews were conducted to explore participants' perceptions of necessary traits for medical school and later practice.

RESULTS

Regression analysis showed that only SGE predicted final academic performance in medical school (β=0.35, 95% confidence interval (CI)=0.06-0.64), while none of the non-academic constructs we assessed predicted this outcome of education. The two generations did not significantly differ in any variable except that intrinsic motivation was higher in the generation that enrolled in 2011 (OR=1.47, 95%CI=1.12-1.93, P=0.005).

DISCUSSION

None of the non-academic constructs predicted academic performance in medical school. Their use as selection criteria may not be warranted as they could impact the academic quality of enrolling medical students.

Selection by design: Using job analysis to guide the selection of postgraduate pharmacy residents

DISCLAIMER

In an effort to expedite the publication of articles related to the COVID-19 pandemic, AJHP is posting these manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

PURPOSE

To help ensure that we were accurately and consistently evaluating applicants to our postgraduate year 1 (PGY1) pharmacy residency program, we performed a job analysis to inform a redesign of our selection process.

SUMMARY

A diverse panel of subject matter experts from our program was convened to develop a task inventory; a list of knowledge, skills, abilities, and other characteristics necessary for success in our program; and behavioral snapshots representing especially strong or weak resident performance (ie, critical incidents). After achieving a priori thresholds of consensus, these items were used to augment our application screening instrument (eg, development of anchored rating scales), build an online supplemental application consisting of a personality test and situational judgment test, develop a work sample consisting of a patient case presentation, and enhance the structure of our interviews (eg, by asking a consistent pattern of questions for all candidates). Preceptors reported that the redesigned process was more organized, easier to complete, and facilitated greater rating consistency.

CONCLUSION

Job analysis represents an approach to designing selection processes that are more valid, reliable, transparent, and fair. Based on our experiences, recommendations for those who are considering changes to their selection process are provided.

The development and psychometric properties of a measure to assess the written submission of an admissions application

Introduction

Health care programs evaluate prospective applicants using cognitive and non-cognitive criteria. The aim of this research was to develop and evaluate the psychometric properties of a measure to evaluate the non-cognitive criteria of admissions applications.

Method

A Masters of Occupational Therapy Written Submission Measure (MOTWSM) was developed and evaluated over 3 phases, using applicants’ written statements, resumes, and reference letters. Participants included 50 students who completed an occupational therapy program for determination of internal consistency and test-retest reliability. Additionally, 195 written submissions selected from the applicants who were admitted, waitlisted, and not admitted to the program were evaluated to determine inter-rater reliability using a two-way ANOVA. Analysis of 195 submissions using a one-way ANOVA determined the measure’s discriminative validity.

Findings

Results indicated test-retest reliability of 0.95 and internal consistency reliability of 0.76. Inter-rater reliability reported a Cronbach’s alpha coefficient of 0.86 using a horizontal scoring method. Good discriminative validity was established.

Conclusion

The MOTWSM is a reliable and valid measure that can be used to evaluate the non-cognitive criteria of admissions applications in health profession programs. Use of this measure can facilitate selection of the highest caliber of students.

Assessing distinguishable social skills in medical admission: does construct-driven development solve validity issues of situational judgment tests?

BACKGROUND

Social skills are important for future physicians and are therefore increasingly considered in selection processes. One economic assessment method from which different social skills can be inferred are Situational Judgment Tests (SJTs) in which applicants are asked to rate behavioral responses in context-relevant situations. However, traditional SJTs have so far failed to distinctively measure specified constructs. To address this shortcoming in the medical admission context, we applied a construct-driven approach of SJT development in which test development was deductively guided by agency and communion as target constructs.

METHOD

The final version of the construct-driven SJT includes 15 items per construct with three behavioral responses. Medical school applicants (N = 1527) completed the construct-driven SJT, a traditional SJT, and an aptitude test under high-stakes condition as part of their application. A subsample (N = 575) participated in a subsequent voluntary online study with self-report measures of personality and past behavior.

RESULTS

The proposed two-factor structure and internal consistency of the construct-driven SJT was confirmed. Communal SJT scores were positively associated with self-reported communal personality and communal behavior, yet effects were smaller than expected. Findings for agentic SJT scores were mixed with positive small associations to self-reported agentic personality scores and agentic behavior but unexpected negative relations to communal self-reported measures.

CONCLUSIONS

Results suggest that construct-driven SJTs might overcome validity limitations of traditional SJTs, although their implementation is challenging. Despite first indicators of validity, future research needs to address practical points of application in high-stakes settings, inclusion of other constructs, and especially prediction of actual behavior before the application of construct-driven SJTs for selection purposes in medical admission can be recommended.

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues

BACKGROUND

Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile.

METHODS

Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508).

RESULTS

The SJT demonstrated good internal consistency (Cronbach's alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity.

CONCLUSIONS

This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

Assessing the predictive validity of the UCAT-A systematic review and narrative synthesis

BACKGROUND

The University Clinical Aptitude Test (UCAT) is an admissions assessment used by a consortium of universities across the UK, Australia, and New Zealand, to aid the selection of applicants to medical and dental degree programmes. The UCAT aims to measure the mental aptitude and professional behaviours required to become successful doctors and dentists. We conducted a systematic review to establish the predictive value of the UCAT across measures of performance at undergraduate and post-graduate levels.

METHODS

A literature search was conducted in April 2020 using eight electronic databases: MEDLINE, APA PsycInfo, SCOPUS, Web of Science, EThOS, OpenGrey, PROSPERO, and the UCAT website. Data were extracted from selected studies and tabulated as results matrices. A narrative synthesis was performed.

RESULTS

Twenty-four studies satisfied our inclusion criteria, 23 of which were deemed to be of good quality (using the Newcastle-Ottawa Scale). For over 70% of univariate data points, the UCAT exerted no statistically significant predictive validity; for the remainder, predictive power was weak. The cognitive total and verbal reasoning subtests had the largest evidence base as weakly positive predictors of academic performance. The SJT subtest was a weak predictor of professional behaviour during medical school. Studies specific to dental schools demonstrated variable findings across the five studies. Only 1 study looked at post-graduate outcome measures and demonstrated that the UCAT was not a predictor of health- or conduct-related fitness to practice declarations at GMC registration.

CONCLUSIONS

These data provide some support for the use of cognitive total and verbal reasoning subtests as part of medical school selection. Further research is needed to investigate outcomes beyond professional registration and for dental students.

Understanding the Link Between Burnout and Sub-Optimal Care: Why Should Healthcare Education Be Interested in Employee Silence?

Evidence on the association of burnout with objective indicators of performance is scarce in healthcare. In parallel, healthcare professionals ameliorate the short-term impact of burnout by prioritizing some tasks over others. The phenomenon of employee silence can help us understand the evolution of how culture is molded toward the prioritization of some tasks over others, and how this contributes to burnout. Silence in healthcare has been associated with concealing errors, reduced patient safety, and covering up errors made by others. Conversely, there is evidence that in organizations where employees are encouraged to speak up about concerns, and where concerns are responded to appropriately, better patient outcomes such as improved patient safety and patient experience occur. Interventions to promote "speaking-up" in healthcare have not been successful and are rooted in a professional culture that does not promote speaking out. In this paper, we review the evidence that exists within healthcare to argue why healthcare education should be interested in employee silence, and how silence is a key factor in understanding how burnout develops and impacts quality of care. The following key questions have been addressed; how employee silence evolves during medical education, how is silence maintained after graduation, and how can leadership style contribute to silence in healthcare. The impact of withholding information on healthcare professional burnout, patient safety and quality of care is significant. The paper concludes with a suggested future research agenda and additional recommendations.

The current dental school applicant: an overview of the admission process for UK dental schools and the sociodemographic status of applicants

Aims To determine the current processes used to assess dental school admissions in the UK as well as compare the applicants' demography.

Methods All 16 dental schools in the UK were invited to complete a questionnaire analysing the admissions protocols between 2018-2019 and 2019-2020. These data were combined with the admission process information available online. Retrospective data from the University Clinical Aptitude Test including the sociodemographic status of dental applicants were collected. For the six dental schools that did not reply, data presented were collected from information available to the public.

Results The majority of applicants were women (63.2% in 2019-2020), white (27.9%) and are sixth form attendees or attend a further education college (40.6%). Of those who apply to study dentistry, 15.5% are graduates who hold a first degree. For each undergraduate dental place available, there were 12.6 applicants and 9.4 applicants to each post-qualified dental undergraduate place.

Conclusion Further advancements are required to widen participation and broaden the sociodemographic status of dental applicants. This area would benefit from a long-term prospective study about recruitment methods and its correlation with performance at dental school. COVID-19 is impacting the application process, the full extent of which is yet to be determined.

Provides an overview of the current admission processes for dental schools within the UK.

Encourages critical analysis of the admission processes used for undergraduate and postgraduate dental students.

Explores how the admission processes may change as a result of the COVID-19 pandemic and gives an indication as to the sociodemographics of the future dental career force.

Weighting admission scores to balance predictiveness-diversity: The Pareto-optimization approach

CONTEXT

Although many medical schools seek to improve diversity, they grapple with the challenge of how to weight the scores of different admission methods to achieve a balance between obtaining high predictiveness and ensuring diversity in the selected student pool. Yet, in large-scale employment settings, substantial progress has been made on this front: Pareto-optimization has been introduced as an elegant statistical tool to assist decision makers in determining the weights assigned to selection methods in advance (before the selection has taken place) so that a selection system is designed to achieve an optimal balance as reflected by the trade-off that one outcome (e.g., predictiveness) cannot be improved without harm to the other outcome (e.g., diversity).

AIMS

This paper reviews the theory and research evidence about Pareto-optimization and explains how Pareto-optimization permits medical schools to better balance predictiveness and diversity in medical admission systems.

METHODS

After reviewing common weighting schemes (unit, regression-based and ad hoc weighting) and their drawbacks, we introduce the theory and logic of Pareto-optimization for better balancing predictiveness and diversity. To this end, we also offer an illustrative example. Next, we review the mathematical basis and available research evidence regarding Pareto-optimization. Finally, we discuss potential criticisms (i.e., complexity and legal concerns).

CONCLUSIONS

Compared to traditional unit weighting, regression-based weighting and ad hoc weighting, Pareto-optimization leads to substantial increases in diversity intake (up to three times more), while keeping the predictiveness of the selection methods at the same level. Moreover, the Pareto-optimization is robust to sampling variability and variability of the input selection parameters.

Does General and Specific Traits of Personality Predict Students' Academic Performance?

Methods

The study includes all fifth-year dental students registered at the College of Dentistry, Ajman University, in 2019/2020. One hundred and seventy students were invited to complete personality and performance measures using the Big Five Inventory (BFI) scale; the weighted grade point average (GPA) was used to assess students' academic performance.

Results

Of the 170 participants, 60% were female and 40% were male. Participants ranged in age from twenty-four to twenty-seven years, with an average age of twenty-four years. There was a relationship between personality scores obtained for the students and their subsequent academic performance. The broad conscientiousness, competence, achievement, and dutifulness predicted academic and clinical success. The prediction accuracy of conscientiousness was improved by the inclusion of dutifulness, self-discipline, and deliberation.

Conclusion

This study confirms that the students' personality profile is a substantial predictor of academic performance and likely to help select future intakes of students, although a prospective study would be required for a definite answer to this question.

Does performance at the intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examination vary according to UK medical school and course type? A retrospective cohort study

OBJECTIVES

The knowledge, skills and behaviours required of new UK medical graduates are the same but how these are achieved differs given medical schools vary in their mission, curricula and pedagogy. Medical school differences seem to influence performance on postgraduate assessments. To date, the relationship between medical schools, course types and performance at the Membership of the Royal Colleges of Surgeons examination (MRCS) has not been investigated. Understanding this relationship is vital to achieving alignment across undergraduate and postgraduate training, learning and assessment values.

DESIGN AND PARTICIPANTS

A retrospective longitudinal cohort study of UK medical graduates who attempted MRCS Part A (n=9730) and MRCS Part B (n=4645) between 2007 and 2017, using individual-level linked sociodemographic and prior academic attainment data from the UK Medical Education Database.

METHODS

We studied MRCS performance across all UK medical schools and examined relationships between potential predictors and MRCS performance using χ2 analysis. Multivariate logistic regression models identified independent predictors of MRCS success at first attempt.

RESULTS

MRCS pass rates differed significantly between individual medical schools (p<0.001) but not after adjusting for prior A-Level performance. Candidates from courses other than those described as problem-based learning (PBL) were 53% more likely to pass MRCS Part A (OR 1.53 (95% CI 1.25 to 1.87) and 54% more likely to pass Part B (OR 1.54 (1.05 to 2.25)) at first attempt after adjusting for prior academic performance. Attending a Standard-Entry 5-year medicine programme, having no prior degree and attending a Russell Group university were independent predictors of MRCS success in regression models (p<0.05).

CONCLUSIONS

There are significant differences in MRCS performance between medical schools. However, this variation is largely due to individual factors such as academic ability, rather than medical school factors. This study also highlights group level attainment differences that warrant further investigation to ensure equity within medical training.

The do's, don'ts and don't knows of redressing differential attainment related to race/ethnicity in medical schools

INTRODUCTION

Systematic and structural inequities in power and privilege create differential attainment whereby differences in average levels of performance are observed between students from different socio-demographic groups. This paper reviews the international evidence on differential attainment related to ethnicity/race in medical school, drawing together the key messages from research to date to provide guidance for educators to operationalize and enact change and identify areas for further research.

METHODS

Authors first identified areas of conceptual importance within differential attainment (learning, assessment, and systems/institutional factors) which were then the focus of a targeted review of the literature on differential attainment related to ethnicity/race in medical education and, where available and relevant, literature from higher education more generally. Each author then conducted a review of the literature and proposed guidelines based on their experience and research literature. The guidelines were iteratively reviewed and refined between all authors until we reached consensus on the Do's, Don'ts and Don't Knows.

RESULTS

We present 13 guidelines with a summary of the research evidence for each. Guidelines address assessment practices (assessment design, assessment formats, use of assessments and post-hoc analysis) and educational systems and cultures (student experience, learning environment, faculty diversity and diversity practices).

CONCLUSIONS

Differential attainment related to ethnicity/race is a complex, systemic problem reflective of unequal norms and practices within broader society and evident throughout assessment practices, the learning environment and student experiences at medical school. Currently, the strongest empirical evidence is around assessment processes themselves. There is emerging evidence of minoritized students facing discrimination and having different learning experiences in medical school, but more studies are needed. There is a pressing need for research on how to effectively redress systemic issues within our medical schools, particularly related to inequity in teaching and learning.

Development and testing of the situational judgement test to measure safety performance of healthcare professionals: An explorative cross-sectional study

AIM

To measure safety performance, situational judgement test, which is a method composed of job-related situations, can be used. This study aimed to develop and test its psychometric properties by measuring the safety performance of healthcare professionals in German hospitals.

DESIGN

An explorative cross-sectional study.

METHODS

A team of researchers, nurses and physicians developed seven items, which focus on different safety areas. Descriptive statistics were calculated for each item. Cronbach's alpha was calculated as an indication of internal consistency. Spearman's correlation between the items was evaluated as analysis of construct validity. A cross-sectional survey with healthcare professionals in three German hospitals was conducted to test the developed instrument.

RESULTS

A total of 168 healthcare professionals participated (response rate: 39.1%). 70.2% were women, and 38.7%, 33.9%, 15.5% and 11.3% were registered nurses, nurses in training, physicians and other healthcare professionals respectively. The situational judgement test demonstrated an acceptable psychometric performance.

Implementation and evaluation of virtual multiple mini interviews as a selection tool for entry into paediatric postgraduate training: A Queensland experience

INTRODUCTION

The Queensland Basic Paediatric Training Network (QBPTN) is the centralised pathway for entry into paediatric training in Queensland, Australia. In response to COVID-19 travel and social distancing restrictions imposed in 2020, QBPTN successfully adopted a Virtual Multiple Mini Interviews (vMMIs) model for the selection of candidates for entry into paediatric training. The authors describe the planning, implementation, challenges, and evaluation of candidates' and interviewers' experiences of vMMIs, including the differences between candidates from two geographical areas.

METHODS

The contents of six vMMI stations were similar to face-to-face MMI. Implementation required the identification of ZOOM^TM as a preferred online platform, securing venues, communication, development of contingency plans and central coordination by the network. Candidates' experiences with vMMI were explored through thematic analysis of the qualitative data from focus groups and free text responses, and descriptive analysis of SurveyMonkey© questionnaire responses. Experiences between 'metropolitan' and 'regional and interstate' candidates were compared.

RESULTS

5-minute stations with 2-minute pre-reading were used. 78 candidates and 14 interviewers participated in the selection process. All candidates attended the focus group. 58.7% of candidates responded to post vMMI questionnaire. 93% of survey responders were happy to undertake vMMI in the future, with 23% feeling they would have performed better in face-to-face. Experiences between 'metropolitan' and 'other' groups were similar. Positive experiences of participants were related to the user-friendly IT platform, successful pre-interview communications, preparation, convenience, time, and cost savings. Stress related IT failures and difficulties establishing rapport with interviewers were reported as the main negative experiences.

CONCLUSION

'vMMI' is a feasible and acceptable method of selection into paediatric training. vMMI has many benefits and can be implemented relatively quickly by addressing key logistical requirements. The model under discussion could be adapted by other centres based on local needs.

Metrics of Resident Achievement for Defining Program Aims

Introduction

Resident achievement data is a powerful but underutilized means of program evaluation, allowing programs to empirically measure whether they are meeting their program aims, facilitate refinement of curricula and improve resident recruitment efforts. The goal was to provide an overview of available metrics of resident achievement and how these metrics can be used to inform program aims.

Methods

A literature search was performed using PubMed and Google Scholar between May and November of 2020. Publications were eligible for inclusion if they discussed or assessed “excellence” or “success” during residency training. A narrative review structure was chosen due to the intention to provide an examination of the literature on available resident achievement metrics.

Results

57 publications met inclusion criteria and were included in the review. Metrics of excellence were grouped into larger categories, including success defined by program factors, academics, national competencies, employer factors, and possible new metrics.

Conclusions

Programs can best evaluate whether they are meeting their program aims by creating a list of important resident-level metrics based on their stated goals and values using one or more of the published definitions as a foundation. Each program must define which metrics align best with their individual program aims and mission.

Reliability and validity of new online selection tests for midwifery students

BACKGROUND AND OBJECTIVE

Newly created midwifery bachelor programmes attract an increasing number of applicants. Hence, educators have moved away from traditional non-standardised selection methods and started developing new tools for the admission of students. Researchers have expressed concerns around the validity of such selection measures and their ability to identify the appropriate candidates. The objective of this study was to develop and implement selection procedures for a new midwifery Bachelor of Science degree in Hamburg, Germany, and assess their psychometric properties.

DESIGN

This is a mixed-methods longitudinal study on the reliability, validity and acceptability of newly developed selection criteria.

SETTING

The newly established midwifery bachelor of science in Hamburg, Germany.

PARTICIPANTS

Upon completion of their online application, all midwifery applicants were invited by email to participate in our research project and thereby informed of its purpose and voluntary nature of their participation. The total number of candidates who took the test was 366, of which 309 agreed to take part in this study. Sixty-five applicants were eventually admitted to the midwifery Bachelor programme, of which 59 were included in the study.

MEASUREMENTS AND FINDINGS

We developed two tests for cognitive ability (HAM-Mid I and II) and adapted one test for personal and professional characteristics (Casper - Computer-based Assessment for Sampling Personal Characteristics). Due to the Covid pandemic, we implemented the tests online instead of on site as initially planned. HAM-Mid I had the lowest (Cronbach's Alpha = 0.38) whereas Casper had the highest internal consistency (Cronbach's Alpha = 0.77) of all three tests. Age, previous academic achievement and native tongue were significantly associated with applicants' performance on HAM-Mid II and Casper admission tests. HAM-Mid II was associated with students' performance in the first year midwifery exam (r = 0.31, p < 0.05). Overall evaluation of HAM-Mid II and Casper was good while HAM-Mid I received a more negative feedback.

KEY CONCLUSIONS

Evidence on the reliability and predictive validity of the newly developed tests suggests that the applied admission criteria are appropriate for an objective selection of bachelor midwifery students.

IMPLICATIONS FOR PRACTICE

The interests of university and clinical stakeholders of a midwifery bachelor degree can be represented through the development of tests for the assessment of cognitive ability and personal and professional characteristics.

[How does a patient-centered, future-oriented medical curriculum look like? The patients' view]

AIM OF THE STUDY

Medical training in Germany is in a state of flux. Medical education must be adapted to fit the changing health care needs of future societies. This study aims to gain the patients' perspective on the education of medical students to improve future curricula.

METHODOLOGY

Using a semi-structured interview guide, 17 interviews were conducted with patients undergoing inpatient (N=9) or outpatient (N=8) medical treatment. The interviews were subjected to computer-assisted analysis guided by Grounded Theory.

RESULTS

480 codings were condensed into seven main categories: Science Fiction Medicine, Society and the Patient, Professional Environment and Health Care System, Medical Curriculum and Continuing Education, Value System and Motivation, Psychosocial Competence and Medical Professional Competence. The largest main categories are Psychosocial Competence (114 codings) and Professional Environment and Health Care System (105 codings).

CONCLUSION

There is a need for psychosocial education in addition to medical knowledge. A future-focused curriculum should entail practice-oriented education to ensure the training of motivated physicians who possess strengthened digital skills and act on ethical principles and values. Assessment of the patients' perspective should play a crucial role in medical education research.

An investigation of the predictive validity of selection tools on performance in physiotherapy training in Australia

OBJECTIVES

Despite a large body of research on selection in medical education, very little is conducted in other health professions. This study investigated the predictive validity of multiple selection tools on academic and clinical performance outcomes of undergraduate physiotherapy students.

DESIGN

A retrospective observational study.

SETTING

Undergraduate physiotherapy program in Australia.

PARTICIPANTS

497 undergraduate physiotherapy students across seven entry cohorts. Including students directly from secondary school (n=381) and with prior tertiary study (n=116).

MAIN OUTCOME MEASURES

Academic performance as measured by written examinations. Clinical performance, measured by Objective Structured Clinical Examinations (OSCEs) during on-campus units and the Assessment of Physiotherapy Practice (APP) for off-campus clinical placements. Predictor variables included selection tools (academic achievement, interview, aptitude test) and demographic variables (age, gender).

RESULTS

Selection interview was a positive predictor of OSCEs and final year clinical performance in direct school leaver participants. Academic achievement scores from selection positively predicted written examinations scores.

CONCLUSION

Clinical and academic performance were predicted by tools measuring different domains at selection. Assessing broadly across academic and non-academic domains at selection can be valuable in identifying applicants who will be able to meet the range of outcomes for course completion and subsequent registration in the physiotherapy profession.

Does the ability to identify the construct on an empathy situational judgment test relate to performance? Exploring a new concept in assessment

INTRODUCTION

The ability to identify criteria (ATIC) refers to an examinee's capacity to distinguish the construct being evaluated. Previous research indicates ATIC can be predictive of performance on some assessments. This exploratory study investigated the relationship between a participant's ability to identify criteria and their performance on an empathy situational judgment test (SJT), an assessment format used to measure social and behavioral attributes.

METHODS

A 12-item empathy SJT was completed by 15 students and 15 pharmacists. During a cognitive interview, participants were asked what they believed each exam question measured. Responses were coded to determine whether participants stated "empathy" (indication of ATIC). The point-biserial correlation coefficient was calculated to explore the relationship of ATIC (correctly or not correctly identifying the item measured empathy) and performance on the SJT (total score).

RESULTS

Participants identified empathy 33.3% of the time, and it was the construct most often identified. Pharmacists (27.5%) identified empathy less often than students (39.2%). When empathy was identified as the construct, it was most often reported for items in a non-healthcare setting (56.3%) rather than a healthcare setting (43.7%) and for questions targeting affective empathy (71.3%) rather than cognitive empathy (28.7%). There were no statistically significant relationships with correctly identifying the construct and performance on individual items and the overall test.

CONCLUSIONS

There is inconclusive evidence that ATIC relates to performance on an empathy SJT. Additional research is needed to evaluate the role of ATIC and assessment performance to corroborate study results.

[Selection interviews at Hanover Medical School (MHH): determinants of student selection in medical studies]

INTRODUCTION

In order to incorporate social and communicative skills in its student admissions process, Hanover Medical School (MHH) has conducted selection interviews (in combination with the high-school GPA) to choose 60 % of its freshmen in medical studies. The present article analyses if applicants' performances in the interviews were the determining criterion of student selection, despite a higher weighting of school grades in the admission process. Furthermore, this article checks whether the grading of the interviews was independent of the applicants' gender, age, origin and educational background.

METHODS

For a sample of more than 3,000 successful and unsuccessful participants in the MHH student admission process in the years 2010-2017, we employ variance analysis and logistic regression analysis to determine those factors that have contributed to the chances of being offered a place at the MHH after a successful interview.

RESULTS

The scores received in the selection interview were the sole determinant of being offered a place at the MHH; neither the applicants'age nor their gender or origin biased the decisions of the selection committees. The grading of the interview was also not affected by school GPAs.

DISCUSSION

The selection interviews at the MHH have been costly in terms of both financial and human resources. However, this selection method has been popular among students and lecturers alike, not least because its elements fit the MHH's focus on early bedside teaching and social skills of prospective physicians. However, there is no evidence that selection interviews are effective in predicting academic success.

CONCLUSIONS

The present article has shown that the selection interviews acted up to the principles defined by the MHH: very homogenous high school GPAs were complemented by differentiated interview assessments that did not discriminate by sociodemographic characteristics. It is, however, unclear if the MHH will resume the interviews after the end of a federally mandated halt to stand-alone selection methods at medical schools in Germany.

Measuring differential attainment: a longitudinal analysis of assessment results for 1512 medical students at four Scottish medical schools

OBJECTIVE

To measure Differential Attainment (DA) among Scottish medical students and to explore whether attainment gaps increase or decrease during medical school.

DESIGN

A retrospective analysis of undergraduate medical student performance on written assessment, measured at the start and end of medical school.

SETTING

Four Scottish medical schools (universities of Aberdeen, Dundee, Edinburgh and Glasgow).

PARTICIPANTS

1512 medical students who attempted (but did not necessarily pass) final written assessment.

MAIN OUTCOME MEASURES

The study modelled the change in attainment gap during medical school for four student demographical categories (white/non-white, international/Scottish domiciled, male/female and with/without a known disability) to test whether the attainment gap grew, shrank or remained stable during medical school. Separately, the study modelled the expected versus actual frequency of different demographical groups in the top and bottom decile of the cohort.

RESULTS

The attainment gap grew significantly for white versus non-white students (t(449.39)=7.37, p=0.001, d=0.49 and 95% CI 0.34 to 0.58), for internationally domiciled versus Scottish-domiciled students (t(205.8) = -7, p=0.01, d=0.61 and 95% CI -0.75 to -0.42) and for male versus female students (t(1336.68)=3.54, p=0.01, d=0.19 and 95% CI 0.08 to 0.27). International, non-white and male students received higher marks than their comparison group at the start of medical school but lower marks by final assessment. No significant differences were observed for disability status. Students with a known disability, Scottish students and non-white students were over-represented in the bottom decile and under-represented in the top decile.

CONCLUSIONS

The tendency for attainment gaps to grow during undergraduate medical education suggests that educational factors at medical schools may-however inadvertently-contribute to DA. It is of critical importance that medical schools investigate attainment gaps within their cohorts and explore potential underlying causes.

Difference in demographics and motivation to study medicine with respect to medical students' channel of admission: A national study

INTRODUCTION

Medical schools employ various tools to select suitable medical students (MS). This study investigated whether MS who were admitted through multiple mini-interviews (MMI) and MS who were admitted through Taiwan's Joint College Entrance Written Test (JCEWT) differed in their characteristics.

METHODS AND SUBJECTS

First-year MS from seven medical schools completed a semi-structured questionnaire that inquired into their channel of admission (MMI or JCEWT), gender, location (metropolitan or rural), high school type (public or private), parents' socioeconomic status (SES), and motivations to study medicine.

RESULTS

In total, 513 MS participated, 493 (96%) returned valid questionnaires, and 397 were enrolled in the study, (MMI group: 205 MS; JCEWT group: 192 MS). Irrespective of channel of admission, most MS came from metropolitan areas (80%-86%), belonged to high-SES families (73%-76%), and had mixed motivations (51%-96%). Female applicants, private school leavers, and those who were less motivated by the physician's SES were more likely to be selected through the MMI channel than the JCEWT channel.

CONCLUSION

Irrespective of the channels of entry, MS had similar demographics and motivations for studying medicine. MS selected through MMI had different characteristics than those selected through a JCEWT.

Systematic review of specialist selection methods with implications for diversity in the medical workforce

PURPOSE

There is growing concern that inequities in methods of selection into medical specialties reduce specialist cohort diversity, particularly where measures designed for another purpose are adapted for specialist selection, prioritising reliability over validity. This review examined how empirical measures affect the diversity of specialist selection. The goals were to summarise the groups for which evidence is available, evaluate evidence that measures prioritising reliability over validity contribute to under-representation, and identify novel measures or processes that address under-representation, in order to make recommendations on selection into medical specialties and research required to support diversity.

METHOD

In 2020-1, the authors implemented a comprehensive search strategy across 4 electronic databases (Medline, PsychINFO, Scopus, ERIC) covering years 2000-2020, supplemented with hand-search of key journals and reference lists from identified studies. Articles were screened using explicit inclusion and exclusion criteria designed to focus on empirical measures used in medical specialty selection decisions.

RESULTS

Thirty-five articles were included from 1344 retrieved from databases and hand-searches. In order of prevalence these papers addressed the under-representation of women (21/35), international medical graduates (10/35), and race/ethnicity (9/35). Apart from well-powered studies of selection into general practice training in the UK, the literature was exploratory, retrospective, and relied upon convenience samples with limited follow-up. There was preliminary evidence that bias in the measures used for selection into training might contribute to under-representation of some groups.

CONCLUSIONS

The review did not find convincing evidence that measures prioritising reliability drive under-representation of some groups in medical specialties, although this may be due to limited power analyses. In addition, the review did not identify novel specialist selection methods likely to improve diversity. Nevertheless, significant and divergent efforts are being made to promote the evolution of selection processes that draw on all the diverse qualities required for specialist practice serving diverse populations. More rigorous prospective research across different national frameworks will be needed to clarify whether eliminating or reducing the weighting of reliable pre-selection academic results in selection decisions will increase or decrease diversity, and whether drawing on a broader range of assessments can achieve both reliable and socially desirable outcomes.

The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students

BACKGROUND

Admission to the Bachelor of Midwifery (BMid) in Australia has traditionally been based on academic ranking. The BMid is a high demand course offered to a limited number of students and therefore choosing applicants who complete the degree is important. Multiple Mini Interviews (MMIs) are used to assess non-cognitive skills and select students into healthcare degrees. One university in Australia has introduced MMIs as part of the application process for the BMid.

AIM

Compare attrition rates and Grade Point Average (GPA) scores between students admitted into the BMid using both academic ranking and MMIs, to those admitted on academic ranking alone.

METHODS

A basic convergent mixed methods design, using an explanatory unidirectional framework to integrate data. Attrition rates, GPA, and multiple mini interview scores (2013-2019), were linked and compared for before and after the use of MMI's. Focus groups with students, interviewers, and hospital-based educators, explored stakeholder experiences. Open-ended questions from an applicant survey were added to the qualitative data set, which was analysed thematically.

FINDINGS

Students who enrolled via the MMI's had significantly lower attrition rates than those enrolled before MMI's were introduced. GPA scores were significantly higher for students who enrolled via the MMI's. Integration of data found MMI's identified students passionate to undertake midwifery, and that success at the interviews increased students' confidence to successfully complete their studies.

CONCLUSION

MMI's as part of the entry process into the BMid enabled identification of applicants more likely to remain in the course and succeed in their studies.

Does performance at medical school predict success at the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination? A retrospective cohort study

BACKGROUND

Identifying predictors of success in postgraduate examinations can help guide the career choices of medical students and may aid early identification of trainees requiring extra support to progress in specialty training. We assessed whether performance on the educational performance measurement (EPM) and situational judgement test (SJT) used for selection into foundation training predicted success at the Membership of the Royal College of Surgeons (MRCS) examination.

METHODS

This was a longitudinal, cohort study using data from the UK Medical Education Database (https://www.ukmed.ac.uk). UK medical graduates who had attempted Part A (n=2585) and Part B (n=755) of the MRCS between 2014 and 2017 were included. χ² and independent t-tests were used to examine the relationship between medical school performance and sociodemographic factors with first-attempt success at MRCS Part A and B. Multivariate logistic regression was employed to identify independent predictors of MRCS performance.

RESULTS

The odds of passing MRCS increased by 55% for Part A (OR 1.55 (95% CI 1.48 to 1.61)) and 23% for Part B (1.23 (1.14 to 1.32)) for every additional EPM decile point gained. For every point awarded for additional degrees in the EPM, candidates were 20% more likely to pass MRCS Part A (1.20 (1.13 to 1.29)) and 17% more likely to pass Part B (1.17 (1.04 to 1.33)). For every point awarded for publications in the EPM, candidates were 14% more likely to pass MRCS Part A (1.14 (1.01 to 1.28)). SJT score was not a statistically significant independent predictor of MRCS success.

CONCLUSION

This study has demonstrated the EPM's independent predictive power and found that medical school performance deciles are the most significant measure of predicting later success in the MRCS. These findings can be used by medical schools, training boards and workforce planners to inform evidence-based and contemporary selection and assessment strategies.

Leveraging knowledge translation and implementation science in the pursuit of evidence informed health professions education

Evidence informed approaches to health professions education can ensure accountability to learners and society in providing meaningful and effective education and helping resource strained systems via streamlined and cost-efficient practices. Knowledge translation and implementation science are two areas of study originally developed in clinical medicine in response to concerns that health care practices were incongruent with the scientific evidence. Two decades of research have led to important advances in our understanding of the nature and magnitude of research-practice gaps, the factors that support or impede adoption of evidence in clinical decision-making, and in the design and evaluation of theory driven interventions to reduce gaps. This paper borrows concepts from knowledge translation and implementation science to further our thinking about how health professions education can 'truly' be evidence informed. The article is organised in four sections: a discussion of the impetus for the evidence informed health professions education movement; a description of the origins of knowledge translation and implementation science; a discussion on how knowledge translation and implementation science can be leveraged to advance the evidence informed health professions education agenda; and suggestions for future discussion and research. An example is used to illustrate the application of the underpinning principles of knowledge translation and implementation science. The authors suggest a theory driven, staged and systematic approach that integrates knowledge translation principles and processes and involves key stakeholders interested in promoting the application of educational research of evidence.

Medical Student Selection

Medical school admissions committees are tasked with fulfilling the values of their institutions through careful recruitment. Making accurate predictions regarding the enrollment behavior of admitted students is critical to intentionally formulating class composition and impacts long-term physician representation.

Perspectives on virtual interviews-A follow-up study of the Comprehensive Endocrine Surgery Fellowship interview process

BACKGROUND

The American Association of Endocrine Surgeons Comprehensive Endocrine Surgery Fellowship interview stakeholders previously favored in-person interviews, despite time and expense. This study assessed perception changes given mandated virtual interviews because of coronavirus disease 2019.

METHODS

Immediately after the 2020 Match, anonymous surveys were distributed to applicants (n = 37) and program directors (n = 22). Mixed-methods analyses were used to evaluate responses. Results were compared to data from a prior study of the 2013 to 2018 in-person interview process.

RESULTS

Response rates were 82% (program directors) and 60% (applicants). Compared with prior applicants, 2020 applicants attended similar numbers of interviews (1-10, 32% vs 37%; P = .61), used fewer vacation days (23% vs 56%; P = .01), and most reported 0 expenses. Burdens included lack of protected time for interviews. The virtual format did not compromise applicant ability to meet faculty (mean rank = 6.8/10) or make favorable impressions (mean rank = 6.8/10). Program directors reported equivalent or improved assessments of applicants. Program directors (72%) and applicants (77%) indicated that future interviews should be partially or completely virtual.

CONCLUSION

In contrast to prior survey data, applicants and program directors now express interest in virtual or hybrid interview processes. Virtual interviews were less costly, less time-consuming, and met goals effectively. Integrating virtual interview components will require innovative strategies to reduce redundancies and promote equitable access.

Dental foundation training in the COVID-19 era - the national recruitment lottery

National recruitment has radically transformed the selection and allocation of dental graduates to dental foundation training (DFT) schemes across England, Wales and Northern Ireland, with recruitment in Scotland via a separate, independent process. It has been reported as a mostly positive change to the previous deanery-led model, in which nepotism allegedly featured too widely. A candidate's ranking is typically based on performance across two face-to-face assessments and a situational judgement test (SJT). The COVID-19 pandemic, however, has created a recruitment 'lottery' of sorts, in which ranking for 2021 is now solely based on the SJT. Subject matter experts assert that neither preparation nor revision for the SJT is required; yet, following the announcement of the adaptation to DFT recruitment for September 2021, a rapid rise of exorbitantly priced SJT question banks, mock papers, workshops and courses has ensued - a shameless monetisation of the collective angst and increased pressure faced by dental students. Preparation courses present a conceivable risk of SJT 'coaching' and 'faking'. Where medicine leads, dentistry usually follows and future selection to DFT needs to strongly consider the introduction of academic performance measures. SJTs remain one of the most well-accepted, reliable and cost-effective means of selection into healthcare roles, however, only when part of a wider selection process.

Meritocratic and fair? The discourse of UK and Australia's widening participation policies

INTRODUCTION

Globally, people with the academic and personal attributes to successfully study medicine experience disadvantages associated with sociodemographic factors. Governments have attempted to address this issue via macrolevel policies aimed at widening participation (WP) to medicine. These policies differ by country, suggesting much can be learned from examining and comparing international policy discourses of WP. Our question was: How are discourses of WP to higher and medical education positioned in the UK and Australia?

METHODS

A systematic search strategy was guided by five a priori themes inspired by United Nations Sustainability Goals (2015). Seventeen policy documents (UK n = 9, Australia n = 8) published between 2008 and 2018 were identified. Analysis involved two over-arching, iterative stages: a document analysis then a Foucauldian critical discourse analysis, the latter with the aim of unveiling the power dynamics at play within policy-related discourses.

RESULTS

Discourses of social mobility and individual responsibility within a meritocracy are still paramount in the UK. In contrast, the dominant discourse in Australia is social accountability in achieving equity and workforce diversity, prioritising affirmative action and community values. Similarities between the two countries in terms of WP policy and policy levers have changed over time, linked to the divergence of internal drivers for societal change. Both nations recognise tensions inherent in striving to achieve both local and global goals, but Australia appears to prioritise community values in working towards 'nation building' whereas in the UK the focus on individuality and meritocracy at times seems at odds with achieving parity for disadvantaged individuals.

DISCUSSION

WP policies and practices are situated and contextual so caution must be taken when extrapolating lessons from one context to another. The history of a country and the nature of marginalisation in that country must be scrutinised when trying to understand what drives WP policy.

Utilization of a New Customizable Scoring Tool to Recruit and Select Pulmonary/Critical Care Fellows

Background: Finding the ideal candidate for a residency/fellowship program has always been difficult. Finding the “perfect” match has always been the ultimate goal. However, many factors affect obtaining that “perfect” match. In the past, we would have each attending physician review around 20 to 50 Electronic Residency Application Service (ERAS) applications and rank them into three categories: high, middle, or low. Depending on their ranking, the applicant would be invited for an interview. After the interview, the applicants’ files (ERAS and interview) would be reviewed and ranked by the faculty as a group. This was time-consuming and fraught with too much subjectivity and minimal objectivity. We, therefore, sought to find a way to assess and rank applicants in a more objective and less time-consuming manner. By creating a customizable scoring tool, we were able to screen applicants to our pulmonary/critical care fellowship program in an efficient and a more objective manner.

Objectives: A customizable scoring tool was developed weighting components in the ERAS and interview process, allowing residency/fellowship programs to create a final rank list consistent with the programs’ desired applicants.

Methods: Two hundred and sixty pulmonary/critical care fellowship applications were reviewed from 2013 to 2018. In 2018, we used our new scoring rubric to create a rank list and rescore previous applicants. The traditional and new lists were compared to the final rank list submitted to the National Residency Matching Program (NRMP) for 2018. We wanted to ascertain which scoring method correlated best with the final rank list submitted to the NRMP. We obtained feedback from eight faculty members who had reviewed applicants with both scoring tools.

Results: The novel customizable scoring tool positively correlated with the final rank list submitted to the NRMP (r= 0.86). The novel tool showed a better correlation to the final rank list than the traditional method. Faculties (6/6, 100%) responded positively to the new tool.

Conclusions: Our new customizable tool has allowed us to create a final rank list that is efficient and more focused on our faculty’s desired applicants. We hope to assess and compare the quality of applicants matched through this scoring system and the traditional method by using faculty evaluations, milestones, and test scores.

New Advances in Physician Assistant Admissions: The History of Situational Judgement Tests and the Development of CASPer

ABSTRACT

During the first 50 years of the physician assistant (PA) profession, admission to PA programs was based primarily on cognitive domains such as academic performance and standardized test scores. Many programs also considered other measurable factors, including patient care experience, community service, and extracurricular activities. While interviews were frequently conducted by the programs, it was not until the applicants had been "pre-screened" for the previously identified qualifications. As the PA profession continued to expand, PA programs began to realize that potentially strong applicants were being excluded from the admissions process because of this emphasis on mostly cognitive factors. In an attempt to reduce this disparity, PA programs have begun to expand their assessment of applicants to include assessment of noncognitive characteristics. This article outlines the history surrounding this change in the approach to admissions in medical education, reviews the development of situational judgement tests and other tools used to assess these noncognitive characteristics, and explores the relationship of these noncognitive characteristics to the development of program-defined competencies.

Multiple Mini-Interview Utilization in United States Physician Assistant Program Admission Processes

PURPOSE

Health care program admission processes utilize multi-faceted approaches to evaluate cognitive and noncognitive attributes of applicants. The multiple mini-interview (MMI) was developed in response to the need for a reliable and validated tool to assess noncognitive factors and has been increasingly incorporated into the admissions process by physician assistant (PA) programs. The study's purpose was to explore the current implementation and utilization of the MMI within PA programs.

METHODS

The study used a mixed-methods exploratory approach including a telephone survey and semi-structured interview of 11 PA programs using the MMI in their admissions process. Quantitative data collected included demographic information, MMI implementation characteristics, station structure, scoring, feasibility, satisfaction with MMI utilization, and MMI evaluation methods.

RESULTS

During the 2015-2016 admissions cycle, the participating programs used from 5 to 10 stations, averaging 7 minutes per station, requiring 8 faculty, 2 staff, and 7 students per interview session. Despite variation in program size, number of applicants, and years of MMI utilization, all participating programs reported that they were satisfied with the format and would continue to utilize the MMI in the admissions process.

CONCLUSIONS

While there is substantial literature describing the use of the MMI within health care programs globally, this study represents the first characterization of its use within PA programs on a national level. Although there was variation among PA program implementation of the MMI, our results are comparable to studies within other health care professions. Additional studies are necessary to further describe the MMI and its correlation with PA program educational outcomes and the impact on diversity.

Exploring the Role of Item Scenario Features on Situational Judgment Test Response Selections

Objective. To explore pharmacists' and pharmacy students' perceptions regarding the significance of changing the features of test item scenario (eg, switching from a health care to a non-health care context) on their situational judgment test (SJT) responses.Methods. Fifteen Doctor of Pharmacy students and 15 pharmacists completed a 12-item SJT intended to measure empathy. The test included six scenarios in a health care context and six scenarios in a non-health care context; participants had to rank potential response options in order of appropriateness and no two items could be of equal rank. Qualitative data were collected individually from participants using think-aloud and cognitive interview techniques. During the cognitive interview, participants were asked how they selected their final responses for each item and whether they would have changed their answer if features of the scenario were switched (eg, changed to a non-health care context if the original item was in a health care context). Interviews were transcribed and a thematic analysis was conducted to identify the features of the scenario for each item that were perceived to impact response selections.Results. Participants stated that they would have changed their responses on average 51.3% of the time (range 20%-100%) if the features of the scenario for an item were changed. Qualitative analysis identified four pertinent scenario features that may influence response selections, which included information about the examinee, the actors in the scenario, the relationship between examinee and actors, and details about the situation. There was no discernible pattern linking scenario features to the component of empathy being measured or participant type.Conclusion. Results from this study suggest that the features of the scenario described in an SJT item could influence response selections. These features should be considered in the SJT design process and require further research to determine the extent of their impact on SJT performance.

Are different medical school admission tests associated with the outcomes of a simulation-based OSCE?

BACKGROUND

Medical school admission procedures have the common goal to select applicants with the greatest potential of becoming successful physicians. Hamburg Medical Faculty selects medical students by grade point average (GPA) and employs a two-step selection process of a natural sciences test (HAM-Nat), in some cases followed by multiple mini-interviews (HAM-Int). Multiple mini-interviews can predict non-cognitive outcomes, while GPA has predictive validity for cognitive outcomes. The aim of our study was to explore communication skills and clinical knowledge of advanced medical students according to their respective admission procedure.

METHODS

In July 2019, 146 students grouped according to their admission procedure into GPA-only (19.2 %), HAM-Nat (33.6 %), HAM-Int (30.8 %), and Waiting List (16.4 %) participated in four OSCE stations which equally assessed students' communication skills (OSCE part 1) and clinical knowledge (OSCE part 2) in simulated patient encounters, rated by physicians with checklists. Additionally, psychosocial assessors ranked communication skills with a global rating scale (GR). The students also participated in a multiple choice (MC) exam testing clinical knowledge. Kruskal-Wallis analyses of variance of test performance and Spearman correlation of instruments were calculated.

RESULTS

Students from the Waiting List group performed significantly worse on the MC exam compared to GPA-only and HAM-Int (adjusted p = .029 and 0.018, respectively). No significant differences were found between the admission groups with respect to communication skills. Global Rating and OSCE part 1 (communication) correlated significantly (ρ = 0.228, p = .006) as did OSCE part 2 (clinical knowledge) and MC exam (ρ = 0.242, p = .003), indicating criterion validity. Constructs did not overlap, indicating divergent validity.

CONCLUSIONS

Advanced medical students selected for undergraduate studies by multiple mini-interviews assessing psychosocial skills showed similar communication skills compared to students admitted to medical school by other entryways. It is unclear whether these similarities are due to an effective undergraduate longitudinal communication curriculum. Assessing baseline communication skills of all medical students at entry-level may aid with this question.

National recruitment system for pre-registration pharmacist trainees in England and Wales: context, process and outcomes

In England and Wales, a new National Recruitment Scheme (NRS) for pre-registration pharmacists was introduced in 2017. The NRS replaced the traditional method of employer-led recruitment or the national recruitment system previously limited to some hospital training programmes. NRS uses assessment methodology involving Multiple-Mini Interviews, Situational Judgement Tests and numeracy skills in ranking and appointing candidates to their preferred training providers through a centralised procedure. This commentary will describe the background, assessment methodology, outcomes of preliminary evaluations and priority research areas for the future in the context of the NRS.

Performance at medical school selection correlates with success in Part A of the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination

Medical schools in the UK typically use prior academic attainment and an admissions test (University Clinical Aptitude Test (UCAT), Biomedical Admissions Test (BMAT) or the Graduate Medical School Admissions Test (GAMSAT)) to help select applicants for interview. To justify their use, more information is needed about the predictive validity of these tests. Thus, we investigated the relationship between performance in admissions tests and the Membership of the Royal College of Surgeons (MRCS) examination.The UKMED database (https://www.ukmed.ac.uk) was used to access medical school selection data for all UK graduates who attempted MRCS Part A (n=11 570) and Part B (n=5690) between 2007 and 2019. Univariate and multivariate logistic regression models identified independent predictors of MRCS success. Pearson correlation coefficients examined the linear relationship between test scores and MRCS performance.Successful MRCS Part A candidates scored higher in A-Levels, UCAT, BMAT and GAMSAT (p<0.05). No significant differences were observed for MRCS Part B. All admissions tests were found to independently predict MRCS Part A performance after adjusting for prior academic attainment (A-Level performance) (p<0.05). Admission test scores demonstrated statistically significant correlations with MRCS Part A performance (p<0.001).The utility of admissions tests is clear with respect to helping medical schools select from large numbers of applicants for a limited number of places. Additionally, these tests appear to offer incremental value above A-Level performance alone. We expect this data to guide medical schools' use of admissions test scores in their selection process.

Unraveling the medical residency selection game

The diversity of modern society is often not represented in the medical workforce. This might be partly due to selection practices. We need to better understand decision-making processes by selection committees in order to improve selection procedures with regard to diversity. This paper reports on a qualitative study with a socio-constructivist perspective conducted in 2015 that explored how residency selection decision-making occurred within four specialties in two regions in the Netherlands. Data included transcripts of the decision-making meetings and of one-on-one interviews with committee members before and after the group decision-making meetings. Candidates struggled to portray themselves favorably as they had to balance playing by the rules and being authentic; between fitting in and standing out. Although admissions committees had a welcoming stance to diversity, their practices were unintentionally preventing them from hiring underrepresented minority (URM) candidates. While negotiating admissions is difficult for all candidates, it is presumably even more complicated for URM candidates. This seems to be having a negative influence on attaining workforce diversity. Current beliefs, which make committees mistakenly feel they are acting fairly, might actually justify biased practices. Awareness of the role of committee members in these processes is an essential first step.

Relationship between admission criteria and academic performance in basic science courses in health science colleges in KAU

BACKGROUND

At King Abdulaziz University, medical and health science schools depend on admission exams (aptitude and achievement) and preparatory year scores in their students' selection. However, with the growing number of applicants and the drastic changes in teaching and assessment in these colleges, continuous assessment and development of admission criteria are needed. In this study, we aimed to evaluate the correlation of admission exam scores, in addition to the preparatory year Grade Point Average (GPA), with academic performance in the basic science subjects such as Clinical Biochemistry and Clinical Pharmacology in health science colleges.

METHODS

The study was conducted on four cohort studies, two faculty of nursing cohorts; nursing students (2017-2018, n=146) nursing students (2018-2019, n=81), and two faculty of applied medical sciences cohorts, clinical nutrition students (2017-2018, n=33), and clinical nutrition students (2018-2019, n=28). The students' scores of General Aptitude Test (GAT), Scholastic Achievement Admission Test (SAAT), and preparatory year GPA were all recorded at the beginning of each semester before the beginning of courses. Clinical Biochemistry and Clinical Pharmacology exam results were recorded at the end of the semester. Correlation was done for each cohort and all cohorts pooled.

RESULTS

Results showed only a weak correlation detected between SAAT and the overall achievement in Clinical Biochemistry (r= 0.192, P= 0.042) in nursing students (2017-2018), but no correlation was seen with SAAT or preparatory year scores. There was also no significant correlation between admission exams scores and the students' academic achievement in Clinical Biochemistry or Clinical Pharmacology. On the other hand Clinical Pharmacology exam results showed a significant positive correlation with Clinical Biochemistry results (r=0.688, P=0.000).

CONCLUSION

Our results could indicate the need to revisit the admission criteria for these colleges. Furthermore, specific preparatory year tracks for health science colleges can ensure that students improve the specific skills and knowledge required for their future college years3.

Predictors of students' academic achievements in allied health professions at King Saud University: a retrospective cohort study

BACKGROUND

The admissions criteria for colleges of medicine and allied health professions include several cognitive predictors. Little is known of the admissions criteria for the allied health professions and their correlation with students' academic performance. This study investigates predictors for students' academic achievements at allied health colleges at King Saud University.

DESIGN

Retrospective cohort study.

SETTINGS

College of Applied Medical Sciences, College of Nursing, and Prince Sultan bin Abdulaziz College for Emergency Medical Services, Saudi Arabia.

PARTICIPANTS

The sample comprised 1634 students.

METHOD

The high school grade average (HSGA), aptitude test (APT) score, achievement test (ACT) score, and current grade point average (GPA) were retrieved. The data were analysed using Pearson's correlation coefficient and regression analysis.

RESULTS

HSGA, ACT, and APT were significantly positively associated with students' academic performance in colleges for all allied health professions. Multivariate regression analysis showed that the most predictive variable for all allied healthcare professions was HSGA (β = 0.347), followed by ACT (β = 0.270) and APT (β = 0.053) scores. The regression model indicated that the HSGA, APT, and ACT together predicted 26.5% of the variation in students' cumulative GPAs at the time of graduation.

CONCLUSION

The admissions criteria for the allied health colleges at King Saud University predicted only 26.5% of the students' cumulative GPA at the time of graduation. Other noncognitive admission criteria should be taken into consideration to improve the prediction of students' academic potential.

Using cognitive interviews and think-aloud protocols to understand thought processes

OUR SITUATION

Cognitive interviews (CIs) and think-aloud protocols (TAPs) are two strategies that offer unique approaches to capture participant thought processes when engaging with instruments and interventions. In this paper, we describe the origins of CIs and TAPs, the types of research questions they can be used to address, and strategies to implement these approaches.

METHODOLOGICAL LITERATURE REVIEW

CIs and TAPs are interview methods that make thought processes more explicit. CIs include targeted interview questions that are focused on evaluating participant comprehension during an activity, such as a survey. TAPs request the participant to think-aloud while they solve a problem or complete a task; this is to help understand the problem-solving process.

OUR RECOMMENDATIONS AND THEIR APPLICATIONS

An understanding of the research aim is critical to identify whether a CI or TAP is preferred. Researchers should ensure that participants are selected based on the needs of the research and that they are appropriately prepared for the selected interview type. Individuals can analyze verbal reports from CIs and TAPs using qualitative and quantitative methods, and researchers are encouraged to disseminate their findings to inform design practices of other researchers.

POTENTIAL IMPACT

CIs and TAPs are methods that help pharmacy education researchers explore thought processes; specifically, these methods are useful to study test response processes, improve survey construction, and map cognitive models of complex thought processes. CIs and TAPs are methodologies that should be in the toolbox of pharmacy education researchers to support design, evaluation, and refinement of instruments and educational interventions.

Benefits and Feasibility of Using Videos to Assess Medical School Applicants' Empathetic Abilities in Multiple Mini Interviews

PURPOSE

We sought to evaluate the feasibility and benefits of using video-based scenarios in Multiple Mini Interviews (MMIs) to assess candidate's empathic abilities by investigating candidate perceptions and the acceptability, fairness, reliability, and validity of the test.

METHODS

The study sample was candidates for admission interviews held in the MMI format at a medical school in South Korea. In this six-station MMI, one station included a 2-min video clip of a patient-doctor communication scenario to assess candidate emphatic abilities, whereas paper-based scenarios were used in the other stations. Candidate's perceptions and acceptability of using the video-based scenario in the empathy station were examined using a 41-item post-MMI questionnaire. Fairness of the test was assessed by means of differences in candidate perceptions and performance across different demographics or backgrounds. Construct validity was assessed by examining the relationship of candidate performances in the empathy station with those in other stations. The G-coefficient was analyzed to estimate the reliability of the test.

RESULTS

Eighty-two questionnaires were returned, a 97.6% response rate. Candidates showed overall positive perceptions of the video-based scenario and they found it authentic and interesting. The test was fair as there were no differences in candidates' perceptions of the patient-doctor relationship presented in the video clip and neither in their performance nor in their perceived difficulty of the station across demographics or backgrounds. Construct validity was established as candidate performance in the empathy station was not associated with that of any other stations. The G-coefficient was 0.74.

CONCLUSIONS

The present study demonstrates that the video-based scenario is a feasible tool to assess candidate's empathy in the MMI.

Does the UKCAT predict performance in medical and dental school? A systematic review

OBJECTIVES

For the first time, this systematic review provides a summary of the literature exploring the relationship between performance in the UK Clinical Aptitude Test (UKCAT) and assessments in undergraduate medical and dental training.

DESIGN

In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis, relevant studies were identified through systematic literature searches. Electronic searches were carried out on EBSCO, EMBASE, Educational Resources Information Centre, SCOPUS, Web of Knowledge. Studies which included the predictive validity of selection criteria including some element of the UKCAT were considered.

RESULTS

22 papers were identified for inclusion in the study. Four studies describe outcomes from dental programmes with limited results reported. 18 studies reported on relationships between the UKCAT and performance in undergraduate medical training. Of these, 15 studies reported relationships between the UKCAT cognitive tests and undergraduate medical assessments. Weak relationships (r=0.00-0.29) were observed in 14 of these studies; four studies reported some moderate relationships (r=0.30-0.49). The strongest relationships with performance in medical school were observed for the UKCAT total score and the verbal reasoning subtest. Relationships with knowledge-based assessments scores were higher than those for assessments of skills as the outcome. Relationships observed in small (single and double centre studies) were larger than those observed in multicentre studies.

CONCLUSION

The results indicate that UKCAT scores predict performance in medical school assessments. The relationship is generally weak, although noticeably stronger for both the UKCAT total score and the verbal reasoning subtest. There is some evidence that UKCAT continues to predict performance throughout medical school. We recommend more optimal approaches to future studies. This assessment of existing evidence should assist medical/dental schools in their evaluation of selection processes.