Adjusting medical school admission: assessing interpersonal skills using situational judgement tests

Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach

Introduction

Situational judgment tests (SJT) are commonly used in admissions to measure skills associated with professionalism. Although open-response SJTs have shown strong psychometric properties, assessors' personal beliefs, experiences, and cultural backgrounds may influence how they perceive, organize and evaluate information within test takers' diverse responses. Additionally, SJT research typically focuses on reliability and predictive validity, whereas the construct validity of open response SJTs remains underexplored. This mixed methods study aims to address this gap by exploring the construct-(ir)relevant factors that may impact assessors' evaluation of professionalism in open response SJTs.

Methods

For this study, we used data from Casper, an open response SJT commonly used in professional program admissions. In Study I, a quantitative content analysis was conducted on 160 responses to identify factors which were significant predictors of low and high scores. Correlation coefficients and logistic regression models were used to evaluate the relationship between each factor and response scores. In Study II, think-aloud activities were conducted with 23 Casper assessors to directly observe how they evaluated responses. All interviews were transcribed verbatim, which were then thematically analyzed using an inductive coding technique.

Results

Results from both the content analyses and think-aloud activities revealed that several construct relevant factors influenced scores. Scores were impacted by the extent to which test takers demonstrated the competencies probed for by the SJT, engaged with the context of the presented ethical dilemma, provided in-depth justifications for their response, considered various perspectives relevant to the presented dilemma, and provided creative solutions or insightful arguments for the suggested approach. Mixed results were found with respect to construct irrelevant factors, such as the flow, cohesion, and kinds of phrases used in the response.

Conclusion

This mixed methods study contributes to the construct validity of SJTs by investigating construct relevant and irrelevant factors that may impact assessors' evaluation of open responses. The findings of this study provide evidence that open-response SJTs are valid approaches to measure professional competencies more broadly, both in terms of what test takers focus on in their responses, as well as in terms of how they construct their responses.

Physicians' Social Skills - Conceptualization, Taxonomy, and Behavioral Assessment

Social skills (e.g., assertiveness, empathy, ability to accept criticism) are essential for the medical profession and therefore also for the selection and development of medical students. However, the term "social skills" is understood differently in different contexts. There is no agreed upon taxonomy for classifying physicians' social skills, and skills with the same meaning often have different names. This conceptual ambiguity presents a hurdle to cross-context communication and to the development of methods to assess social skills. Drawing from behavioral psychology, we aim to contribute to a better understanding of social skills in the medical context. To this end, we introduce a theoretically and empirically informed taxonomy that can be used to integrate the large number of different social skills. We consider how skills manifest at the behavioral level to ensure that we focus only on skills that are actually observable, distinguishable, and measurable. Here, behavioral research has shown that three overarching skill dimensions can be seen in interpersonal situations and are clearly distinguishable from each other: agency skill (i.e., getting ahead in social situations), communion skill (i.e., getting along in social situations), and interpersonal resilience (i.e., staying calm in social situations). We show that almost all social skills relevant for physicians fit into this structure. The approach presented allows redundant descriptions to be combined under three clearly distinguishable and behavior-based dimensions of social skills. This approach has implications for the assessment of social skills in both the selection and development of students.

Development and validation of situational judgement test for assessment of behavioural competencies required for effective medical practice in Nigeria

BACKGROUND

The personal qualities of health workers determine the way health services are provided to clients. Some key personal qualities (also called behavioural competencies) of physicians that contribute to quality healthcare delivery include ethical responsibility, empathy, patient-centeredness, diligence, good judgment, respectful, teamwork, team leadership/conflict management, ability to take correction and tolerance. In this study, we developed and validated clinical scenarios (dilemmas) for assessing priority behavioural competencies for medical practice in Nigeria.

METHODS

Drawing on prioritized competencies generated via a scoping review and nominal group technique (NGT) exercises in a previous study in the study series, Faculty members from the University of Nigeria Teaching Hospital were consulted to develop or adapt clinical scenarios that could be used to assess these competencies in a physician. The clinical scenarios and options were framed as situational judgement tests (SJTs) and these tests were administered to a random sample of 192 undergraduate and 111 postgraduate medical doctors in a tertiary hospital in Enugu State. Using Kane's validity argument framework, we assessed scoring and generalization inferences of situational judgment tests (SJT) based on the developed scenarios.

RESULT

Scoring inference - difficulty and discrimination index - shows that most of the SJT items are good test items and can differentiate between high and low performers. The corrected point biserial correlations show positive correlation for most of the items. Generalization inference shows the items represent the domains of interest and are internally consistent. However, few items that show poor difficulty and discrimination index were subjected to re-evaluation and possible elimination.

CONCLUSION

This study has produced a set of valid clinical scenarios that can be used to evaluate specific behavioural competencies among trainee medical doctors. It demonstrates that SJTs can be used to assess behavioural competencies for medical practice. However, further research is needed to establish the applicability of SJT beyond the immediate context, such as the medical school, in which it is developed.

Predicting undergraduate OSCE performance using traditional and construct-driven situational judgment tests at admission

Situational Judgement Tests (SJTs) are popular to screen for social skills during undergraduate medical admission as they have been shown to predict relevant study outcomes. Two different types of SJTs can be distinguished: Traditional SJTs, which measure general effective behavior, and construct-driven SJTs which are designed to measure specific constructs. To date, there has been no comparison of the predictive validity of these two types of SJTs in medical admission. With the present research, we examine whether the HAM-SJT, a traditional SJT, and the CD-SJT, a construct-driven SJT with an agentic and a communal scale, administered during undergraduate medical admission can predict OSCE (i.e., objective structured clinical examination) performance in a low-stakes (nLS = 159) and a high-stakes (nHS = 160) sample of medical students. Results showed a moderate positive relation between the communal scale of the CD-SJT and performance in OSCE stations with trained patients in the high-stakes sample (r =.20, p =.009). This SJT had also an incremental value in predicting the OSCE performance above and beyond GPA (i.e., grade point average), a science test (i.e., HAM-Nat), and gender (ß = 0.18, 95% CI [0.03; 0.33], p =.020). That is, individuals who chose more communal behavioral responses in the SJT were rated more favorably in interactions with trained patients in the OSCE. A comparable correlation coefficient was observed for the HAM-SJT when controlling for range restriction due to admission (rraw = 0.14 vs. rcontrolled = 0.20). Our research provides a first indication for the predictive validity of construct-driven SJTs in high-stakes undergraduate medical admission.

Predictors of Success in a Graduate, Entry-Level Professional Program: From Admissions to Graduation

INTRODUCTION

Admission to health professional programs (HPPs) in Canada is competitive. The purpose of this study is to evaluate how factors identifiable by the admissions package may predict incidences of academic concerns in one physiotherapy program in Canada.

REVIEW OF LITERATURE

Previous literature has identified many concepts that contribute to "academic success." Some HPPs have investigated if admissions criteria can predict students' academic performance. However, this has not been reported in physiotherapy programs in Canada.

SUBJECTS

Study data included candidates' admissions' metrics and physiotherapy students' program data for 4 graduating cohorts, who were admitted from 2016 to 2019 inclusive (N = 256).

METHODS

A retrospective, nonconcurrent cohort study was used to estimate the relationship between applicant's admissions data and students' program data pertaining to academic success. Data were summarized as frequencies for categorical variables and means for continuous variables. We calculated odds ratios (ORs) and probabilities of an academic or professional concern for standard scores. Significance was set at P < .05.

RESULTS

Cohorts participating in the multiple mini-interview (MMI) had an academic concern incidence of 14/131. The virtual MMI (VMMI) cohort had an incidence of 7/125. Students with higher MMI scores were less likely to have an academic concern (OR = 0.52 [95% CI: 0.30-0.89, P = .017]). Grade point average was not significantly associated with an academic concern when combined with either MMI or VMMI (Ps > 0.05). Admissions round offer was also significantly associated with an academic concern (OR = 2.48 [95% CI: 1.00-6.12, P = .049]), with those beyond the initial round of offers having increased risk of concerns.

DISCUSSION AND CONCLUSION

Results of the study reflect the generally low event rates for incidences of academic concerns and the relative homogeneity and range restriction of independent variables across the 4 cohorts of students. HPP's reflection on current admissions processes and ability to identify opportunities for change in admission processes helps ensure that programs are selecting candidates who are likely to succeed.

Developing a Situational Judgement Test to Assess Clinical Judgement in Fourth-Year Medical Students: A Pilot Study

Introduction Assessing clinical judgement objectively and economically presents a challenge in academic medicine. The authors developed a situational judgement test (SJT) to measure fourth-year medical students' clinical judgement. Methods A knowledge-based, single-best-answer SJT was developed by a panel of subject matter experts (SMEs). The SJT included 30 scenarios, each with five response options ranked ordinally from most to least appropriate. A computer-based format was used, and the SJT was piloted by two cohorts of fourth-year medical students at California University of Science and Medicine in 2022 and 2023 upon completion of an internship preparation course. Subsequently, students completed an optional survey. Evaluated scoring methods included original ordinal ranking, dichotomous, dichotomous with negative correction, distance from SME best answer, and distance from SME best answer squared. Results The SJT was completed by 142 fourth-year medical students. Cronbach's alpha ranged from 0.39 to 0.85, depending on the scoring method used. The distance-from-SME-best-answer-squared method yielded the highest internal consistency, which was considered acceptable. Using this scoring method, the mean score was 72.89 (SD = 48.32, range = 26-417), and the standard error of measurement was 18.41. Item analysis found that seven (23%) scenarios were of average difficulty, 13 (43%) had a good or satisfactory discrimination index, and nine (30%) had a distractor efficiency of at least 66%. Most students preferred the SJT to a traditional multiple-choice exam (16; 62%) and felt it was an appropriate tool to assess clinical judgement (15; 58%). Conclusions The authors developed and piloted an SJT to assess clinical judgement among medical students. Although not achieving validation, subsequent development of the SJT will focus on expanding the SME concordance panel, improving difficulty and discrimination indices, and conducting parallel forms reliability and adverse impact analyses.

Going Above and Beyond With SJTs: Impact of Applicant Characteristics on Open Response SJT Participation

BACKGROUND

Situational judgment tests (SJT) have gained popularity as a standardized assessment of nontechnical competencies for applicants to medical school and residency. SJT formats range from rating the effectiveness of potential response options to solely open response. We investigated differences in test-taking patterns between responders and nonresponders to optional open response SJT questions during the application process.

METHODS

This was a prospective multi-institutional study of general surgery applicants to seven residency programs. Applicants completed a 32-item SJT designed to measure ten core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Each SJT item included an optional, nonscored, open response space for applicants to provide a behavioral response if they desired. Trends in applicant gender, race, ethnicity, medical school ranking, and USMLE scores were examined between the responder versus nonresponder group.

RESULTS

In total, 1491 general surgery applicants were invited to complete the surgery-specific SJT. Of these, 1454 (97.5%) candidates completed the assessment and 1177 (78.9%) provided additional responses to at least one of the 32 SJT scenario sets. There were no differences in overall SJT performance, USMLE scores (Step 1: 235, SD 14, Step 2: 250, SD 11), race and/or ethnicity between the responder and nonresponder groups. Responders were more likely to be from a top 25 medical school (p < 0.05) compared to the nonresponder group. Among applicants who completed any open response questions, women completed a significantly higher number of questions compared to men (7.21 vs 6.07, p = 0.003). The number of open responses provided correlated with higher scores on SJT items measuring dependability (r = 0.07, p = 0.007).

CONCLUSIONS

SJT design and format has the potential to impact test-taker response patterns. SJT developers and adopters should ensure test format and design have no unintended consequences prior to implementation.

Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes

BACKGROUND

Computer-based assessment for sampling personal characteristics (Casper), an online situational judgement test, is a broad measure of personal and professional qualities. We examined the impact of Casper in the residency selection process on professionalism concerns, learning interventions and resource utilization at an institution.

METHODS

In 2022, admissions data and information in the files of residents in difficulty (over three years pre- and post- Casper implementation) was used to determine the number of residents in difficulty, CanMEDS roles requiring a learning intervention, types of learning interventions (informal learning plans vs. formal remediation or probation), and impact on the utilization of institutional resource (costs and time). Professionalism concerns were mapped to the 4I domains of a professionalism framework, and their severity was considered in mild, moderate, and major categories. Descriptive statistics and between group comparisons were used for quantitative data.

RESULTS

In the pre- and post- Casper cohorts the number of residents in difficulty (16 vs. 15) and the number of learning interventions (18 vs. 16) were similar. Professionalism concerns as an outcome measure decreased by 35% from 12/16 to 6/15 (p < 0.05), were reduced in all 4I domains (involvement, integrity, interaction, introspection) and in their severity. Formal learning interventions (15 vs. 5) and informal learning plans (3 vs. 11) were significantly different in the pre- and post-Casper cohorts respectively (p < 0.05). This reduction in formal learning interventions was associated with a 96% reduction in costs f(rom hundreds to tens of thousands of dollars and a reduction in time for learning interventions (from years to months).

CONCLUSIONS

Justifiable from multiple stakeholder perspectives, use of an SJT (Casper) improves a clinical performance measure (professionalism concerns) and permits the institution to redirect its limited resources (cost savings and time) to enhance institutional endeavors and improve learner well-being and quality of programs.

Demographic disparities in dental school selection: An analysis of current UK practices

INTRODUCTION

Specific social groups remain under-represented within dentistry. While the University Clinical Aptitude Test (UCAT) aims to widen participation in under-represented social groups, there is no evidence in dental education that this aim is being met.

MATERIALS AND METHODS

Data over two admission cycles (2012 and 2013), including 3246 applicants to 10 UK dental schools, were analysed. Applicant and selected pools were compared to the UK population. Multiple logistic regression was used to investigate the association between demographic variables and UCAT and receiving an offer of a place at dental school.

RESULTS

Over-representation of Female, Asian, least deprived and grammar school groups were found in applicant and selected pools compared with the UK population. White ethnic applicants were significantly more selected than Black (OR 0.25), Asian (OR 0.57) and Mixed (OR 0.80) ethnicities, while least deprived applicants were significantly more selected than most deprived (OR 0.59). Grammar school education increased odds of selection by 1.8 when compared to state school. The addition of UCAT to the model for applicants reduced ethnic disparities but led to disparities between other groups.

CONCLUSION

Current widening participation practices focus on attracting applicants from lower socio-economic groups. However, this study showed that ethnicity, sex and educational background biases also affect demographic diversity in dentistry. The UCAT shows promise in levelling the playing field; however, widening access measures will only succeed if selection committees radically change selection processes to address the systemic biases, enabling the dentists of tomorrow to represent the society they serve.

Evaluating non-cognitive skills in medical school applicants

Medical school candidates must have both the cognitive and professional competencies required to become good physicians. In this commentary, we outline the evidence and outcomes associated with the implementation of these selection methodologies and evaluate their ability to assess non-cognitive skills.

Impact of patterns of language use and socio-economic status on a constructed response Situational Judgment Test (SJT)

The study explored the impacts of patterns of language use (PLU) and socio-economic status (SES) on Casper, a constructed-response situational judgment test (SJT). 10,266 applicants from two U.S. medical schools were grouped into self-reported balanced bilinguals, unbalanced bilinguals, English monolinguals, and English as a Second Language (ESL) students. A multicomponent SES composite was used to assess the degree of socioeconomic disadvantage (DSD). Results from a hierarchical regression analysis showed that after accounting for demographic variables, both PLU and DED were significant factors on applicants' Casper performance. Bilingualism was associated with better Casper performance compared to English monolinguals and ESL students. No significant effect of speaking English as a native language was found on applicants' Casper performance. English monolinguals and ESL students performed equivalently on Casper. Finally, high DSD was associated with better Casper performance than low DSD, and the impact of DSD on Casper held the same across all four language groups. These findings provide evidence that socio-cultural factors, such as PLU and DSD have important impacts on SJT performance. Further research is needed to understand the role of differences in language construction across socio-cultural factors on constructed-response SJT performance.

Against Intelligence: Rethinking Criteria for Medical School Admissions

Intelligence, as measured by grades and/or standardized test scores, plays a principal role in the medical school admissions process in most nations. Yet while sufficient intelligence is necessary to practice medicine effectively, no evidence suggests that surplus intelligence beyond that threshold is correlated with providing higher quality medical care. This paper argues that using perceived measures of intelligence to distinguish between applicants, at levels that exceed the level of intelligence required to practice medicine, is both unfair to applicants and fails to serve the interests of patients.

A Semistructured Interview for Surgical Residency Targeting Nontechnical Skills

OBJECTIVE

We review the development, implementation, and initial outcomes of a semistructured interview process to assess the nontechnical skills of surgical residency applicants.

DESIGN

In 2018, we restructured our residency selection interview process. Through semistructured faculty interviews, we sought to evaluate candidates along seven nontechnical skills (grit, ownership, rigor, teamwork, presence, impact, and organizational alignment). We plotted each candidate's scores on a radar plot for graphical representation and calculated the plot area of each candidate. We retrospectively evaluated 3 years of data, comparing the nontechnical skill scores of matriculants into the training program to those of nonmatriculants.

SETTING

Tenwek Hospital is a 361-bed tertiary teaching and referral hospital in rural western Kenya with a 5-year general surgery residency program.

PARTICIPANTS

Thirty-one applicants were interviewed over 3 years. Thirteen matriculated into the program.

RESULTS

Scores for grit, (4.8 vs 3.9; p = 0.0004), impact (4.2 vs 3.5; p = 0.014), ownership (4.2 vs 3.6; p = 0.01), and organizational alignment (4.3 vs 3.8; p = 0.008) were significantly higher in matriculants.

CONCLUSIONS

This semistructured interview process provides a robust and beneficial mechanism for assessing applicants' nontechnical skills, which may allow for the matriculation of more well-rounded candidates into surgical residency and, ultimately, surgical practice.

A video-based situational judgement test of medical students' communication competence in patient encounters: Development and first evaluation

OBJECTIVE

We developed and evaluated the Video-Based Assessment of Medical Communication Competence (VA-MeCo), a construct-driven situational judgement test measuring medical students' communication competence in patient encounters.

METHODS

In the construction phase, we conducted two expert studies (n_panel1 = 6, n_panel2 = 13) to ensure curricular and content validity and sufficient expert agreement on the answer key. In the evaluation phase, we conducted a cognitive pre-test (n = 12) and a pilot study (n = 117) with medical students to evaluate test usability and acceptance, item statistics and test reliability depending on the applied scoring method (raw consensus vs. pairwise comparison scoring).

RESULTS

The results of the expert interviews indicated good curricular and content validity. Expert agreement on the answer key was high (ICCs> .86). The pilot study showed favourable usability and acceptance by students. Irrespective of the scoring method, reliability for the complete test (Cronbach's α >.93) and its subscales (α >.83) was high.

CONCLUSION

There is promising evidence that medical communication competence can be validly and reliably measured using a construct-driven and video-based situational judgement test.

PRACTICE IMPLICATIONS

Video-based SJTs allow efficient online assessment of medical communication competence and are well accepted by students and educators.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery

OBJECTIVE

The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences.

DESIGN

PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded.

RESULTS

Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation.

CONCLUSIONS

As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

Assessing distinguishable social skills in medical admission: does construct-driven development solve validity issues of situational judgment tests?

BACKGROUND

Social skills are important for future physicians and are therefore increasingly considered in selection processes. One economic assessment method from which different social skills can be inferred are Situational Judgment Tests (SJTs) in which applicants are asked to rate behavioral responses in context-relevant situations. However, traditional SJTs have so far failed to distinctively measure specified constructs. To address this shortcoming in the medical admission context, we applied a construct-driven approach of SJT development in which test development was deductively guided by agency and communion as target constructs.

METHOD

The final version of the construct-driven SJT includes 15 items per construct with three behavioral responses. Medical school applicants (N = 1527) completed the construct-driven SJT, a traditional SJT, and an aptitude test under high-stakes condition as part of their application. A subsample (N = 575) participated in a subsequent voluntary online study with self-report measures of personality and past behavior.

RESULTS

The proposed two-factor structure and internal consistency of the construct-driven SJT was confirmed. Communal SJT scores were positively associated with self-reported communal personality and communal behavior, yet effects were smaller than expected. Findings for agentic SJT scores were mixed with positive small associations to self-reported agentic personality scores and agentic behavior but unexpected negative relations to communal self-reported measures.

CONCLUSIONS

Results suggest that construct-driven SJTs might overcome validity limitations of traditional SJTs, although their implementation is challenging. Despite first indicators of validity, future research needs to address practical points of application in high-stakes settings, inclusion of other constructs, and especially prediction of actual behavior before the application of construct-driven SJTs for selection purposes in medical admission can be recommended.

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues

BACKGROUND

Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile.

METHODS

Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508).

RESULTS

The SJT demonstrated good internal consistency (Cronbach's alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity.

CONCLUSIONS

This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

The selection gap in teacher education: Adverse effects of ethnicity, gender, and socio-economic status on situational judgement test performance

Background

Situational judgement tests (SJTs) measure non‐cognitive attributes and have recently drawn attention as a selection method for initial teacher education programmes. To date, very little is known about adverse impact in teacher selection SJT performance.

Aims

This study aimed to shed light on adverse effects of gender, ethnicity, and socio‐economic status (SES) on SJT scores, by exploring both main effects and interactions, and considering both overall SJT performance and separate SJT domain scores (mindset, emotion regulation, and conscientiousness).

Sample

A total of 2,808 prospective teachers from the United Kingdom completed the SJTs as part of the initial stage of selection into a teacher education programme.

Methods

In addition to SJT scores, the variables gender (female vs. male), ethnicity (majority group vs. minority group), and home SES background (higher SES status vs. lower SES status) were used in the analyses. Regression models and moderated regression models were employed.

Results and conclusions

Results from the regression models revealed that gender effects (females scoring higher than males) were restricted to emotion regulation, while ethnicity effects (ethnic majority group members scoring higher than ethnic minority group members) emerged for SJT overall scores and all three domains. Moderated regression modelling results furthermore showed significant interactions (gender and ethnicity) for SJT overall scores and two domains. Considering the importance of reducing subgroup differences in selection test scores to ensure equal access to teacher education, this study’s findings are a critical contribution. The partially differentiated results for overall vs. domain‐specific scores point towards the promise of applying a domain‐level perspective in research on teacher selection SJTs.

[Situational judgement test as teaching method for the critical discussion on scientific practice and misconduct]

Wissenschaftskompetenz ist eine Schlüsselqualifikation für jede ärztliche Tätigkeit und sollte ebenso wie die Auseinandersetzung mit Entscheidungsprozessen von Beginn an ins Medizinstudium integriert werden. Ziel der Studie war, die Themen der guten wissenschaftlichen Praxis und des wissenschaftlichen Fehlverhaltens zu vermitteln. Ferner wurde durch die methodische Intervention „Gruppendiskussion“ eine Reflexion im Kontext der wissenschaftlichen Angemessenheit herbeigeführt. Hierfür wurde der Situational Judgement Test (SJT) von den Studierenden (N = 743) (individuell und in der Gruppe) bearbeitet, und dessen Resultate wurden mit den Antworten von Expert/innen/en (N = 23) verglichen. Nach der Gruppendiskussion näherten sich die Ergebnisse in der Verteilung und Reihenfolge den Antwortmöglichkeiten der Expert/innen/en an. Jedoch tendierten die Studierenden signifikant häufiger zu jenen Antworten, die hilfesuchende, passive und verantwortungsübertragende Optionen bedeuteten. Insgesamt hat sich der SJT als didaktische Intervention bewährt. Die Studierenden setzten sich aktiv mit den Themen auseinander, eine Diskussion konnte angeregt und das eigene Verhalten kritisch reflektiert werden.

Blackface in White Space: Using Admissions to Address Racism in Medical Education

Given the long history and pervasive nature of racism in medical culture, this essay argues that diversifying efforts alone cannot address systemic racism in medical education. Positive affirmation of anti-racist values and racial consciousness in the admissions process is necessary to create a truly inclusive culture in medical education and begin to undo centuries of racial prejudice in medicine. Drawing from historic examples, scholarship on the sociology of racialized space, recent research on race and medical education, and personal experience, we propose that medical educational institutions make a more concerted effort to consider racial attitudes and awareness as part of the admissions process as well as curricular reform efforts. We also provide examples of potential ways to practically implement this proposal in the admissions process.

Situational judgement test validity for selection: A systematic review and meta-analysis

CONTEXT

Situational judgement tests (SJTs) are widely used to evaluate 'non-academic' abilities in medical applicants. However, there is a lack of understanding of how their predictive validity may vary across contexts. We conducted a systematic review and meta-analysis to synthesise existing evidence relating to the validity of such tools for predicting outcomes relevant to interpersonal workplace performance.

METHODS

Searches were conducted in relevant databases to June 2019. Study quality and risk of bias were assessed using the Quality In Prognosis Studies (QUIPS) tool. Results were pooled using random effects meta-analysis and meta-regressions.

RESULTS

Initially, 470 articles were identified, 218 title or abstracts were reviewed, and 44 full text articles were assessed with 30 studies meeting the final inclusion criteria and were judged, overall, to be at moderate risk of bias. Of these, 26 reported correlation coefficients relating to validity, with a pooled estimate of 0.32 (95% confidence interval 0.26 to 0.39, P < .0001). Considerable heterogeneity was observed (I²  = 96.5%) with the largest validity coefficients tending to be observed for postgraduate, rather than undergraduate, selection studies (β = 0.23, 0.11 to 0.36, P < .001). The correction of validity coefficients for attenuation was also independently associated with larger effects (β = 0.13, 0.03 to 0.23, P = .01). No significant associations with test medium (video vs text format), cross-sectional study design, or period of assessment (one-off vs longer-term) were observed. Where reported, the scores generally demonstrated incremental predictive validity, over and above tests of knowledge and cognitive ability.

CONCLUSIONS

The use of SJTs in medical selection is supported by the evidence. The observed trend relating to training stage requires investigation. Further research should focus on developing robust criterion-relevant outcome measures that, ideally, capture interpersonal aspects of typical workplace performance. This will facilitate additional work identifying the optimal place of SJTs within particular selection contexts and further enhancing their effectiveness.

The association between Situational Judgement Test (SJT) scores and professionalism concerns in undergraduate medical education

Introduction: Situational judgement tests (SJTs) are widely used in selecting medical students and doctors. Emerging evidence suggests SJTs are capable of testing an individual's ability to respond to role-relevant professionalism scenarios, however, evidence is lacking for their use in identifying students with concerning professional behaviours.This study aimed to determine the association between medical student professionalism-based SJT scores and recorded professionalism concerns during training.Methods: A cross-sectional study was conducted utilising SJT scores from second-year medical students and occurrences of student professionalism concerns. Concerns were reviewed and mapped to General Medical Council standards. Multivariate logistic regression was used to determine associations between SJT scores and professionalism concerns.Results: 247 students were included in the study. For every point increase in SJT score, students were 10% less likely to have multiple professionalism concerns [OR (95% CI) 0.90 (0.83-0.97); p = .007].Students scoring below 1 and 2 standard deviations from the mean score were 4 and 11-times more likely to have multiple concerns [OR (95% CI) 4.52 (1.12-18.25); p = .034] and [OR (95% CI) 11.45 (1.72-76.15); p = .012].Conclusion: Lower SJT scores were significantly associated with an increased risk of professionalism concerns. These findings support the potential for SJT exams to identify medical students that may require closer supervision and remediation during undergraduate education.

Construct validity of multiple mini interviews - Investigating the role of stations, skills, and raters using Bayesian G-theory

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures.Methods: We used data from three MMI samples (N = 376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills.Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures.Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.

Evaluation of a Situational Judgement Test to Develop Non-Academic Skills in Pharmacy Students

Objective. To design, implement, and psychometrically evaluate a situational judgement test (SJT) to use as a formative assessment of pharmacy students' non-academic skills in an Australian-based university. Methods. An SJT was developed using a previously validated design process including involvement of subject matter experts. The first phase included design of a blueprint through stakeholder consultation and the development of bespoke attribute definitions and a tool specification. Following on from this, SJT items were developed through subject matter expert interviews and in-depth review process. Results. Students (702) from four different cohorts (first through fourth years) of a Bachelor of Pharmacy degree program completed the situational judgment test. Data from 648 students was eligible for inclusion in the analysis. The SJT demonstrated good reliability, appropriateness for use (difficulty and quality), fairness, and face validity. The variability in students' scores suggested that the SJT may be a useful metric to identify students most in need of additional support. Conclusion. Evaluation of the SJT demonstrated that the tool was valid, reliable, fair and appropriate to use as a formative assessment. Through implementing an SJT such as this, pharmacy students are provided the opportunity to receive feedback on their non-academic skills and consider how to approach challenging or unfamiliar situations before entering the profession.

Medical student selection criteria and junior doctor workplace performance

BACKGROUND

Medical school selection decisions have consequences beyond graduation. With generally low attrition rates, most medical students become junior doctors. Universities are therefore not just selecting students into a medical course; they are choosing the future medical workforce. Understanding the relationship between selection criteria and outcomes beyond the successful completion of a medical degree may inform approaches to student selection.

METHODS

A retrospective data matching study was conducted involving 39 interns employed by a South Australian local health network in 2017 who had originally entered Flinders University's medical school through a graduate pathway. Student selection data were matched with internship workplace performance scores (measured by supervising consultants' reports across five clinical rotations using a standardised assessment). Correlational analyses then examined associations between these two sets of variables.

RESULTS

An overall selection rank (equal thirds of weighted Grade Point Average from a prior degree, a panel interview, and a national selection test) was moderately associated with all performance measures, accounting for up to 25% of variance. Both weighted Grade Point Average and the interview had multiple and mostly moderate correlations with performance. An increasing number of years taken to complete the course was associated with poorer workplace performance across multiple outcome measures (moderate to strong negative associations with 31 to 62% of shared variance), as was age to a lesser extent (7 to 14%). The national selection test contributed a single and small relationship accounting for 5% of variance with one outcome measure.

CONCLUSIONS

Selection into medicine is a critical assessment given that most students become doctors. This study found multiple associations between selection scores and junior doctor workplace performance measures in the internship year, with weighted Grade Point Average from a prior degree and an interview appearing more important than the national selection test. Future collaborative research should map desired workplace performance outcomes to initial student selection and explore the impact of changes to selection which focus on assessment of these domains. The association between slower course progression and poorer workplace performance should also be examined.

Lines in the sand: pre-interview rank and probability of receiving admission to medical school

BACKGROUND

We provide an examination of one medical school's attempt to determine whether their cut-off point for number of interviews offered is congruent with the probability these applicants' have for admission post- interview.

METHODS

Offer probability was determined by organizing pre-interview rankings from 2013-2017 (n = 2,659) applicant cohorts into bins of 50 applicants and finding the quotient of successful and total applicants in each bin. A linear-by-linear association Chi-square test and adjusted standardized residuals with an applied Bonferroni correction were used to determine if the observed frequencies in each bin were different than expected by chance. A Spearman Correlation analysis between pre- and post-interview ranks was conducted.

RESULTS

All applicants have between a 50.0% and 76.4% chance of admission. Observed frequencies are different than chance (χ(1)=50.835, p<.001), with a significantly greater number of offers seen in the bins between 1 and 100 (p<.001 for both bins). There is a weak positive relationship between pre- and post-rank, r_s(2657)= 0.258, p<.001.

CONCLUSION

The results indicate the number of interviews conducted does not exceed a threshold wherein individuals with a relatively low chance of admission are interviewed. Findings are interpreted with respect to ethical resource allocation for both programs and applicants.

[Selection of medical students : Measurement of cognitive abilities and psychosocial competencies]

The German Constitutional Court is currently reviewing whether the actual study admission process in medicine is compatible with the constitutional right of freedom of profession, since applicants without an excellent GPA usually have to wait for seven years. If the admission system is changed, politicians would like to increase the influence of psychosocial criteria on selection as specified by the Masterplan Medizinstudium 2020.What experiences have been made with the actual selection procedures? How could Situational Judgement Tests contribute to the validity of future selection procedures to German medical schools?High school GPA is the best predictor of study performance, but is more and more under discussion due to the lack of comparability between states and schools and the growing number of applicants with top grades. Aptitude and knowledge tests, especially in the natural sciences, show incremental validity in predicting study performance. The measurement of psychosocial competencies with traditional interviews shows rather low reliability and validity. The more reliable multiple mini-interviews are superior in predicting practical study performance. Situational judgement tests (SJTs) used abroad are regarded as reliable and valid; the correlation of a German SJT piloted in Hamburg with the multiple mini-interview is cautiously encouraging.A model proposed by the Medizinischer Fakultätentag and the Bundesvertretung der Medizinstudierenden considers these results. Student selection is proposed to be based on a combination of high school GPA (40%) and a cognitive test (40%) as well as an SJT (10%) and job experience (10%). Furthermore, the faculties still have the option to carry out specific selection procedures.

Embracing standardisation and contextualisation in medical education

CONTEXT

The tensions that emerge between the universal and the local in a global world require continuous negotiation. However, in medical education, standardization and contextual diversity tend to operate as separate philosophies, with little attention to the interplay between them.

METHODS

The authors synthesise the literature related to the intersections and resulting tensions between standardization and contextual diversity in medical education. In doing so, the authors analyze the interplay between these competing concepts in two domains of medical education (admissions and competency-based medical education), and provide concrete examples drawn from the literature.

RESULTS

Standardization offers many rewards: its common articulations and assumptions promote patient safety, foster continuous quality improvement, and enable the spread of best practices. Standardization may also contribute to greater fairness, equity, reliability and validity in high stakes processes, and can provide stakeholders, including the public, with tangible reassurance and a sense of the stable and timeless. At the same time, contextual variation in medical education can afford myriad learning opportunities, and it can improve alignment between training and local workforce needs. The inevitable diversity of contexts for learning and practice renders any absolute standardization of programs, experiences, or outcomes an impossibility.

CONCLUSIONS

The authors propose a number of ways to examine the interplay of contextual diversity and standardization and suggest three ways to move beyond an either/or stance. In reconciling the laudable goals of standardization and the realities of the innumerable contexts in which we train and deliver care, we are better positioned to design and deliver a medical education system that is globally responsible and locally engaged.

Advancing the Science of Collaborative Problem Solving

Collaborative problem solving (CPS) has been receiving increasing international attention because much of the complex work in the modern world is performed by teams. However, systematic education and training on CPS is lacking for those entering and participating in the workforce. In 2015, the Programme for International Student Assessment (PISA), a global test of educational progress, documented the low levels of proficiency in CPS. This result not only underscores a significant societal need but also presents an important opportunity for psychological scientists to develop, adopt, and implement theory and empirical research on CPS and to work with educators and policy experts to improve training in CPS. This article offers some directions for psychological science to participate in the growing attention to CPS throughout the world. First, it identifies the existing theoretical frameworks and empirical research that focus on CPS. Second, it provides examples of how recent technologies can automate analyses of CPS processes and assessments so that substantially larger data sets can be analyzed and so students can receive immediate feedback on their CPS performance. Third, it identifies some challenges, debates, and uncertainties in creating an infrastructure for research, education, and training in CPS. CPS education and assessment are expected to improve when supported by larger data sets and theoretical frameworks that are informed by psychological science. This will require interdisciplinary efforts that include expertise in psychological science, education, assessment, intelligent digital technologies, and policy.

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection

BACKGROUND

This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.

METHODS

Medical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.

RESULTS

Applicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants - but not ethnic majority applicants - showed greater preference for should-do than would-do instructions. Additionally, first-generation university students - but not non-first-generation university students - were more favourable of rating formats than of pick-one formats.

CONCLUSIONS

Findings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

Do changing medical admissions practices in the UK impact on who is admitted? An interrupted time series analysis

INTRODUCTION

Medical admissions must balance two potentially competing missions: to select those who will be successful medical students and clinicians and to increase the diversity of the medical school population and workforce. Many countries address this dilemma by reducing the heavy reliance on prior educational attainment, complementing this with other selection tools. However, evidence to what extent this shift in practice has actually widened access is conflicting.

AIM

To examine if changes in medical school selection processes significantly impact on the composition of the student population.

DESIGN AND SETTING

Observational study of medical students from 18 UK 5-year medical programmes who took the UK Clinical Aptitude Test from 2007 to 2014; detailed analysis on four schools.

PRIMARY OUTCOME

Proportion of admissions to medical school for four target groups (lower socioeconomic classes, non-selective schooling, non-white and male).

DATA ANALYSIS

Interrupted time-series framework with segmented regression was used to identify the impact of changes in selection practices in relation to invitation to interview to medical school. Four case study medical schools were used looking at admissions within for the four target groups.

RESULTS

There were no obvious changes in the overall proportion of admissions from each target group over the 8-year period, averaging at 3.3% lower socioeconomic group, 51.5% non-selective school, 30.5% non-white and 43.8% male. Each case study school changed their selection practice in decision making for invite to interview during 2007-2014. Yet, this within-school variation made little difference locally, and changes in admission practices did not lead to any discernible change in the demography of those accepted into medical school.

CONCLUSION

Although our case schools changed their selection procedures, these changes did not lead to any observable differences in their student populations. Increasing the diversity of medical students, and hence the medical profession, may require different, perhaps more radical, approaches to selection.

Becoming more holistic: A literature review of nonacademic factors in the admissions process of colleges and schools of pharmacy and other health professions

BACKGROUND

The expanding role of pharmacists has influenced admissions committees to consider nonacademic factors other than grade point average (GPA) and test scores and has focused more attention on holistic admissions.

METHODS

Database searches were conducted in PubMed, CINAHL, ERIC, and PsychINFO using keywords, "holistic admissions", "holistic review", and "pharmacy admissions" plus "critical thinking skills", "extracurricular", "communication skills", "essay", or "interview". Overall, 64 studies were identified, 17 were excluded, and ultimately, 47 were reviewed.

RESULTS

Holistic admissions is not an industry standard in pharmacy but more so in other health professions. For critical thinking skills, the Health Sciences Reasoning Test (HSRT) was not a good predictor of academic performance even though it was effective in ranking admission applicants. The California Critical Thinking Skills Test (CCTST) however, was a significant predictor for clerkship and practice-related courses. It is unclear whether pharmacy admissions committees are utilizing the Pharmacy College Admissions Test (PCAT) reading and writing scores, the interview, or other measures to evaluate communication skills. The Multiple Mini Interview (MMI) is an effective assessment tool for measuring noncognitive attributes; however, the efficacy of unstructured interviews in evaluating noncognitive skills was less clear.

IMPLICATIONS

Academic measures alone are not nuanced enough to predict success throughout the entire curriculum. An integration of factors, both academic and nonacademic, would be more relevant to predict success. Critical thinking skills and extracurricular experiences may be more significant for admissions selection and admissions ranking respectively, and more predictive of academic success during didactic and experiential stages of the curriculum respectively.

Selection into medicine: the predictive validity of an outcome-based procedure

BACKGROUND

Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program.

METHODS

We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students.

RESULTS

Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs.

CONCLUSIONS

The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.

A systematic review of stakeholder views of selection methods for medical schools admission

BACKGROUND

The purpose of this paper is to systematically review the literature with respect to stakeholder views of selection methods for medical school admissions.

METHODS

An electronic search of nine databases was conducted between January 2000-July 2014. Two reviewers independently assessed all titles (n = 1017) and retained abstracts (n = 233) for relevance. Methodological quality of quantitative papers was assessed using the MERSQI instrument. The overall quality of evidence in this field was low. Evidence was synthesised in a narrative review.

RESULTS

Applicants support interviews, and multiple mini interviews (MMIs). There is emerging evidence that situational judgement tests (SJTs) and selection centres (SCs) are also well regarded, but aptitude tests less so. Selectors endorse the use of interviews in general and in particular MMIs judging them to be fair, relevant and appropriate, with emerging evidence of similarly positive reactions to SCs. Aptitude tests and academic records were valued in decisions of whom to call to interview. Medical students prefer interviews based selection to cognitive aptitude tests. They are unconvinced about the transparency and veracity of written applications. Perceptions of organisational justice, which describe views of fairness in organisational processes, appear to be highly influential on stakeholders' views of the acceptability of selection methods. In particular procedural justice (perceived fairness of selection tools in terms of job relevance and characteristics of the test) and distributive justice (perceived fairness of selection outcomes in terms of equal opportunity and equity), appear to be important considerations when deciding on acceptability of selection methods. There were significant gaps with respect to both key stakeholder groups and the range of selection tools assessed.

CONCLUSIONS

Notwithstanding the observed limitations in the quality of research in this field, there appears to be broad concordance of views on the various selection methods, across the diverse stakeholders groups. This review highlights the need for better standards, more appropriate methodologies and for broadening the scope of stakeholder research.

Admission testing for higher education: A multi-cohort study on the validity of high-fidelity curriculum-sampling tests

We investigated the validity of curriculum-sampling tests for admission to higher education in two studies. Curriculum-sampling tests mimic representative parts of an academic program to predict future academic achievement. In the first study, we investigated the predictive validity of a curriculum-sampling test for first year academic achievement across three cohorts of undergraduate psychology applicants and for academic achievement after three years in one cohort. We also studied the relationship between the test scores and enrollment decisions. In the second study, we examined the cognitive and noncognitive construct saturation of curriculum-sampling tests in a sample of psychology students. The curriculum-sampling tests showed high predictive validity for first year and third year academic achievement, mostly comparable to the predictive validity of high school GPA. In addition, curriculum-sampling test scores showed incremental validity over high school GPA. Applicants who scored low on the curriculum-sampling tests decided not to enroll in the program more often, indicating that curriculum-sampling admission tests may also promote self-selection. Contrary to expectations, the curriculum-sampling tests scores did not show any relationships with cognitive ability, but there were some indications for noncognitive saturation, mostly for perceived test competence. So, curriculum-sampling tests can serve as efficient admission tests that yield high predictive validity. Furthermore, when self-selection or student-program fit are major objectives of admission procedures, curriculum-sampling test may be preferred over or may be used in addition to high school GPA.

Response distortion on personality tests in applicants: comparing high-stakes to low-stakes medical settings

The current study examined the degree to which applicants applying for medical internships distort their responses to personality tests and assessed whether this response distortion led to reduced predictive validity. The applicant sample (n = 530) completed the NEO Personality Inventory whilst applying for one of 60 positions as first-year post-graduate medical interns. Predictive validity was assessed using university grades, averaged over the entire medical degree. Applicant responses for the Big Five (i.e., neuroticism, extraversion, openness, conscientiousness, and agreeableness) and 30 facets of personality were compared to a range of normative samples where personality was measured in standard research settings including medical students, role model physicians, current interns, and standard young-adult test norms. Applicants had substantially higher scores on conscientiousness, openness, agreeableness, and extraversion and lower scores on neuroticism with an average absolute standardized difference of 1.03, when averaged over the normative samples. While current interns, medical students, and especially role model physicians do show a more socially desirable personality profile than standard test norms, applicants provided responses that were substantially more socially desirable. Of the Big Five, conscientiousness was the strongest predictor of academic performance in both applicants (r = .11) and medical students (r = .21). Findings suggest that applicants engage in substantial response distortion, and that the predictive validity of personality is modest and may be reduced in an applicant setting.

Integrity situational judgement test for medical school selection: judging 'what to do' versus 'what not to do'

CONTEXT

Despite their widespread use in medical school selection, there remains a lack of clarity on exactly what situational judgement tests (SJTs) measure.

OBJECTIVES

We aimed to develop an SJT that measures integrity by combining critical incident interviews (inductive approach) with an innovative deductive approach. The deductive approach guided the development of the SJT according to two established theoretical models, of which one was positively related to integrity (honesty-humility [HH]) and one was negatively related to integrity (cognitive distortions [CD]). The Integrity SJT covered desirable (HH-based) and undesirable (CD-based) response options. We examined the convergent and discriminant validity of the Integrity SJT and compared the validity of the HH-based and CD-based subscores.

METHODS

The Integrity SJT was administered to 402 prospective applicants at a Dutch medical school. The Integrity SJT consisted of 57 scenarios, each followed by four response options, of which two represented HH facets and two represented CD categories. Three SJT scores were computed, including a total, an HH-based and a CD-based score. The validity of these scores was examined according to their relationships with external integrity-related measures (convergent validity) and self-efficacy (discriminant validity).

RESULTS

The three SJT scores correlated significantly with all integrity-related measures and not with self-efficacy, indicating convergent and discriminant validity. In addition, the CD-based SJT score correlated significantly more strongly than the HH-based SJT score with two of the four integrity-related measures.

CONCLUSIONS

An SJT that assesses the ability to correctly recognise CD-based response options as inappropriate (i.e. what one should not do) seems to have stronger convergent validity than an SJT that assesses the ability to correctly recognise HH-based response options as appropriate (i.e. what one should do). This finding might be explained by the larger consensus on what is considered inappropriate than on what is considered appropriate in a challenging situation. It may be promising to focus an SJT on the ability to recognise what one should not do.

Exploring uses of the UK Clinical Aptitude Test-situational judgement test in a dental student selection process

AIM

In 2013 the UKCAT included a non-cognitive situational judgement test in addition to the four cognitive subtests commonly used by UK dental schools to select students. However, little is known about the situational judgement test's psychometric properties and relationship to other selection tools. This study's aim was explore these issues to inform decisions about the inclusion of the UKCAT SJT in the dental student selection process.

METHOD

The sample comprised a cohort of applicants to a Bachelor of Dental Surgery programme, at a UK dental school, which does not use achievement in the UKCAT SJT in its selection process. The impact on applicant outcomes of two theoretical uses of the UKCAT SJT was examined. First, SJT Band was used as the criterion for offers of a place instead of the school's admission interview, and second, weighted UKCAT scores, including a weighted SJT Band score, were ranked to make interview invitation decisions. Associations between SJT Band, UKCAT cognitive scores, interview score and performance in first year assessments were examined.

RESULTS

If SJT Band 1 & 2 were used as the criterion for an offer of a place, some applicants rejected by this school's interview, including 'red flagged' applicants, would have received an offer of a place. Using a weighted UKCAT/SJT system for invitation for interview decisions increased the mean total UKCAT cognitive score of those invited for interview but included applicants rejected by this school's structured interview, including 'red flagged' applicants. Neither usage disadvantaged under-represented groups. SJT Band correlated with UKCAT score (n = 228, r_s = -0.38, P < 0.01) with interview score (n = 186, r_s = -0.17, P < 0.05) but not with first year study assessments.

CONCLUSION

This study has shown that the UKCAT SJT does not add value to the existing methods of this dental school to make valid, reliable and fair student selection decisions.

Lessons learned from 15 years of non-grades-based selection for medical school

CONTEXT

Thirty years ago, it was suggested in the Edinburgh Declaration that medical school applicants should be selected not only on academic, but also on non-academic, attributes. The main rationale behind extending medical school selection procedures with the evaluation of (non-academic) personal qualities is that this will lead to the selection of students who will perform better as a doctor than those who are selected on the basis of academic measures only. A second rationale is the expectation that this will lead to a representative health workforce as a result of reduced adverse impact. The aims of this paper are (i) to describe what can be learned about the use of selection criteria other than grades from over 15 years of Dutch experience and (ii) to summarise current knowledge on the issue of adverse impact in relation to non-grades-based selection.

METHODS

A narrative review was undertaken of the (published) evidence that has resulted from non-grades-based school-specific selection procedures in the Netherlands and from recent explorations of the effect of the use of non-grades-based selection criteria on student diversity.

RESULTS

The Dutch evidence is grouped into five key themes: the effect of participation in voluntary selection procedures, the assessment of pre-university extracurricular activities, the use of work samples, Dutch experiences with situational judgement tests and the effects of changing circumstances. This is followed by several lessons learned for medical schools that aim to increase their student diversity.

CONCLUSION

Over the last 30 years, important steps towards reliable and valid methods for measuring non-academic abilities have been taken. The current paper describes several lessons that can be learned from the steps taken in the Dutch context. The importance of sharing evidence gathered around the globe and building on this evidence to reach our goal of predicting who will be a good doctor is acknowledged.

Is perfect good? - Dimensions of perfectionism in newly admitted medical students

BACKGROUND

Society expects physicians to perform perfectly but high levels of perfectionism are associated with symptoms of distress in medical students. This study investigated whether medical students admitted to medical school by different selection criteria differ in the occurrence of perfectionism.

METHODS

Newly enrolled undergraduate medical students (n = 358) filled out the following instruments: Multidimensional Perfectionism Scale (MPS-H), Multidimensional Perfectionism Scale (MPS-F), Big Five Inventory (BFI-10), General Self-Efficacy Scale (GSE), Patient Health Questionnaire 9 (PHQ-9), and Generalized Anxiety Disorder 7 (GAD-7). Sociodemographic data such as age, gender, high school degrees, and the way of admission to medical school were also included in the questionnaire.

RESULTS

The 298 participating students had significantly lower scores in Socially-Prescribed Perfectionism than the general population independently of their way of admission to medical school. Students who were selected for medical school by their high school degree showed the highest score for Adaptive Perfectionism. Maladaptive Perfectionism was the strongest predictor for the occurrence symptoms of depression and anxiety regardless of the way of admission.

CONCLUSIONS

Students from all admission groups should be observed longitudinally for performance and to assess whether perfectionism questionnaires might be an additional useful instrument for medical school admission processes.

Medical doctors' job specification analysis: A qualitative inquiry

Purpose: A qualitative inquiry was conducted to investigate the qualification requirements of medical doctors in different professional fields and from different perspectives. The inquiry was part of an empirical workplace analysis. Methods: Seventy-four structured interviews were conducted and analyzed to examine critical incidents and behaviors of medical doctors working in different professional fields (clinical theory, clinical practice, practitioner) and disciplines, and from three different perspectives (medical doctors, non-medical staff, and patients). In addition, the National Competency-based Catalogue of Learning Objectives for Medical Education (Nationaler Kompetenzbasierter Lernzielkatalog Medizin / NKLM) was used. Results: The results revealed eleven relevant competencies, which could be categorized into three superordinate competence clusters: interpersonal, work-related, and self-related. The perspectives of medical doctors and non-medical staff included all eleven competencies. However, the perspective of patients did not include one interpersonal and two self-related competencies. Nearly all of the critical behaviors mentioned are included in the NKLM. However, the NKLM also includes behaviors that were not mentioned in the interviews. Conclusions: The behavior-oriented interviews resulted in a requirement profile that is very similar in structure to other competency models in occupational contexts. Comparisons of the different perspectives predominantly revealed similarities. However, the patient perspective also revealed interesting differences compared to the perspectives of medical doctors and non-medical staff. The behavior-related results of the interviews can be directly used for the development of exercises in selection and personnel development contexts and for potential appraisals specific to different medical disciplines. In future steps, the results of this initial qualitative step are to be replicated and extended using quantitative studies and a representative sample. The main overall objective is the definition of relevant competencies both for the selection and development of medical students and for the design of potential appraisals as part of personnel development programs in different medical disciplines.

The Predictive Validity of a Text-Based Situational Judgment Test in Undergraduate Medical and Dental School Admissions

PROBLEM

Situational judgment tests (SJTs) can be used to assess the nonacademic attributes necessary for medical and dental trainees to become successful practitioners. Evidence for SJTs' predictive validity, however, relates predominantly to selection in postgraduate settings or using video-based SJTs at the undergraduate level; it may not be directly transferable to text-based SJTs in undergraduate medical and dental school selection. This preliminary study aimed to address these gaps by assessing the validity of the UK Clinical Aptitude Test (UKCAT) text-based SJT.

APPROACH

Study participants were 218 first-year medical and dental students from four UK undergraduate schools who completed the first UKCAT text-based SJT in 2013. Outcome measures were educational supervisor ratings of in-role performance in problem-based learning tutorial sessions-mean rating across the three domains measured by the SJT (integrity, perspective taking, and team involvement) and an overall judgment of performance-collected in 2015.

OUTCOMES

There were significant correlations between SJT scores and both mean supervisor ratings (uncorrected r = 0.24, P < .001; corrected r = 0.34) and overall judgments (uncorrected rs = 0.16, P < .05; corrected rs = 0.20). SJT scores predicted 6% of variance in mean supervisor ratings across the three nonacademic domains.

NEXT STEPS

The results provide evidence that a well-designed text-based SJT can be appropriately integrated, and add value to, the selection process for undergraduate medical and dental school. More evidence is needed regarding the longitudinal predictive validity of SJTs throughout medical and dental training pathways, with appropriate outcome criteria.

A new admission method may select applicants with a distinct personality profile

BACKGROUND

There is little data to suggest that a specific admission method can select students with a distinct personality profile. We have recently introduced a new admission process that combines a computerized personality test, with a single interview. The purpose of the current study was to find whether the new method selects applicants with a different personality profile and attitudes compared with the previous method.

METHOD

Using a validated personality questionnaire (HEXACO) and attitudes questionnair, that were filled anonymously between November 2014 and May 2015, the authors compared two groups of students: group A comprising students accepted with the new method (first and second year) with group B comprising students accepted with the previous method (third to sixth year).

RESULTS

In group A, 157 responded out of 250 (63%), while in group B 194 out of 352 (55%). Group A students ranked significantly higher in honesty-humility, extraversion, agreeableness and openness to experience, and lower in emotionality. Physicians' role in society was perceived to be more meaningful among Group A students (M = 4.19, SD = 0.50, N = 152) compared to Group B students (M = 3.86, SD = 0.57, N = 184).

CONCLUSIONS

The new method may select applicants with a distinct personality profile and different attitudes toward the physicians' role in the society.

Evaluating the complementary roles of an SJT and academic assessment for entry into clinical practice

Although there is extensive evidence confirming the predictive validity of situational judgement tests (SJTs) in medical education, there remains a shortage of evidence for their predictive validity for performance of postgraduate trainees in their first role in clinical practice. Moreover, to date few researchers have empirically examined the complementary roles of academic and non-academic selection methods in predicting in-role performance. This is an important area of enquiry as despite it being common practice to use both types of methods within a selection system, there is currently no evidence that this approach translates into increased predictive validity of the selection system as a whole, over that achieved by the use of a single selection method. In this preliminary study, the majority of the range of scores achieved by successful applicants to the UK Foundation Programme provided a unique opportunity to address both of these areas of enquiry. Sampling targeted high (>80th percentile) and low (<20th percentile) scorers on the SJT. Supervisors rated 391 trainees' in-role performance, and incidence of remedial action was collected. SJT and academic performance scores correlated with supervisor ratings (r = .31 and .28, respectively). The relationship was stronger between the SJT and in-role performance for the low scoring group (r = .33, high scoring group r = .11), and between academic performance and in-role performance for the high scoring group (r = .29, low scoring group r = .11). Trainees with low SJT scores were almost five times more likely to receive remedial action. Results indicate that an SJT for entry into trainee physicians' first role in clinical practice has good predictive validity of supervisor-rated performance and incidence of remedial action. In addition, an SJT and a measure of academic performance appeared to be complementary to each other. These initial findings suggest that SJTs may be more predictive at the lower end of a scoring distribution, and academic attainment more predictive at the higher end.