CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores

Efficiency, Safety, Quality, and Empathy: Balancing Competing Perioperative Challenges in the Older Adults

Although baby boomer generation accounts for a little more than 15% of the US population, the cohort represents a disproportionate percentage of patients undergoing surgery. As this group continues to age, a multitude of challenges have arisen in health care regarding the safest and most effective means of providing anesthesia services to these patients. Many older adults patients may be exquisitely sensitive to the effects of anesthesia and surgery and may experience cognitive and physical decline before, during, or after hospital admission. In this review article, the authors briefly examine the physiologic processes underlying aging and explore steps necessary to deliver safe, empathetic care.

Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach

Introduction

Situational judgment tests (SJT) are commonly used in admissions to measure skills associated with professionalism. Although open-response SJTs have shown strong psychometric properties, assessors' personal beliefs, experiences, and cultural backgrounds may influence how they perceive, organize and evaluate information within test takers' diverse responses. Additionally, SJT research typically focuses on reliability and predictive validity, whereas the construct validity of open response SJTs remains underexplored. This mixed methods study aims to address this gap by exploring the construct-(ir)relevant factors that may impact assessors' evaluation of professionalism in open response SJTs.

Methods

For this study, we used data from Casper, an open response SJT commonly used in professional program admissions. In Study I, a quantitative content analysis was conducted on 160 responses to identify factors which were significant predictors of low and high scores. Correlation coefficients and logistic regression models were used to evaluate the relationship between each factor and response scores. In Study II, think-aloud activities were conducted with 23 Casper assessors to directly observe how they evaluated responses. All interviews were transcribed verbatim, which were then thematically analyzed using an inductive coding technique.

Results

Results from both the content analyses and think-aloud activities revealed that several construct relevant factors influenced scores. Scores were impacted by the extent to which test takers demonstrated the competencies probed for by the SJT, engaged with the context of the presented ethical dilemma, provided in-depth justifications for their response, considered various perspectives relevant to the presented dilemma, and provided creative solutions or insightful arguments for the suggested approach. Mixed results were found with respect to construct irrelevant factors, such as the flow, cohesion, and kinds of phrases used in the response.

Conclusion

This mixed methods study contributes to the construct validity of SJTs by investigating construct relevant and irrelevant factors that may impact assessors' evaluation of open responses. The findings of this study provide evidence that open-response SJTs are valid approaches to measure professional competencies more broadly, both in terms of what test takers focus on in their responses, as well as in terms of how they construct their responses.

Associations between an open-response situational judgment test and performance on OSCEs and fieldwork: implications for admissions decisions and matriculant diversity in an occupational therapy program

BACKGROUND

Casper, an online open-response situational judgement test that assesses social intelligence and professionalism [1], is used in admissions to health professions programs.

METHOD

This study (1) explored the incremental validity of Casper over grade point average (GPA) for predicting student performance on objective structured clinical examinations (OSCEs) and fieldwork placements within an occupational therapy program, (2) examined optimal weighting of Casper in GPA in admissions decisions using non-linear optimization and regression tree analysis to find the weights associated with the highest average competency scores, and (3) investigated whether Casper could be used to impact the diversity of a cohort selected for admission to the program.

RESULTS

Multiple regression analysis results indicate that Casper improves the prediction of OSCE and fieldwork performance over and above GPA (change in Adj. R2 = 3.2%). Non-linear optimization and regression tree analysis indicate the optimal weights of GPA and Casper for predicting performance across fieldwork placements are 0.16 and 0.84, respectively. Furthermore, the findings suggest that students with a slightly lower GPA (e.g., 3.5-3.6) could be successful in the program as assessed by fieldwork, which is considered to be the strongest indicator of success as an entry-level clinician. In terms of diversity, no statistically significant differences were found between those actually admitted and those who would have been admitted using Casper.

CONCLUSION

These results constitute preliminary validity evidence supporting the integration of Casper into applicant selection in an occupational therapy graduate program.

Predicting undergraduate OSCE performance using traditional and construct-driven situational judgment tests at admission

Situational Judgement Tests (SJTs) are popular to screen for social skills during undergraduate medical admission as they have been shown to predict relevant study outcomes. Two different types of SJTs can be distinguished: Traditional SJTs, which measure general effective behavior, and construct-driven SJTs which are designed to measure specific constructs. To date, there has been no comparison of the predictive validity of these two types of SJTs in medical admission. With the present research, we examine whether the HAM-SJT, a traditional SJT, and the CD-SJT, a construct-driven SJT with an agentic and a communal scale, administered during undergraduate medical admission can predict OSCE (i.e., objective structured clinical examination) performance in a low-stakes (nLS = 159) and a high-stakes (nHS = 160) sample of medical students. Results showed a moderate positive relation between the communal scale of the CD-SJT and performance in OSCE stations with trained patients in the high-stakes sample (r =.20, p =.009). This SJT had also an incremental value in predicting the OSCE performance above and beyond GPA (i.e., grade point average), a science test (i.e., HAM-Nat), and gender (ß = 0.18, 95% CI [0.03; 0.33], p =.020). That is, individuals who chose more communal behavioral responses in the SJT were rated more favorably in interactions with trained patients in the OSCE. A comparable correlation coefficient was observed for the HAM-SJT when controlling for range restriction due to admission (rraw = 0.14 vs. rcontrolled = 0.20). Our research provides a first indication for the predictive validity of construct-driven SJTs in high-stakes undergraduate medical admission.

From admissions to licensure: education data associations from a multi-centre undergraduate medical education collaboration

This paper reports the findings of a Canada based multi-institutional study designed to investigate the relationships between admissions criteria, in-program assessments, and performance on licensing exams. The study's objective is to provide valuable insights for improving educational practices across different institutions. Data were gathered from six medical schools: McMaster University, the Northern Ontario School of Medicine University, Queen's University, University of Ottawa, University of Toronto, and Western University. The dataset includes graduates who undertook the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1) between 2015 and 2017. The data were categorized into five distinct sections: demographic information as well as four matrices: admissions, course performance, objective structured clinical examination (OSCE), and clerkship performance. Common and unique variables were identified through an extensive consensus-building process. Hierarchical linear regression and a manual stepwise variable selection approach were used for analysis. Analyses were performed on data set encompassing graduates of all six medical schools as well as on individual data sets from each school. For the combined data set the final model estimated 32% of the variance in performance on licensing exams, highlighting variables such as Age at Admission, Sex, Biomedical Knowledge, the first post-clerkship OSCE, and a clerkship theta score. Individual school analysis explained 41-60% of the variance in MCCQE1 outcomes, with comparable variables to the analysis from of the combined data set identified as significant independent variables. Therefore, strongly emphasising the need for variety of high-quality assessment on the educational continuum. This study underscores the importance of sharing data to enable educational insights. This study also had its challenges when it came to the access and aggregation of data. As such we advocate for the establishment of a common framework for multi-institutional educational research, facilitating studies and evaluations across diverse institutions. This study demonstrates the scientific potential of collaborative data analysis in enhancing educational outcomes. It offers a deeper understanding of the factors influencing performance on licensure exams and emphasizes the need for addressing data gaps to advance multi-institutional research for educational improvements.

Measuring personal characteristics in applicants to German medical schools: Piloting an online Situational Judgement Test with an open-ended response format

Objectives

Situational Judgement Tests (SJT) are a cost-efficient method for the assessment of personal characteristics (e.g., empathy, professionalism, ethical thinking) in medical school admission. Recently, complex open-ended response format SJTs have become more feasible to conduct. However, research on their applicability to a German context is missing. This pilot study tests the acceptability, reliability, subgroup differences, and validity of an online SJT with open-ended response format developed in Canada ("Casper").

Methods

German medical school applicants and students from Hamburg were invited to take Casper in 2020 and 2021. The test consisted of 12 video- and text-based scenarios, each followed by three open-ended questions. Participants subsequently evaluated their test experience in an online survey. Data on sociodemographic characteristics, other admission criteria (Abitur, TMS, HAM-Nat, HAM-SJT) and study success (OSCE) was available in a central research database (stav).

Results

The full sample consisted of 582 participants. Test-takers' global perception of Casper was positive. Internal consistency was satisfactory in both years (α=0.73; 0.82) while interrater agreement was moderate (ICC(1,2)=0.54). Participants who were female (d=0.37) or did not have a migration background (d=0.40) received higher scores. Casper scores correlated with HAM-SJT (r=.18) but not with OSCE communication stations performance. The test was also related to Abitur grades (r=-.15), the TMS (r=.18), and HAM-Nat logical reasoning scores (r=.23).

Conclusion

This study provides positive evidence for the acceptability, internal consistency, and convergent validity of Casper. The selection and training of raters as well as the scenario content require further observation and adjustments to a German context to improve interrater reliability and predictive validity.

A survey of medical school aspirant perceptions of an unexpected lottery-facilitated admissions adaptation

INTRODUCTION

Due to the COVID-19 pandemic, the Undergraduate Medical Doctor (MD) Programme at McMaster University (Hamilton, Canada) was unable to run in-person medical school interviews in March 2020, prompting an alternate solution that maximised admission opportunities for Indigenous applicants, prioritised admission for those rated most highly in the interview determination process, and allocated subsequent offers via lottery.

METHODS

A short survey was administered to applicants who had been offered an admissions interview and were subsequently impacted by the admissions adaptations. The survey elicited perceptions of the adaptation through Likert scale ratings and free-text responses. Survey data were analysed via a sequential (quantitative to qualitative) mixed-methods design.

RESULTS

196 of 552 potential participants completed the survey. Across quantitative and qualitative analyses, respondents reported that the adaptation had a negative impact on their professional development and personal life. Ratings of negative perception were greater for those who did not receive an offer than for those who accepted or declined an offer. Free text responses emphasised considerable criticism for the lottery portion of the adaptation and displeasure that efforts made in constructing applications were less relevant than anticipated.

DISCUSSION

The negative responses to this unexpected change highlight the profound upstream impact admission policies have on the preapplication behaviours of aspiring medical students. The outcomes support a refined understanding of the value candidates place on the interview in appraising their own suitability for a career as a physician.

Lessons learned from a non-traditional application process: Integrating user experiences to inform admission application design

PURPOSE

The research aim was to gather applicant expectations and perspectives with a user-centered approach during a pilot application process for a new dental school. The goal was to inform future admissions practices at our institution and for programs broadly.

METHODS

The High Point University admission team developed an electronic application external to current centralized services that was launched from May to August 2022. Participants who completed the application were requested to complete an electronic survey of closed- and open-ended questions about the factors influencing their application decisions, feedback about how to optimize the experience, and their overall satisfaction using a Net Promoter Score.

RESULTS

A total of 303 applicants (79% response rate) completed the survey about their application experience. The most influential reason for applying to the program was the optional Dental Admissions Test requirement (43%). Participants frequently referenced a need for a more human-centered process focused on the candidate rather than their numerical values and that the process should be smooth, straight-forward, short, easy, and simple. Participants requested that experiences should be weighed the most in the application process followed by letters of recommendation. Most applicants were "promoters" (68%) for the program with a Net Promoter Score of 62. Moreover, 32% and 45% rated the application process as the best or better application experience compared to other programs.

CONCLUSION

Programs can benefit from collecting applicant's (i.e., users) opinions and feedback about their expectations and experiences to inform more user-centered application processes.

Evaluating non-cognitive skills in medical school applicants

Medical school candidates must have both the cognitive and professional competencies required to become good physicians. In this commentary, we outline the evidence and outcomes associated with the implementation of these selection methodologies and evaluate their ability to assess non-cognitive skills.

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

OBJECTIVE

The primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness.

METHODS

An SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics.

RESULTS

Thirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics.

CONCLUSION

An SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.

Recommendations for Improvement of Equity, Diversity, and Inclusion in the CaRMs Selection Process

Equity, diversity and inclusion (EDI) in the medical field is crucial for meeting the healthcare needs of a progressively diverse society. A diverse physician workforce enables culturally sensitive care, promotes health equity, and enhances the comprehension of the various needs and viewpoints of patients, ultimately resulting in more effective treatments and improved patient outcomes. However, despite the recognized benefits of diversity in the medical field, certain specialties, such as Radiology, have struggled to achieve adequate equity, diversity and inclusion, which results in a discrepancy in the demographics of Canadian radiologists and the patients we serve. In this review, we propose strategies from a committee within the Canadian Association of Radiologists (CAR) EDI working group to improve EDI in the CaRMS selection process. By adopting these strategies, residency programs can foster a more diverse and inclusive environment that is better positioned to address the health needs of a progressively diverse patient population, leading to improved patient outcomes, greater patient satisfaction, and advancements in medical innovation.

Efficiency, Safety, Quality, and Empathy: Balancing Competing Perioperative Challenges in the Elderly

Although baby boomer generation accounts for a little more than 15% of the US population, the cohort represents a disproportionate percentage of patients undergoing surgery. As this group continues to age, a multitude of challenges have arisen in health care regarding the safest and most effective means of providing anesthesia services to these patients. Many elderly patients may be exquisitely sensitive to the effects of anesthesia and surgery and may experience cognitive and physical decline before, during, or after hospital admission. In this review article, the authors briefly examine the physiologic processes underlying aging and explore steps necessary to deliver safe, empathetic care.

Impact of patterns of language use and socio-economic status on a constructed response Situational Judgment Test (SJT)

The study explored the impacts of patterns of language use (PLU) and socio-economic status (SES) on Casper, a constructed-response situational judgment test (SJT). 10,266 applicants from two U.S. medical schools were grouped into self-reported balanced bilinguals, unbalanced bilinguals, English monolinguals, and English as a Second Language (ESL) students. A multicomponent SES composite was used to assess the degree of socioeconomic disadvantage (DSD). Results from a hierarchical regression analysis showed that after accounting for demographic variables, both PLU and DED were significant factors on applicants' Casper performance. Bilingualism was associated with better Casper performance compared to English monolinguals and ESL students. No significant effect of speaking English as a native language was found on applicants' Casper performance. English monolinguals and ESL students performed equivalently on Casper. Finally, high DSD was associated with better Casper performance than low DSD, and the impact of DSD on Casper held the same across all four language groups. These findings provide evidence that socio-cultural factors, such as PLU and DSD have important impacts on SJT performance. Further research is needed to understand the role of differences in language construction across socio-cultural factors on constructed-response SJT performance.

The CASPER preparation program innovation: increasing self-perceived competence and confidence of underrepresented applicants on the novel CASPER Snapshot and CanMEDS roles

BACKGROUND

Underrepresented Minorities in Medicine (URMMs) may face financial and social limitations when matriculating into medical schools. Performance on situational judgment tests such as Computer-based Assessment for Sampling Personal Characteristics (CASPER) can be enhanced by coaching and mentorship. The CASPER Preparation Program (CPP) coaches URMMs to prepare for the CASPER test. During the coronavirus 2019 pandemic (COVID-19), CPP implemented novel curricula on the CASPER Snapshot and CanMEDS roles.

METHODS

Pre and post-program questionnaires were completed by the students, which assessed their: 1) confidence in understanding the CanMEDS roles, and 2) perceived confidence in performing well and their familiarity and preparedness with the CASPER Snapshot. With a second post-program questionnaire, participants' scores on the CASPER test as well as medical school application outcome were also assessed.

RESULTS

Participants reported a significant increase in the URMMs' knowledge, self-perceived competency to complete the CASPER Snapshot, and their anxiety significantly decreased. The level of confidence in understanding CanMEDS roles for a career in healthcare increased as well. The majority (91%) agreed that the feedback received from tutors was adequate and the virtual component of the program was beneficial during COVID-19. 51% of students scored in the highest quartile on the CASPER test and 35% received an offer of admission from CASPER-requiring medical schools.

CONCLUSION

Pathway coaching programs have the potential to increase confidence and familiarity amongst URMMs for the CASPER tests and CanMEDS roles. Similar programs should be developed with the aim to increase the chances of URMMs matriculating into medical schools.

Evaluating situational judgment test use and diversity in admissions at a southern US medical school

INTRODUCTION

Situational judgment tests have been adopted by medical schools to assess decision-making and ethical characteristics of applicants. These tests are hypothesized to positively affect diversity in admissions by serving as a noncognitive metric of evaluation. The purpose of this study was to evaluate the performance of the Computer-based Assessment for Sampling Personal Characteristics (CASPer) scores in relation to admissions interview evaluations.

METHODS

This was a cohort study of applicants interviewing at a public school of medicine in the southeastern United States in 2018 and 2019. Applicants took the CASPer test prior to their interview day. In-person interviews consisted of a traditional interview and multiple-mini-interview (MMI) stations. Between subjects, analyses were used to compare scores from traditional interviews, MMIs, and CASPer across race, ethnicity, and gender.

RESULTS

1,237 applicants were interviewed (2018: n = 608; 2019: n = 629). Fifty-seven percent identified as female. Self-identified race/ethnicity included 758 White, 118 Black or African-American, 296 Asian, 20 Native American or Alaskan Native, 1 Native Hawaiian or Other Pacific Islander, and 44 No response; 87 applicants identified as Hispanic. Black or African-American, Native American or Alaskan Native, and Hispanic applicants had significantly lower CASPer scores than other applicants. Statistically significant differences in CASPer percentiles were identified for gender and race; however, between subjects, comparisons were not significant.

CONCLUSIONS

The CASPer test showed disparate scores across racial and ethnic groups in this cohort study and may not contribute to minimizing bias in medical school admissions.

Validity of constructed-response situational judgment tests in training programs for the health professions: A systematic review and meta-analysis protocol

BACKGROUND

Situational judgments tests have been increasingly used to help training programs for the health professions incorporate professionalism attributes into their admissions process. While such tests have strong psychometric properties for testing professional attributes and are feasible to implement in high-volume, high-stakes selection, little is known about constructed-response situational judgment tests and their validity.

METHODS

We will conduct a systematic review of primary published or unpublished studies reporting on the association between scores on constructed-response situational judgment tests and scores on other tests that measure personal, interpersonal, or professional attributes in training programs for the health professions. In addition to searching electronic databases, we will contact academics and researchers and undertake backward and forward searching. Two reviewers will independently screen the papers and decide on their inclusion, first based on the titles and abstracts of all citations, and then according to the full texts. Data extraction will be done independently by two reviewers using a data extraction form to chart study details and key findings. Studies will be assessed for the risk of bias and quality by two reviewers using the "Quality In Prognosis Studies" tool. To synthesize evidence, we will test the statistical heterogeneity and conduct a psychometric meta-analysis using a random-effects model. If adequate data are available, we will explore whether the meta-analytic correlation varies across different subgroups (e.g., race, gender).

DISCUSSION

The findings of this study will inform best practices for admission and selection of applicants for training programs for the health professions and encourage further research on constructed-response situational judgment tests, in particular their validity.

TRIAL REGISTRATION

The protocol for this systematic review has been registered in PROSPERO [CRD42022314561]. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022314561.

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study

PURPOSE

To determine whether overall situational judgment test (SJT) scores are associated with programs' clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods.

METHOD

In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019-2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods.

RESULTS

Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings ( r = .21 and .14, P < .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear ( P = .005) and a .22 SD increase at year-end ( P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear.

CONCLUSIONS

Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.

Online student culture as site for negotiating assessment in medical education

Classic studies of medical education have examined how professional socialization reproduces the prevailing professional culture, as well as how students actively negotiate their place in educational processes. However, sociological research has not re-examined student culture in light of structural transformations in medical education, such as the introduction of new assessment types and their use as modes of commensuration. In this paper, we examine data from two studies of online forums where medical trainees and applicants to medical school discuss their experiences preparing for tests of professional skills, including judgment, empathy, and communication. Examining how medical students talk about these tests on such forums allows us to understand the meaning-making processes at work as students negotiate the commensuration processes such tests enable. We examine how these negotiations take place in online forums, where participants confront common challenges, form common perspectives, and share common solutions, all hallmarks of student culture. Through qualitative analysis, we find that online communities are spaces where students grapple with these new forms of commensuration, interrogate the standards and quantifications that underlie them, and collectively negotiate how to approach these assessments. Using the case of online forum communities, our findings advance past work on student culture in medical sociology by theorizing student culture as an extra-organizational phenomenon that spans multiple career stages. In so doing, we highlight the importance of online forum data for studying social processes.

Exploring professional identity in rehabilitation professions: a scoping review

Professional identity is believed to foster self-confidence and resilience in health care professionals. While literature exists describing professional identity in medicine, the relevance of this evidence to rehabilitation professionals (occupational therapy (OT), physical therapy (PT) and speech-language pathology (S-LP)) is limited due to differences between professions in decision-making authority (patient care), professional autonomy and understanding of their scope of practice. The objective was to determine the extent, range and nature of the literature on professional identity/professional identity formation in rehabilitation professionals. Findings from the scoping review based on Arksey and O'Malley's methodological framework are presented. A search was conducted on MEDLINE (Ovid), Embase (Ovid), AMED, CINAHL, and ProQuest Dissertations and Theses from 1996 to October 2020 for empirical and conceptual studies on OT, PT, and S-LP clinicians or students. Of 4983 retrieved records, 53 papers were selected for data extraction. Data were organised into themes for professional identity/professional identity formation: conceptual descriptors (dynamic state, multiple identities); influences (person, professional education/environments, profession-at-large). Findings are consistent with the professional identity literature in medicine. However, they point to gaps for further empirical inquiry in the role of symbols and rituals in the professional identity/professional identity formation of rehabilitation professionals, profession-specific differences between OT, PT and S-LP, and influences related to the profession-at-large on the professional identity/professional identity formation of rehabilitation professionals. These findings may help to inform professional education programs and health care and professional systems in developing resources to support professional identity formation of rehabilitation professionals.

Licensing exams in Canada: a closer look at the validity of the MCCQE Part II

The Medical Council of Canada Qualifying Exam (MCCQE) Part II aims to protect societal interests through examining recently graduated physicians using clinical scenarios with standardized patients. This position paper debates the role of the MCCQE Part II in the national licensing of physicians in Canada by focusing on the consequential validity evidence of this exam and considering future directions through discussing contemporary developments in high stakes examinations. Specifically, this paper compares both MCCQE Part I and Part II in their ability to predict future practice patterns of physicians and generalizability across specialties. In weighing up the evidence this paper considers commonly used counterarguments as well as the financial implications of this exam for both the candidates and the MCC. Finally, it concludes by providing recommendations for future licensing of physicians in Canada. The available consequential validity evidence for MCCQE Part II is limited. Though still limited, MCCQE Part I has more robust evidence that it is a better predictor of future practice patterns compared to with Part II. Combined with a lack of evidence that national licensing examinations lead to graduation of substandard doctors or an improvement of care, and the shift away from assessment of learning towards assessment for learning, the maximum impact of the MCC on safeguarding public's interests will lie in working closely with residency programs and specialty colleges to facilitate a robust assessment program of essential competencies and clinical skills during residency training and specialty certification.

Associations between admissions factors and the need for remediation

This study examines the way in which student characteristics and pre-admissions measures are statistically associated with the likelihood a student will require remediation for academic and professionalism offenses. We anchor our inquiry within Irby and Hamstra's (2016) conceptual framework of constructs of professionalism. Data from five graduating cohorts (2014-2018) from McMaster University (Hamilton, Canada) (N = 1,021) were retroactively collected and analyzed using traditional and multinominal logistic regression analyses. The relationship among student characteristics, pre-admissions variables, and referral for potential remediation both by occurrence (yes/no) as well as type (academic/professional/no referral) were examined separately. Findings indicate that gender (OR = 0.519, 95% CI 0.326-0.827, p < 0.01) and undergraduate grade point average (GPA) (OR = 0.245, 95% CI 0.070-0.855, p < 0.05) were significantly associated with instances of referral for potential professionalism and academic remediation, respectively. Women were less likely than men to require remediation for professionalism (OR = 0.332, 95% CI 0.174-0.602, p < 0.001). Undergraduate GPAs (OR = 0.826, 95% CI 0.021-0.539, p < 0.01) were significantly associated with remediation for academic reasons. Lower undergraduate GPAs were associated with a higher likelihood of remediation. These findings point to the admissions variables that are associated with instances that prompt referral for potential remediation. Where associations are not significant, we consider the application of different conceptualizations of professionalism across periods of admissions and training. We encourage those involved in applicant selection and student remediation to emphasize the importance of the interactions that occur between personal and contextual factors to influence learner behaviour and professional identity formation.

What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills

Once medical students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. These traits are significantly more difficult to assess than cognitive performance, creating a potential gap in measurement. Despite its promise, competency-based medical education (CBME) has yet to bridge this gap, partly due to a lack of well-defined noncognitive observable behaviors that assessors and educators can use in formative and summative assessment. As a result, typical undergraduate to graduate medical education handovers stress standardized test scores, and program directors trust little of the remaining information they receive, sometimes turning to third-party companies to better describe potential residency candidates. The authors have created a list of noncognitive attributes, with associated definitions and noncognitive skills-called observable practice activities (OPAs)-written for learners across the continuum to help educators collect assessment data that can be turned into valuable information. OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, entrustable professional activities, or competencies, to create learning trajectories for formative and summative decisions. Medical schools and graduate medical education programs could adapt these OPAs or determine ways to create new ones specific to their own contexts. Once OPAs are created, programs will have to find effective ways to assess them, interpret the data, determine consequence validity, and communicate information to learners and institutions. The authors discuss the need for culture change surrounding assessment-even for the adoption of behavior-based tools such as OPAs-including grounding the work in a growth mindset and the broad underpinnings of CBME. Ultimately, improving assessment of noncognitive capacity should benefit learners, schools, programs, and most importantly, patients.

Adapting the Admissions Interview During COVID-19: A Comparison of In-Person and Video-Based Interview Validity Evidence

COVID-19 physical distancing limited many medical schools' abilities to conduct in-person interviews for the 2020 admissions cycle. The University of Toronto (U of T) Temerty Faculty of Medicine was already in the midst of its interview process, with two-thirds of applicants having completed the in-person modified personal interview (MPI). As the university and surrounding region were shut down, the shift was made in the middle of the application cycle to a semisynchronous video-based MPI interview (vMPI) approach. U of T undertook the development, deployment, and evaluation of the 2 approaches mid-admissions cycle. Existing resources and tools were used to create a tailored interview process with the assistance of applicants. The vMPI was similar in content and process to the MPI: a 4-station interview with each station mapped to attributes relevant to medical school success. Instead of live interviews, applicants recorded 5-minute responses to questions for each station using their own hardware. These responses were later assessed by raters asynchronously. Out of 627 applicants, 232 applicants completed the vMPI. Validity evidence was generated for the vMPI and compared with the MPI on the internal structure, relationship to other variables, and consequential validity, including applicant and interviewer acceptability. Overall, the vMPI demonstrated similar reliability and factor structure to the MPI. As with the MPI, applicant performance was predicted by nonacademic screening tools but not academic measures. Applicants' acceptance of the vMPI was positive. Most interviewers found the vMPI to be acceptable and reported confidence in their ratings. Continuing physical distancing concerns will require multiple options for admissions committees to select medical students. The vMPI is an example of a customized approach that schools can implement and may have advantages for selection beyond the COVID-19 pandemic. Future evaluation will examine additional validity evidence for the tool.

Evaluating the Whole Applicant: Use of Situational Judgment Testing and Personality Testing to Address Disparities in Resident Selection

PURPOSE OF REVIEW

Urology program directors are faced with increasing numbers of applications annually, making holistic review of each candidate progressively more difficult. Efforts to streamline evaluation using traditional cognitive metrics have fallen short as these do not predict overall resident performance. Situational judgment tests (SJTs) and personality assessment tools (PATs) have been used in business and industry for decades to evaluate candidates and measure non-cognitive attributes that better predict subsequent performance. The purpose of this review is to describe what these assessments are and the current literature on the use of these metrics in medical education.

RECENT FINDINGS

SJTs relative to PATs have more original research. Data suggests that SJTs decrease bias, increase diversity, and may be predictive of performance in residency. PATs are also emerging with data to support use with ability to assess fit to program and certain traits identified more consistently among high-performing residents and correlation to performance on ACGME milestones. PATs may be more coachable than SJTs. SJTs and PATs are emerging as techniques to supplement the current resident application review process. Early evidence supports their use in undergraduate medical education as does some early preliminary results in graduate medical education.

Reliability and validity of new online selection tests for midwifery students

BACKGROUND AND OBJECTIVE

Newly created midwifery bachelor programmes attract an increasing number of applicants. Hence, educators have moved away from traditional non-standardised selection methods and started developing new tools for the admission of students. Researchers have expressed concerns around the validity of such selection measures and their ability to identify the appropriate candidates. The objective of this study was to develop and implement selection procedures for a new midwifery Bachelor of Science degree in Hamburg, Germany, and assess their psychometric properties.

DESIGN

This is a mixed-methods longitudinal study on the reliability, validity and acceptability of newly developed selection criteria.

SETTING

The newly established midwifery bachelor of science in Hamburg, Germany.

PARTICIPANTS

Upon completion of their online application, all midwifery applicants were invited by email to participate in our research project and thereby informed of its purpose and voluntary nature of their participation. The total number of candidates who took the test was 366, of which 309 agreed to take part in this study. Sixty-five applicants were eventually admitted to the midwifery Bachelor programme, of which 59 were included in the study.

MEASUREMENTS AND FINDINGS

We developed two tests for cognitive ability (HAM-Mid I and II) and adapted one test for personal and professional characteristics (Casper - Computer-based Assessment for Sampling Personal Characteristics). Due to the Covid pandemic, we implemented the tests online instead of on site as initially planned. HAM-Mid I had the lowest (Cronbach's Alpha = 0.38) whereas Casper had the highest internal consistency (Cronbach's Alpha = 0.77) of all three tests. Age, previous academic achievement and native tongue were significantly associated with applicants' performance on HAM-Mid II and Casper admission tests. HAM-Mid II was associated with students' performance in the first year midwifery exam (r = 0.31, p < 0.05). Overall evaluation of HAM-Mid II and Casper was good while HAM-Mid I received a more negative feedback.

KEY CONCLUSIONS

Evidence on the reliability and predictive validity of the newly developed tests suggests that the applied admission criteria are appropriate for an objective selection of bachelor midwifery students.

IMPLICATIONS FOR PRACTICE

The interests of university and clinical stakeholders of a midwifery bachelor degree can be represented through the development of tests for the assessment of cognitive ability and personal and professional characteristics.

Implicit and inferred: on the philosophical positions informing assessment science

Assessment practices have been increasingly informed by a range of philosophical positions. While generally beneficial, the addition of options can lead to misalignment in the philosophical assumptions associated with different features of assessment (e.g., the nature of constructs and competence, ways of assessing, validation approaches). Such incompatibility can threaten the quality and defensibility of researchers' claims, especially when left implicit. We investigated how authors state and use their philosophical positions when designing and reporting on performance-based assessments (PBA) of intrinsic roles, as well as the (in)compatibility of assumptions across assessment features. Using a representative sample of studies examining PBA of intrinsic roles, we used qualitative content analysis to extract data on how authors enacted their philosophical positions across three key assessment features: (1) construct conceptualizations, (2) assessment activities, and (3) validation methods. We also examined patterns in philosophical positioning across features and studies. In reviewing 32 papers from established peer-reviewed journals, we found (a) authors rarely reported their philosophical positions, meaning underlying assumptions could only be inferred; (b) authors approached features of assessment in variable ways that could be informed by or associated with different philosophical assumptions; (c) we experienced uncertainty in determining (in)compatibility of philosophical assumptions across features. Authors' philosophical positions were often vague or absent in the selected contemporary assessment literature. Leaving such details implicit may lead to misinterpretation by knowledge users wishing to implement, build on, or evaluate the work. As such, assessing claims, quality and defensibility, may increasingly depend more on who is interpreting, rather than what is being interpreted.

Assessing professionalism in health profession degree programs: A scoping review

BACKGROUND

To identify and classify methods for assessing professionalism across health profession degree programs and identify gaps in the literature regarding types of assessments.

METHODS

The authors conducted a scoping review of articles published from database inception through 24 January 2020. Included articles described an assessment approach for professionalism in health profession degree programs available in full-text in the English language. Articles were classified based on profession, timing of assessment, feedback type, assessment type, professionalism dimension, and Barr's modified Kirkpatrick hierarchy.

RESULTS

Authors classified 277 articles meeting inclusion criteria. Most articles were from medical education (62.5%) conducted during didactic (62.1%) or experiential/clinical curriculum (49.8%). Few articles (15.5%) described longitudinal assessment. Feedback type was formative (32.2%) or summative (35%), with only 8.3% using both. Assessment types frequently reported included self-administered rating scales (30%), reflections (18.8%), observed clinical encounters (17.3%), and knowledge-based tests (13.4%). Ethical practice principles (65%) and effective interactions with patients (48.4%) were the most frequently assessed dimensions of professionalism. Authors observed balanced distribution among Barr's modified Kirkpatrick model at levels of reaction (38.3%), modification of perceptions and attitudes (33.6%), acquisition of knowledge and skills (39%), and behavioral change (36.1%).

IMPLICATIONS

The classification scheme identified in current literature on professionalism assessment does not align with International Ottawa Conference Working Group on the Assessment of Professionalism recommendations. Gaps identified were limited description of professionalism assessment during admissions, infrequent longitudinal assessment, limited use of methods for both formative and summative assessment, and limited reports of assessments applicable to interprofessional education settings.

Holistic Admission: What Is It? How Successful Has It Been in Nursing, and What Are the Possibilities?

Holistic admission review (HAR) practices have been identified as a means to increase diversity in the nursing profession. HAR practices consider the whole applicant in a dignified and respectful manner, while focusing on valued experiences, attributes, and metrics needed for the nursing profession. Nursing has lagged behind other healthcare professions in implementing holistic admissions and with support from national organizations are starting to move onward. Interviewing applicants appears to be a process embraced not only by healthcare service organizations but now by schools (and colleges) of nursing (SONs) that have implemented HAR practices. There is much to gain and not much to lose by using holistic admissions. In addition, increasing diversity in nursing is the only way one can determine if nursing diversity improves health disparities. Faculty need to consider all that their SONs, the profession, and those they serve can gain by implementing holistic admissions.

Personal qualities needed by undergraduate nursing students for a successful work integrated learning (WIL) experience

BACKGROUND

International research has shown that during work integrated learning (WIL) undergraduate nursing students may experience stress, self-doubt, and coping issues, however personal qualities these students bring to their WIL journey or those needed to succeed during WIL remain unknown.

OBJECTIVE

To describe the views of students midway through their Bachelor of Nursing program regarding personal qualities they bring to and need for WIL success.

DESIGN

This article reports on a descriptive study that used an electronic questionnaire to collect the data.

SETTINGS AND PARTICIPANTS

Bachelor of nursing students (n = 421), who were studying fulltime or part time equivalent, at a regional university in Queensland Australia, were invited to complete the anonymous electronic questionnaire that was developed by the research team following an extensive review of literature. Inclusion criteria required participants to have completed at least one WIL experience and be midway through their three-year Bachelor of Nursing program.

METHODS

Data were collected during January and February 2017. The electronic questionnaire consisted of demographic, forced choice questions, and open-ended questions. This paper presents the qualitative analysis of responses to two of the open-ended questions that focused on personal strengths and work integrated learning experiences. Data were analysed using the inductive content analysis approach.

RESULTS

Six personal qualities were identified including: enthusiasm to succeed, commitment to learning, self-motivation, confidence, effective communication skills, and compassion.

CONCLUSIONS

Pre-entry screening of applicants to bachelor level nursing programs and the delivery of learning opportunities that encourages both reflection and debriefing may assist in describing personal qualities needed to succeed in WIL.

Academic redlining in medicine

Despite concerted efforts over the past decade to increase diversity in U.S. medical schools, persistent applicant and enrollment gaps remain for students from underrepresented racial and economic backgrounds. To understand these gaps, we propose a new theory of 'academic redlining' as a widespread practice in medical schools that systematically excludes students from underrepresented backgrounds from entry into medicine through the nearly universal use of Medical College Admissions Test (MCAT) cutoff scores. In this paper, we provide evidence that academic redlining via the MCAT disenfranchises students from underrepresented backgrounds prior to and during the admissions process due to structural racism, and describe the three core mechanisms that cause medical schools to engage in academic redlining: (1) the pursuit of institutional prestige, (2) market competition and pressure, and (3) market bands. Given the persistent lack of diversity in medicine-which contributes to devastating health care disparities-as medical schools redouble their commitments to diversity, equity, and inclusion, structural alternatives within medical schools' admissions and education practices are offered to curtail the practice of academic redlining in medical school admissions and medicine.

New Advances in Physician Assistant Admissions: The History of Situational Judgement Tests and the Development of CASPer

ABSTRACT

During the first 50 years of the physician assistant (PA) profession, admission to PA programs was based primarily on cognitive domains such as academic performance and standardized test scores. Many programs also considered other measurable factors, including patient care experience, community service, and extracurricular activities. While interviews were frequently conducted by the programs, it was not until the applicants had been "pre-screened" for the previously identified qualifications. As the PA profession continued to expand, PA programs began to realize that potentially strong applicants were being excluded from the admissions process because of this emphasis on mostly cognitive factors. In an attempt to reduce this disparity, PA programs have begun to expand their assessment of applicants to include assessment of noncognitive characteristics. This article outlines the history surrounding this change in the approach to admissions in medical education, reviews the development of situational judgement tests and other tools used to assess these noncognitive characteristics, and explores the relationship of these noncognitive characteristics to the development of program-defined competencies.

Bridging the gap: improving CASPer test confidence and competency for underrepresented minorities in medicine through interactive peer-assisted learning

BACKGROUND

The Computer-based Assessment for Sampling Personal Characteristics (CASPer) is a situational judgement test (SJT) adopted by medical schools to assess applicants' interpersonal skills. CASPer applicants must compose their responses to ethical dilemmas, thereby highlighting the applicant's rationale for ethical decision-making. Minority applicants usually lack access to a network of individuals who can offer guidance and expertise on ethical decision-making. As such, this study investigated the impact of a CASPer coaching program designed for minority applicants.

METHODS

A free online intervention was designed to help minority applicants prepare for the CASPer test. The program consisted of 35 learners and three medical student tutors. Important attributes of the 4-week program included free access to a medical ethics book, feedback provision to in-class and homework student responses, and facilitation of a mock CASPer. Course feedback was collected. Additionally, a pre and post-program survey was administered to assess learners' competence and confidence surrounding CASPer test-taking.

RESULTS

Our pre and post-program survey showed significant student improvement in familiarity with the test, increased competence, confidence and preparedness, as well as reduced anxiety (p < 0.05).

CONCLUSIONS

Through peer-to-peer teaching and access to medical student mentors, our program addresses socioeconomic barriers that several minority applicants face when applying to medical school.

The New USMLE Step 1 Paradigm: An Opportunity to Cultivate Diversity of Excellence

The February 2020 announcement that United States Medical Licensing Examination (USMLE) Step 1 results will be reported as pass/fail instead of numerical scores has been controversial. Step 1 scores have played a key role in residency selection, including screening for interviews. Although Step 1 scores are viewed as an objective criterion, they have been shown to disadvantage female and underrepresented minority applicants, cause student anxiety and financial burden, and affect student well-being. Furthermore, Step 1 scores incompletely predict applicants' overall residency performance. With this paradigm shift in Step 1 score reporting, residency programs will have fewer objective, standardized metrics for selection decisions, which may lead to greater emphasis on USMLE Step 2 Clinical Knowledge scores or yield unintended consequences, including shifting weight to metrics such as medical school reputation.Yet, greater breadth in residency selection metrics will better serve both applicants and programs. Some students excel in coursework, others in research or leadership. All factors should be recognized, and broader metrics should be implemented to promote and recognize diversity of excellence. Given the need for metrics for residency selection as well as for a more holistic approach to evaluating residency applicants, assessment during medical school should be revisited and made more meaningful. Another opportunity may involve use of situational judgment tests to predict professionalism and performance on other competencies. It will be important to evaluate the impact of the new Step 1 paradigm and related initiatives going forward. Residency application overload must also be addressed.

COVID-19: A Driver for Disruptive Innovation of the Emergency Medicine Residency Application Process

The coronavirus disease (COVID-19) pandemic has had a significant impact on undergraduate medical education with limitation of patient care activities and disruption to medical licensing examinations. In an effort to promote both safety and equity, the emergency medicine (EM) community has recommended no away rotations for EM applicants and entirely virtual interviews during this year's residency application cycle. These changes affect the components of the EM residency application most highly regarded by program directors - Standardized Letters of Evaluation from EM rotations, board scores, and interactions during the interview. The Council of Residency Directors in Emergency Medicine Application Process Improvement Committee suggests solutions not only for the upcoming year but also to address longstanding difficulties within the process, encouraging residency programs to leverage these challenges as an opportunity for disruptive innovation.

Social justice as an expression of caring through holistic admissions in a nursing program: A proposed conceptual model

AIM

This paper presents HAR as an expression of caring to create social justice within nursing and achieve a workforce that is representative of those being served.

BACKGROUND

The lack of diversity within the health professions has been expressly linked in the literature to health disparities among underrepresented and marginalized groups.

RECOMMENDATIONS

Recognizing the value of diversity within healthcare has been the impetus for some health profession programs to use holistic admissions review (HAR) in the assessment and evaluation of applicant suitability. While current HAR recommendations in nursing broaden the lens on which criteria should be used to determine applicant suitability beyond standard academic metrics, existing models do not examine applicants' caring capacity.

CONCLUSION

Given caring is the essence of nursing, the authors offer a guiding framework to supplement the American Association of Colleges of Nursing criteria for HAR and a model by which nursing applicants are evaluated on their capacity to care.

Development and Validation of a Situational Judgement Test to Assess Professionalism

Objective. The purpose of this study was to develop, pilot, and validate a situational judgement test (SJT) to assess professionalism in Doctor of Pharmacy (PharmD) students. Methods. Test specifications were developed and faculty members were educated on best practices in item writing for SJTs. The faculty members then developed 75 pilot scenarios. From those, two versions of the SJT, each containing 25 scenarios, were created. The pilot population for the SJT was student pharmacists in their third professional year, just prior to starting their advanced pharmacy practice experiences. The students completed the two versions of the test on different days, approximately 48 hours apart, with 50 minutes allowed to complete each. Subsequently, students completed a questionnaire regarding the SJT at the conclusion of the second test. Results. Version 1 of the SJT was completed by 228 students, and version 2 was completed by 225 students. Mean scores were 390 (SD=20, range 318-429) and 342 (SD=21, range 263-387) on test versions 1 and 2, respectively. The reliability of the tests was appropriate (test version 1, α=0.77; test version 2, α=0.79). Students felt that the content of the tests was realistic with respect to pharmacy practice (90.1%), and that the tests gave them an opportunity to reflect on how to approach challenging situations (82.6%). Conclusion. We developed a reliable SJT to assess professionalism in PharmD students. Future research should focus on creating a personalized learning plan for students who do not meet minimum performance standards on this SJT.

The economic costs of selecting medical students: An Australian case study

CONTEXT

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained.

METHODS

A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated.

RESULTS

A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%).

CONCLUSIONS

The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

The influence of income on medical school admissions in Canada: a retrospective cohort study

BACKGROUND

The socioeconomic status of applicants to Canadian medical schools has been understudied in the past two decades. Institutional efforts have been made to address the lack of socioeconomic diversity across Canada during this time. We investigated the income characteristics of medical school applicants, as well as the relationship between applicant income and offer of admission, to characterize the current state of socioeconomic diversity in medical admissions.

METHODS

We conducted a retrospective cohort study on 26,120 applicants at one Ontario medical school from 2013 to 2018. Characteristics of applicants who were offered admission were compared to the general population and applicants not offered admission. Regression analyses were used to investigate the association between median total neighborhood income and successful admission.

RESULTS

The median total neighborhood income for medical school applicants was $98,816, which was approximately $28,480 higher than the Canadian general population. Those not admitted to the medical school had a median total neighborhood income of $98,304 compared to $105,984 for those offered admission (p < 0.001). This trend was seen in every province and territory in Canada. Median total neighborhood income was a predictor of an offer of admission; applicants in the >75th percentile income group had 54% increased odds of being offered admission when compared to applicants in the <25th percentile in our unadjusted model. Income was not significant in our adjusted models but showed that the income medians drastically shifted between pre-interview and post-interview periods, from $98,816 to $104,960 (p < 0.001).

CONCLUSION

Medical school applicants are from higher economic strata compared to the general population. Despite already representing a high economic stratum, a higher median total neighborhood income relative to other applicants was associated with an offer of admission.

Opening the black box of selection

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

McMaster at 50: lessons learned from five decades of PBL

Although educators frequently act as if curricula are as standardized as drug doses (300 mg of PBL t.i.d.), such is not the case. As a case in point, at its inception, Problem Based Learning was hailed as a major curriculum innovation, with the promise of enormous gains in learning outcomes. Very quickly, ecclesiastical debates arose as what was true PBL and what was "modified PBL". Ironically, systematic reviews conducted fairly early in its evolution showed that the gains in learning outcome from PBL were neither large nor uniform (Vernon and Blake in Acad Med 68:550-563, 1993), and the most consistent finding was greater student satisfaction. In this paper, we review five decades of experience with the first PBL curriculum at McMaster. We point out how the curriculum has evolved, both theoretically and practically, in response to external influences, based both on empirical evidence and practical demands. We describe these changes in four broad domains-theoretical rationale, the curriculum, assessment and admissions.

Addressing the Diversity-Validity Dilemma Using Situational Judgment Tests

PURPOSE

To examine the magnitudes of score differences across different demographic groups for three academic (grade point average [GPA], old Medical College Admission Test [MCAT], and MCAT 2015) and one nonacademic (situational judgment test [SJT]) screening measures and one nonacademic (multiple mini-interview [MMI]) interview measure (analysis 1), and the demographic implications of including an SJT in the screening stage for the pool of applicants who are invited to interview (analysis 2).

METHOD

The authors ran the analyses using data from New York Medical College School of Medicine applicants from the 2015-2016 admissions cycle. For analysis 1, effect sizes (Cohen d) were calculated for GPA, old MCAT, MCAT 2015, CASPer (an online SJT), and MMI. Comparisons were made across gender, race, ethnicity (African American, Hispanic/Latino), and socioeconomic status (SES). For analysis 2, a series of simulations were conducted to estimate the number of underrepresented in medicine (UIM) applicants who would have been invited to interview with different weightings of GPA, MCAT, and CASPer scores.

RESULTS

A total of 9,096 applicants were included in analysis 1. Group differences were significantly smaller or reversed for CASPer and MMI compared with the academic assessments (MCAT, GPA) across nearly all demographic variables/indicators. The simulations suggested that a higher weighting of CASPer may help increase gender, racial, and ethnic diversity in the interview pool; results for low-SES applicants were mixed.

CONCLUSIONS

The inclusion of an SJT in the admissions process has the potential to widen access to medical education for a number of UIM groups.

A Reflection Upon the Impact of Early 21st-Century Technological Innovations on Medical School Admissions

The authors describe influences associated with the incorporation of modern technologies into medical school admissions processes. Their purpose is not to critique or support specific technologies but, rather, to prompt reflection on the evolution that is afoot. Technology is now integral to the administration of multiple admissions tools, including the Medical College Admission Test, situational judgment tests, and standardized video interviews. Consequently, today's admissions landscape is transforming into an online, globally interconnected marketplace for health professions admissions tools. Academic capitalism and distance-based technologies combine to enable global marketing and dissemination of admissions tests beyond the national jurisdictions in which they are designed. As predicted by disruptive business theory, they are becoming key drivers of transformative change. The seeds of technological disruption are present now rather than something to be wary of in the future. The authors reflect on this transformation and the need for tailoring test modifications to address issues of medical student diversity and social responsibility. They comment on the online assessment of applicants' personal competencies and the potential detriments if this method were to replace admissions methods involving human contact, thanks to the ease with which institutions can implement them without cost to themselves and without adequate consideration of measurement utility or contextual appropriateness. The authors advocate for socially responsible academic capitalism within this interconnected admissions marketplace: Attending to today's transformative challenges may inform how health professions education responds to tomorrow's admissions technologies and, in turn, how tomorrow's health professionals respond to their patients' needs.

[Selection of medical students : Measurement of cognitive abilities and psychosocial competencies]

The German Constitutional Court is currently reviewing whether the actual study admission process in medicine is compatible with the constitutional right of freedom of profession, since applicants without an excellent GPA usually have to wait for seven years. If the admission system is changed, politicians would like to increase the influence of psychosocial criteria on selection as specified by the Masterplan Medizinstudium 2020.What experiences have been made with the actual selection procedures? How could Situational Judgement Tests contribute to the validity of future selection procedures to German medical schools?High school GPA is the best predictor of study performance, but is more and more under discussion due to the lack of comparability between states and schools and the growing number of applicants with top grades. Aptitude and knowledge tests, especially in the natural sciences, show incremental validity in predicting study performance. The measurement of psychosocial competencies with traditional interviews shows rather low reliability and validity. The more reliable multiple mini-interviews are superior in predicting practical study performance. Situational judgement tests (SJTs) used abroad are regarded as reliable and valid; the correlation of a German SJT piloted in Hamburg with the multiple mini-interview is cautiously encouraging.A model proposed by the Medizinischer Fakultätentag and the Bundesvertretung der Medizinstudierenden considers these results. Student selection is proposed to be based on a combination of high school GPA (40%) and a cognitive test (40%) as well as an SJT (10%) and job experience (10%). Furthermore, the faculties still have the option to carry out specific selection procedures.

Embracing standardisation and contextualisation in medical education

CONTEXT

The tensions that emerge between the universal and the local in a global world require continuous negotiation. However, in medical education, standardization and contextual diversity tend to operate as separate philosophies, with little attention to the interplay between them.

METHODS

The authors synthesise the literature related to the intersections and resulting tensions between standardization and contextual diversity in medical education. In doing so, the authors analyze the interplay between these competing concepts in two domains of medical education (admissions and competency-based medical education), and provide concrete examples drawn from the literature.

RESULTS

Standardization offers many rewards: its common articulations and assumptions promote patient safety, foster continuous quality improvement, and enable the spread of best practices. Standardization may also contribute to greater fairness, equity, reliability and validity in high stakes processes, and can provide stakeholders, including the public, with tangible reassurance and a sense of the stable and timeless. At the same time, contextual variation in medical education can afford myriad learning opportunities, and it can improve alignment between training and local workforce needs. The inevitable diversity of contexts for learning and practice renders any absolute standardization of programs, experiences, or outcomes an impossibility.

CONCLUSIONS

The authors propose a number of ways to examine the interplay of contextual diversity and standardization and suggest three ways to move beyond an either/or stance. In reconciling the laudable goals of standardization and the realities of the innumerable contexts in which we train and deliver care, we are better positioned to design and deliver a medical education system that is globally responsible and locally engaged.

Selection into medicine: the predictive validity of an outcome-based procedure

BACKGROUND

Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program.

METHODS

We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students.

RESULTS

Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs.

CONCLUSIONS

The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.

Lessons learned from 15 years of non-grades-based selection for medical school

CONTEXT

Thirty years ago, it was suggested in the Edinburgh Declaration that medical school applicants should be selected not only on academic, but also on non-academic, attributes. The main rationale behind extending medical school selection procedures with the evaluation of (non-academic) personal qualities is that this will lead to the selection of students who will perform better as a doctor than those who are selected on the basis of academic measures only. A second rationale is the expectation that this will lead to a representative health workforce as a result of reduced adverse impact. The aims of this paper are (i) to describe what can be learned about the use of selection criteria other than grades from over 15 years of Dutch experience and (ii) to summarise current knowledge on the issue of adverse impact in relation to non-grades-based selection.

METHODS

A narrative review was undertaken of the (published) evidence that has resulted from non-grades-based school-specific selection procedures in the Netherlands and from recent explorations of the effect of the use of non-grades-based selection criteria on student diversity.

RESULTS

The Dutch evidence is grouped into five key themes: the effect of participation in voluntary selection procedures, the assessment of pre-university extracurricular activities, the use of work samples, Dutch experiences with situational judgement tests and the effects of changing circumstances. This is followed by several lessons learned for medical schools that aim to increase their student diversity.

CONCLUSION

Over the last 30 years, important steps towards reliable and valid methods for measuring non-academic abilities have been taken. The current paper describes several lessons that can be learned from the steps taken in the Dutch context. The importance of sharing evidence gathered around the globe and building on this evidence to reach our goal of predicting who will be a good doctor is acknowledged.