Evaluating candidate reactions to selection practices using organisational justice theory

Reframing selection as a learning experience: Insights from a residency selection assessment

PURPOSE

Selection for postgraduate medical training is high-stakes and complex. The learning value of assessments for selection has, thus far, been underexplored, limiting their uptake as potentially meaningful learning experiences. The aim of this study was to explore the learning value residency applicants derive from an intelligence, personality, motivation and competency selection assessment and what factors influence the experienced learning value.

METHODS

In Autumn 2020 and Spring 2021, we conducted individual semi-structured interviews with sixteen applicants for pediatric residency training. Selection outcomes were unknown at the time of the interview. Interviews were transcribed verbatim and thematically analyzed.

RESULTS

Participants reported that the assessment was valuable in fostering self-reflection and self-awareness, embracing self-acceptance, pursuing development goals, assessing professional fit, and harnessing motivational drivers in work. The experienced learning value was influenced by applicants' ability to interpret its results, their focus on the high-stakes selection process and concerns regarding the acceptability and credibility of the selection tool.

CONCLUSIONS

While the selection assessment showed learning potential, its learning value was impeded by a preoccupation with the high-stakes nature of the selection procedure. Intentional integration of the selection assessment in the learning curriculum may play a pivotal role in realizing its learning potential.

Improving selection procedures in health professions education from the applicant perspective: an interview study

INTRODUCTION

Applicant perceptions of selection impact motivation and performance during selection, and student diversity. However, in-depth insight into which values underly these perceptions is lacking, creating challenges for aligning selection procedures with applicant perceptions. This qualitative interview study aimed to identify values applicants believe should underlie selection, and how, according to applicants, these values should be used to make specific improvements to selection procedures in undergraduate health professions education (HPE).

METHODS

Thirty-one applicants to five undergraduate HPE programs in the Netherlands participated in semi-structured interviews using Appreciative Inquiry, an approach that focuses on what goes well to create vision for improvement, to guide the interviews. Transcriptions were analyzed using thematic analysis, adopting a constructivist approach.

RESULTS

Applicants' values related to the aims of selection, the content of selection, and the treatment of applicants. Applicants believed that selection procedures should aim to identify students who best fit the training and profession, and generate diverse student populations to fulfill societal needs. According to applicants, the content of selection should be relevant for the curriculum and profession, assess a comprehensive set of attributes, be of high quality, allow applicants to show who they are, and be adapted to applicants' current developmental state. Regarding treatment, applicants believed that selection should be a two-way process that fosters reflection on study choice, be transparent about what applicants can expect, safeguard applicants' well-being, treat all applicants equally, and employ an equitable approach by taking personal circumstances into account. Applicants mentioned specific improvements regarding each value.

DISCUSSION

Applicants' values offer novel insights into what they consider important preconditions for the design of selection procedures. Their suggested improvements can support selection committees in better meeting applicants' needs.

Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes

BACKGROUND

Computer-based assessment for sampling personal characteristics (Casper), an online situational judgement test, is a broad measure of personal and professional qualities. We examined the impact of Casper in the residency selection process on professionalism concerns, learning interventions and resource utilization at an institution.

METHODS

In 2022, admissions data and information in the files of residents in difficulty (over three years pre- and post- Casper implementation) was used to determine the number of residents in difficulty, CanMEDS roles requiring a learning intervention, types of learning interventions (informal learning plans vs. formal remediation or probation), and impact on the utilization of institutional resource (costs and time). Professionalism concerns were mapped to the 4I domains of a professionalism framework, and their severity was considered in mild, moderate, and major categories. Descriptive statistics and between group comparisons were used for quantitative data.

RESULTS

In the pre- and post- Casper cohorts the number of residents in difficulty (16 vs. 15) and the number of learning interventions (18 vs. 16) were similar. Professionalism concerns as an outcome measure decreased by 35% from 12/16 to 6/15 (p < 0.05), were reduced in all 4I domains (involvement, integrity, interaction, introspection) and in their severity. Formal learning interventions (15 vs. 5) and informal learning plans (3 vs. 11) were significantly different in the pre- and post-Casper cohorts respectively (p < 0.05). This reduction in formal learning interventions was associated with a 96% reduction in costs f(rom hundreds to tens of thousands of dollars and a reduction in time for learning interventions (from years to months).

CONCLUSIONS

Justifiable from multiple stakeholder perspectives, use of an SJT (Casper) improves a clinical performance measure (professionalism concerns) and permits the institution to redirect its limited resources (cost savings and time) to enhance institutional endeavors and improve learner well-being and quality of programs.

Patient Engagement in Medical Trainee Selection: A Scoping Review

PURPOSE

The stakes of medical trainee selection are high, making it ironic and somewhat paradoxical that patients and the public often get little say in selection practices. The authors sought to undertake a knowledge synthesis to uncover what is known about patient engagement across the medical trainee selection continuum.

METHOD

The authors conducted a scoping review aimed at exploring the current state of practice and research on patient engagement in medical trainee selection in 2017-2021. MeSH headings and keywords were used to capture patient, community, and standardized patient engagement in selection processes across multiple health professions. The authors employed broad inclusion criteria and iteratively refined the corpus, ultimately, limiting study selection to those reporting engagement of actual patients in selection within medicine, but maintaining a broad focus on any patient contributions across the entire selection continuum. The Cambridge Framework was adapted and used to organize the included studies.

RESULTS

In total, 2,858 abstracts were reviewed, and ultimately, 28 papers were included in the final corpus. The included studies were global but nascent. Most of the literature on this topic appears in the form of individual projects advocating for patient engagement in selection rather than cohesive programs with empirical exploration of patient engagement in selection. Job analysis methodology was particularly prominent for incorporating the patient voice into identifying competencies of relevance to selection. Direct patient engagement in early selection activities allowed the patient voice to assist candidates in determining their fit for medicine.

CONCLUSIONS

Patient engagement has not been made a specific focus of study in its own right, leading the authors to encourage researchers to turn their lens more directly on patient engagement to explore how it complements the professional voice in medical trainee selection.

Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework

Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.

National recruitment scheme for pre-registration pharmacist training in England and Wales: a mixed method evaluation of experiences of applicant pharmacy students

BACKGROUND

A national pre-registration pharmacist training recruitment scheme, which replaces local recruitment models, was introduced in England and Wales in 2017. The national recruitment system allows pharmacy students to apply for the 52 weeks training programmes (mandatory requirement for registration as a pharmacist), through a single application system prior to undertaking a nationally administered assessment. This study aimed to explore experiences of pharmacy students on the national recruitment scheme, particularly their views on the selection methodology, application process, and offer outcomes.

METHODS

This mixed method study involved a) an online survey of all (approximate n = 2800) year 4 (final year of MPharm degree) pharmacy students in England and Wales and b) a qualitative focus group with four students. The study population was eligible to participate in the 2017/18 national recruitment scheme. Survey respondents were invited to participate in a focus group. Quantitative data were analysed using descriptive and inferential analysis. Qualitative data were analysed using the framework technique. Participation was voluntary. Ethical approval from University of Birmingham was obtained.

RESULTS

A total of 307 completed surveys were returned (approximate response rate 11%). Respondents were generally satisfied with the application process and commended the fairness of the selection methodology and convenience in allowing them to apply to multiple training providers. Most survey respondents (n = 181, 72.9%) were either satisfied or highly satisfied with the training programme they were offered based on their assessment performances. Three themes and eight sub-themes obtained from the analysis of over 200 open comments data from the survey and transcript of a focus group with four participants. Results suggested the need to widen the timeframe available for applicants to shortlist their preferred employers, improve the method of programme listing in the application system, and consideration of prior achievements including academic performances and placement experiences to be included in the selection methodology.

CONCLUSIONS

Experiences of pharmacy students on the national recruitment scheme suggest that respondents considered the selection methodology to be fair. Student engagement and satisfaction with the recruitment system can be maximised through improved listing of employers and widening the timescales for students to shortlist their preferred employers during application process. Inclusion of University achievements in the selection methodology will require consideration of evidence based approaches. Low response rate limits generalisation of findings.

Evaluation of a Situational Judgement Test to Develop Non-Academic Skills in Pharmacy Students

Objective. To design, implement, and psychometrically evaluate a situational judgement test (SJT) to use as a formative assessment of pharmacy students' non-academic skills in an Australian-based university. Methods. An SJT was developed using a previously validated design process including involvement of subject matter experts. The first phase included design of a blueprint through stakeholder consultation and the development of bespoke attribute definitions and a tool specification. Following on from this, SJT items were developed through subject matter expert interviews and in-depth review process. Results. Students (702) from four different cohorts (first through fourth years) of a Bachelor of Pharmacy degree program completed the situational judgment test. Data from 648 students was eligible for inclusion in the analysis. The SJT demonstrated good reliability, appropriateness for use (difficulty and quality), fairness, and face validity. The variability in students' scores suggested that the SJT may be a useful metric to identify students most in need of additional support. Conclusion. Evaluation of the SJT demonstrated that the tool was valid, reliable, fair and appropriate to use as a formative assessment. Through implementing an SJT such as this, pharmacy students are provided the opportunity to receive feedback on their non-academic skills and consider how to approach challenging or unfamiliar situations before entering the profession.

Student perceptions of a sequential objective structured clinical examination

BACKGROUND

Well-designed objective structured clinical examinations (OSCEs) are reliable and valid, but expensive. The sequential OSCE (sOSCE) aims to balance robustness with affordability. In a sOSCE all students undertake a screening test (Day 1), with 'failing' or 'borderline' candidates sitting a second examination (Day 2). Current research has focused on psychometric properties of the sOSCE. Our aim was to examine the acceptability of the sOSCE, by identifying students' views.

METHODS

Final-year students at one Scottish university completed a questionnaire after Day 1 of a sOSCE. Analysis included descriptive statistics and thematic analysis.

RESULTS

A total of 107 out of 154 students (69.5%) responded. Most respondents strongly agreed/agreed that they: felt stressed about the sOSCE (98.1%); would feel like a failure if taking Day 2 (89.7%); and that Day 2 seems the same as a re-sit (78.5%). However, 61.7% agreed that fewer exams days was a positive aspect of the sOSCE. Open comments indicated feelings of increased stress, anxiety and frustration associated with the sOSCE.

CONCLUSIONS

Novelty or 'fear of the unknown' regarding the sOSCE seemed to be associated with negative attitudes. Further studies are required to explore student views of the sOSCE at less pressured times in the curriculum.

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection

BACKGROUND

This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.

METHODS

Medical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.

RESULTS

Applicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants - but not ethnic majority applicants - showed greater preference for should-do than would-do instructions. Additionally, first-generation university students - but not non-first-generation university students - were more favourable of rating formats than of pick-one formats.

CONCLUSIONS

Findings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

A systematic review of stakeholder views of selection methods for medical schools admission

BACKGROUND

The purpose of this paper is to systematically review the literature with respect to stakeholder views of selection methods for medical school admissions.

METHODS

An electronic search of nine databases was conducted between January 2000-July 2014. Two reviewers independently assessed all titles (n = 1017) and retained abstracts (n = 233) for relevance. Methodological quality of quantitative papers was assessed using the MERSQI instrument. The overall quality of evidence in this field was low. Evidence was synthesised in a narrative review.

RESULTS

Applicants support interviews, and multiple mini interviews (MMIs). There is emerging evidence that situational judgement tests (SJTs) and selection centres (SCs) are also well regarded, but aptitude tests less so. Selectors endorse the use of interviews in general and in particular MMIs judging them to be fair, relevant and appropriate, with emerging evidence of similarly positive reactions to SCs. Aptitude tests and academic records were valued in decisions of whom to call to interview. Medical students prefer interviews based selection to cognitive aptitude tests. They are unconvinced about the transparency and veracity of written applications. Perceptions of organisational justice, which describe views of fairness in organisational processes, appear to be highly influential on stakeholders' views of the acceptability of selection methods. In particular procedural justice (perceived fairness of selection tools in terms of job relevance and characteristics of the test) and distributive justice (perceived fairness of selection outcomes in terms of equal opportunity and equity), appear to be important considerations when deciding on acceptability of selection methods. There were significant gaps with respect to both key stakeholder groups and the range of selection tools assessed.

CONCLUSIONS

Notwithstanding the observed limitations in the quality of research in this field, there appears to be broad concordance of views on the various selection methods, across the diverse stakeholders groups. This review highlights the need for better standards, more appropriate methodologies and for broadening the scope of stakeholder research.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

Using a situational judgement test for selection into dental core training: a preliminary analysis

Objective and setting This paper describes the evaluation of a pilot situational judgement test (SJT) for selection into UK Dental Core Training (DCT). The SJT's psychometric properties, group differences based on gender and ethnicity, and candidate reactions were assessed.Methods The SJT targets four non-academic attributes important for success in DCT. Data were collected alongside live selection processes from five Health Education England local teams in the UK (N = 386). Candidates completed the pilot SJT and an evaluation questionnaire to examine their reactions to the test.Results SJT scores were relatively normally distributed and showed acceptable levels of internal reliability (α = 0.68). Difficulty level and partial correlations between scenarios and SJT total score were in the expected ranges (64.61% to 90.03% and r = 0.06 to 0.41, respectively). No group differences were found for gender, and group differences between White and BME candidates were minimal. Most candidates perceived the SJT as relevant to the target role, appropriate and fair.Conclusions This study demonstrated the potential suitability of an SJT for use in DCT selection. Future research should replicate these preliminary findings in other cohorts, and assess the predictive validity of the SJT for predicting key training and practice-based outcomes.

The construct validity of HPAT-Ireland for the selection of medical students: unresolved issues and future research implications

Aptitude tests are widely used in selection. However, despite certain advantages their use remains controversial. This paper aims to critically appraise five sources of evidence for the construct validity of the Health Professions Admission Test (HPAT)-Ireland, an aptitude test used for selecting undergraduate medical students. The objectives are to identify gaps in the evidence, draw comparisons with other aptitude tests and outline future research directions. Our appraisal of the literature found that stakeholder feedback indicates that there is reasonable evidence for test content validity for two of the three sections of HPAT-Ireland. By contrast the Non-Verbal Reasoning section is widely criticised as having limited relevance to medical school performance and future clinical practice. In terms of concurrent validity there is a significant small to medium, negative correlation with school exit examinations, but not consistently so across all studies (r = -0.18, -0.28, 0.017). Likewise predictive validity studies vary, from negative to moderate strength correlations with examination performance during early years at medical school. Five studies indicate that HPAT-Ireland is supported in principle by the majority of stakeholders. While one consequence of its introduction is that successful applicants are now coming from more diverse academic backgrounds, there is no evidence that the socio-economic background of medical school entrants has been altered significantly. Negative perceptions of unfairness relating to gender, coaching and socio-economics remain. The evidence to date suggests that while there are slight gender differences, initially favouring males, these vary year on year. In conclusion, the attitudes towards, and performance of, HPAT-Ireland is not unlike that of other aptitude tests widely used internationally. The main justifications for its introduction have been achieved, in that Ireland no longer relies exclusively on a single measure of academic record for selection to medical school. However a number of areas require further research and exploration.

Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

Situational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used.

Recruiting for values in healthcare: a preliminary review of the evidence

Displaying compassion, benevolence and respect, and preserving the dignity of patients are important for any healthcare professional to ensure the provision of high quality care and patient outcomes. This paper presents a structured search and thematic review of the research evidence relating to values-based recruitment within healthcare. Several different databases, journals and government reports were searched to retrieve studies relating to values-based recruitment published between 1998 and 2013, both in healthcare settings and other occupational contexts. There is limited published research related to values-based recruitment directly, so the available theoretical context of values is explored alongside an analysis of the impact of value congruence. The implications for the design of selection methods to measure values is explored beyond the scope of the initial literature search. Research suggests some selection methods may be appropriate for values-based recruitment, such as situational judgment tests (SJTs), structured interviews and multiple-mini interviews (MMIs). Personality tests were also identified as having the potential to compliment other methods (e.g. structured interviews), as part of a values-based recruitment agenda. Methods including personal statements, references and unstructured/'traditional' interviews were identified as inappropriate for values-based recruitment. Practical implications are discussed in the context of values-based recruitment in the healthcare context. Theoretical implications of our findings imply that prosocial implicit trait policies, which could be measured by selection tools such as SJTs and MMIs, may be linked to individuals' values via the behaviours individuals consider to be effective in given situations. Further research is required to state this conclusively however, and methods for values-based recruitment represent an exciting and relatively unchartered territory for further research.

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

Why use SJTs? Traditionally, selection into medical education professions has focused primarily upon academic ability alone. This approach has been questioned more recently, as although academic attainment predicts performance early in training, research shows it has less predictive power for demonstrating competence in postgraduate clinical practice. Such evidence, coupled with an increasing focus on individuals working in healthcare roles displaying the core values of compassionate care, benevolence and respect, illustrates that individuals should be selected on attributes other than academic ability alone. Moreover, there are mounting calls to widen access to medicine, to ensure that selection methods do not unfairly disadvantage individuals from specific groups (e.g. regarding ethnicity or socio-economic status), so that the future workforce adequately represents society as a whole. These drivers necessitate a method of assessment that allows individuals to be selected on important non-academic attributes that are desirable in healthcare professionals, in a fair, reliable and valid way. What are SJTs? Situational judgement tests (SJTs) are tests used to assess individuals' reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance. From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others, expense). Are SJTs reliable, valid and fair? Several studies, including good quality meta-analytic and longitudinal research, consistently show that SJTs used in many different occupational groups are reliable and valid. Although there is over 40 years of research evidence available on SJTs, it is only within the past 10 years that SJTs have been used for recruitment into medicine. Specifically, evidence consistently shows that SJTs used in medical selection have good reliability, and predict performance across a range of medical professions, including performance in general practice, in early years (foundation training as a junior doctor) and for medical school admissions. In addition, SJTs have been found to have significant added value (incremental validity) over and above other selection methods such as knowledge tests, measures of cognitive ability, personality tests and application forms. Regarding differential attainment, generally SJTs have been found to have lower adverse impact compared to other selection methods, such as cognitive ability tests. SJTs have the benefit of being appropriate both for use in selection where candidates are novices (i.e. have no prior role experience or knowledge such as in medical school admissions) as well as settings where candidates have substantial job knowledge and specific experience (as in postgraduate recruitment for more senior roles). An SJT specification (e.g. scenario content, response instructions and format) may differ depending on the level of job knowledge required. Research consistently shows that SJTs are usually found to be positively received by candidates compared to other selection tests such as cognitive ability and personality tests. Practically, SJTs are difficult to design effectively, and significant expertise is required to build a reliable and valid SJT. Once designed however, SJTs are cost efficient to administer to large numbers of candidates compared to other tests of non-academic attributes (e.g. personal statements, structured interviews), as they are standardised and can be computer-delivered and machine-marked.

The social validity of a national assessment centre for selection into general practice training

BACKGROUND

Internationally, recruiting the best candidates is central to the success of postgraduate training programs and the quality of the medical workforce. So far there has been little theoretically informed research considering selection systems from the perspective of the candidates. We explored candidates' perception of the fairness of a National Assessment Centre (NAC) approach for selection into Australian general practice training, where candidates were assessed by a Multiple Mini Interview (MMI) and a written Situational Judgment Test (SJT), for suitability to undertake general practice (GP) training.

METHODS

In 2013, 1,930 medical practitioners, who were eligible to work in Australia attended one of 14 NACs in each of 5 states and 2 territories. A survey was distributed to each candidate at the conclusion of their assessment, which included open-ended questions aimed at eliciting candidates' perceived benefits and challenges of the selection process. A framework analysis was informed by the theoretical lens of Social Validity Theory.

RESULTS

Qualitative data was available from 46% (n = 886/1,930) of candidates, who found the NAC experience fair and informative for their training and career goals, but wanted to be provided with more information in preparation. Candidates valued being able to communicate their skills during the MMI, but found some difficulty in interpreting the questions. A significant minority had concerns that a lack of relevant GP experience may inhibit their performance. Candidates also expressed concerns about the time limits within the written paper, particularly if English was not their first language. They also expressed a desire for formative feedback during the interview process.

CONCLUSION

During any job selection process, not only is the organisation assessing the candidates, but the candidates are also assessing the organisation. However, a focus on the candidate experience throughout an organisation's selection process may provide benefits to both candidates and the organisation, regardless of whether or not candidates secured the job. Social Validity Theory is a useful addition to the methods for demonstrating the reasonableness of any selection system.

The fairness, predictive validity and acceptability of multiple mini interview in an internationally diverse student population--a mixed methods study

BACKGROUND

International medical students, those attending medical school outside of their country of citizenship, account for a growing proportion of medical undergraduates worldwide. This study aimed to establish the fairness, predictive validity and acceptability of Multiple Mini Interview (MMI) in an internationally diverse student population.

METHODS

This was an explanatory sequential, mixed methods study. All students in First Year Medicine, National University of Ireland Galway 2012 were eligible to sit a previously validated 10 station MMI. Quantitative data comprised: demographics, selection tool scores and First Year Assessment scores. Qualitative data comprised separate focus groups with MMI Assessors, EU and Non-EU students.

RESULTS

109 students participated (45% of class). Of this 41.3% (n = 45) were Non-EU and 35.8% (n = 39) did not have English as first language. Age, gender and socioeconomic class did not impact on MMI scores. Non-EU students and those for whom English was not a first language achieved significantly lower scores on MMI than their EU and English speaking counterparts (difference in mean 11.9% and 12.2% respectively, P<0.001). MMI score was associated with English language proficiency (IELTS) (r = 0.5, P<0.01). Correlations emerged between First Year results and IELTS (r = 0.44; p = 0.006; n = 38) and EU school exit exam (r = 0.52; p<0.001; n = 56). MMI predicted EU student OSCE performance (r = 0.27; p = 0.03; n = 64). In the analysis of focus group data two overarching themes emerged: Authenticity and Cultural Awareness. MMI was considered a highly authentic assessment that offered a deeper understanding of the applicant than traditional tools, with an immediate relevance to clinical practice. Cultural specificity of some stations and English language proficiency were seen to disadvantage international students. Recommendations included cultural awareness training for MMI assessors, designing and piloting culturally neutral stations, lengthening station duration and providing high quality advance information to candidates.

CONCLUSION

MMI is a welcome addition to assessment armamentarium for selection, particularly with regard to stakeholder acceptability. Understanding the mediating and moderating influences for differences in performance of international candidates is essential to ensure that MMI complies with the metrics of good assessment practice and principles of both distributive and procedural justice for all applicants, irrespective of nationality and cultural background.

Views of doctors of varying disciplines on HPAT-Ireland as a selection tool for medicine

BACKGROUND

Selection tools for medicine must achieve political validity and enjoy stakeholder acceptability. This qualitative study aimed to establish the perspectives of doctors, from various clinical specialities, on HPAT-Ireland, a new selection tool for undergraduate medical students.

METHODS

Fifteen doctors participated over three iterative cycles of recruitment, interviewing and analysis. Prior to interview, participants sat a practice HPAT-Ireland test. HPAT-Ireland has three sections: (1) Logical reasoning/problem solving; (2) Interpersonal understanding and (3): Non-verbal reasoning.

SUMMARY OF RESULTS

Three themes emerged: job relatedness; utility of HPAT-Ireland and diversity. Sections 1 and 2 were considered very job related however Section 3 was widely criticised for lacking clinical relevance. Doctors did not think that the test would reliably predict future performance. However, one-third felt it was acceptable as a selection tool in conjunction with academic record. Those who found it unacceptable were influenced by its perceived narrow focus, limited job relatedness, potential for socioeconomic bias, impact on gender and potential for negative influence on student diversity.

CONCLUSIONS

A selection tool that does not enjoy the confidence of the medical profession is unlikely to achieve political validity and may ultimately fail, regardless of other objective measures of its effectiveness such as predictive validity.

Developing national selection processes for entry into postgraduate specialty training: the case of trauma and orthopedics in the United Kingdom

A large body of international research has been done investigating best practice selection across many occupational groups, but there is relatively little research on developing selection methodology for entry to postgraduate training. Hitherto, various selection processes have been used, some of which relied heavily on patronage. Developments in Medical Education, including curriculum design, formal work-based place assessment, alongside the introduction of MMC (Modernizing Medical Careers) paved the way for significant change. Trauma and orthopedics in England is the last surgical specialty to adopt National Selection and did so first in 2013. This paper sets out that journey, the implementation of National selection in 2013, what has been learned, and our goals for the future.

A competency based selection procedure for Dutch postgraduate GP training: a pilot study on validity and reliability

Abstract Background: Historically, semi-structured interviews (SSI) have been the core of the Dutch selection for postgraduate general practice (GP) training. This paper describes a pilot study on a newly designed competency-based selection procedure that assesses whether candidates have the competencies that are required to complete GP training.

OBJECTIVES

The objective was to explore reliability and validity aspects of the instruments developed.

METHODS

The new selection procedure comprising the National GP Knowledge Test (LHK), a situational judgement tests (SJT), a patterned behaviour descriptive interview (PBDI) and a simulated encounter (SIM) was piloted alongside the current procedure. Forty-seven candidates volunteered in both procedures. Admission decision was based on the results of the current procedure.

RESULTS

Study participants did hardly differ from the other candidates. The mean scores of the candidates on the LHK and SJT were 21.9 % (SD 8.7) and 83.8% (SD 3.1), respectively. The mean self-reported competency scores (PBDI) were higher than the observed competencies (SIM): 3.7(SD 0.5) and 2.9(SD 0.6), respectively. Content-related competencies showed low correlations with one another when measured with different instruments, whereas more diverse competencies measured by a single instrument showed strong to moderate correlations. Moreover, a moderate correlation between LHK and SJT was found. The internal consistencies (intraclass correlation, ICC) of LHK and SJT were poor while the ICC of PBDI and SIM showed acceptable levels of reliability.

CONCLUSION

Findings on content validity and reliability of these new instruments are promising to realize a competency based procedure. Further development of the instruments and research on predictive validity should be pursued.

A new competency model for general practice: implications for selection, training, and careers

BACKGROUND

Recent structural and policy changes in the UK health service have significantly changed the job responsibilities for the GP role.

AIM

To replicate a previous job analysis study to examine the relevance of current competency domains and selection criteria for doctors entering training.

DESIGN AND METHOD

A multisource, multimethod approach comprising three phases: (1) stakeholder consultation (n = 205) using interviews, focus groups and behavioural observation of practising GPs; (2) a validation questionnaire based on results from phase 1 (n = 1082); followed by (3) an expert panel (n = 6) to review and confirm the final competency domains.

RESULTS

Eleven competency domains were identified, which extends previous research findings. A new domain was identified called Leading for Continuing Improvement. Results show that, Empathy and Perspective Taking, Communication Skills, Clinical Knowledge and Expertise, and Professional Integrity are currently rated the most important domains. Results indicate a significant increase in ratings of importance for each domain in the future (P<0.001), except for Communication Skills and Empathy and Perspective Taking, which consistently remain high.

CONCLUSION

The breadth of competencies required for GPs has increased significantly. GPs are now required to resolve competing tensions to be effective in their role, such as maintaining a patient focus while overseeing commissioning, with a potential ethical conflict between these aspects. Selection criteria remain largely unchanged but with increased priority in some domains (for example, Effective Teamworking). However, there is an urgent need to review the training provision arrangements to reflect the greater breadth of competencies now required.

Adjusting medical school admission: assessing interpersonal skills using situational judgement tests

CONTEXT

Today's formal medical school admission systems often include only cognitively oriented tests, although most medical school curricula emphasise both cognitive and non-cognitive factors. Situational judgement tests (SJTs) may represent an innovative approach to the formal measurement of interpersonal skills in large groups of candidates in medical school admission processes. This study examined the validity of interpersonal video-based SJTs in relation to a variety of outcome measures.

METHODS

This study used a longitudinal and multiple-cohort design to examine anonymised medical school admissions and medical education data. It focused on data for the Flemish medical school admission examination between 1999 and 2002. Participants were 5444 candidates taking the medical school admission examination. Outcome measures were first-year grade point average (GPA), GPA in interpersonal communication courses, GPA in non-interpersonal courses, Bachelor's degree GPA, Master's degree GPA and final-year GPA (after 7 years). For students pursuing careers in general practice, additional outcome measures (9 years after sitting examinations) included supervisor ratings and the results of an interpersonal objective structured clinical examination (OSCE), a general practice knowledge test and a case-based interview.

RESULTS

Interpersonal skills assessment carried out using SJTs had significant added value over cognitive tests for predicting interpersonal GPA throughout the curriculum, doctor performance, and performance on an OSCE and in a case-based interview. For the other outcomes, cognitive tests emerged as the better predictors. Females significantly outperformed males on the SJT (d = -0.26). The interpersonal SJT was perceived as significantly more job-related than the cognitive tests (d = 0.55).

CONCLUSIONS

Video-based SJTs as measures of procedural knowledge about interpersonal behaviour show promise as complements to cognitive examination components. The interpersonal skills training received during medical education does not negate the selection of students on the basis of interpersonal skills. Future research is needed to examine the use of SJTs in other cultures and student populations.

Evaluations of situational judgement tests to assess non-academic attributes in selection

OBJECTIVES

This paper presents a systematic review of the emerging international research evidence for the use of situational judgement tests (SJTs) for testing important non-academic attributes (such as empathy, integrity and resilience) in selection processes.

METHODS

Several databases (e.g. MEDLINE, PsycINFO, Web of Science) were searched to retrieve empirical studies relating to SJTs published between 1990 and 2010. Personal contact with experts in the field was made to identify any unpublished research or work in progress to obtain the most current material. Finally, reference lists were checked to access other relevant journal articles and further research. All research studies were required to meet specific inclusion criteria selected by two independent reviewers.

RESULTS

Over 1000 citations were identified during the initial literature search; following the review of abstracts, full-text copies of 76 articles were retrieved and evaluated. A total of 39 articles that adequately met the inclusion criteria were included in the final review. The research evidence shows that, compared with personality and IQ tests, SJTs have good levels of reliability, predictive validity and incremental validity for testing a range of professional attributes, such as empathy and integrity.

CONCLUSIONS

SJTs can be designed to test a broad range of non-academic constructs depending on the selection context. As a relatively low-fidelity assessment, SJTs are a cost-efficient methodology compared with high-fidelity assessments of non-academic attributes, such as those used in objective structured clinical examinations. In general, SJTs are found to demonstrate less adverse impact than IQ tests and are positively received by candidates. Further research is required to explore theoretical developments and the underlying construct validity of SJTs.

Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection

OBJECTIVES

  This study aimed to evaluate the validity and utility of and candidate reactions towards cognitive ability tests, and current selection methods, including a clinical problem-solving test (CPST) and a situational judgement test (SJT), for postgraduate selection. METHODS;  This was an exploratory, longitudinal study to evaluate the validities of two cognitive ability tests (measuring general intelligence) compared with current selection tests, including a CPST and an SJT, in predicting performance at a subsequent selection centre (SC). Candidate reactions were evaluated immediately after test administration to examine face validity. Data were collected from candidates applying for entry into training in UK general practice (GP) during the 2009 recruitment process. Participants were junior doctors (n = 260). The mean age of participants was 30.9 years and 53.1% were female. Outcome measures were participants' scores on three job simulation exercises at the SC.

RESULTS

  Findings indicate that all tests measure overlapping constructs. Both the CPST and SJT independently predicted more variance than the cognitive ability test measuring non-verbal mental ability. The other cognitive ability test (measuring verbal, numerical and diagrammatic reasoning) had a predictive value similar to that of the CPST and added significant incremental validity in predicting performance on job simulations in an SC. The best single predictor of performance at the SC was the SJT. Candidate reactions were more positive towards the CPST and SJT than the cognitive ability tests.

CONCLUSIONS

  In terms of operational validity and candidate acceptance, the combination of the current CPST and SJT proved to be the most effective administration of tests in predicting selection outcomes. In terms of construct validity, the SJT measures procedural knowledge in addition to aspects of declarative knowledge and fluid abilities and is the best single predictor of performance in the SC. Further research should consider the validity of the tests in this study in predicting subsequent performance in training.

When and why do doctors decide to become general practitioners? Implications for recruitment into UK general practice specialty training

All applicants to round 1 of national recruitment into the general practice specialty recruitment process were surveyed as to the reasons for, and the timing of their career choices. Most applicants reported decision making after completing undergraduate training citing variety, continuity of care and work-life balance as their main drivers for a career in general practice. Applicants were statistically more likely to have undertaken a Foundation placement in general practice than their peers on a Foundation programme. Reasons for choice of deanery were largely related to location and social ties, rather than to the educational 'reputation' of its programmes.

New machine-marked tests for selection into core medical training: evidence from two validation studies

This study examined whether two machine-marked tests (MMTs; a clinical problem-solving test and situational judgement test), previously validated for selection into U.K. general practice (GP) training, could provide a valid methodology for shortlisting into core medical training (CMT). A longitudinal design was used to examine the MMTs' psychometric properties in CMT samples, and correlations between MMT scores and CMT interview outcomes. Independent samples from two years were used: in 2008, a retrospective analysis was conducted (n=1711), while in 2009, CMT applicants completed the MMTs for evaluation purposes (n=2265). Both MMTs showed good reliability in CMT samples, similar to GP samples. Both MMTs were good predictors of CMT interview performance (r = 0.56, p < 0.001 in 2008; r = 0.61, p < 0.001 in 2009) and offered incremental validity over the current shortlisting process. The GP MMTs offer an appropriate measurement methodology for selection into CMT, representing a significant innovation for selection methodology.