Situational judgment tests: An introduction for clinician educators

Evaluation of Incremental Validity of Casper in Predicting Program and National Licensure Performance of Undergraduate Nursing Students: Protocol for a Mixed Methods Study

BACKGROUND

Academic success has been the primary criterion for admission to many nursing programs. However, academic success as an admission criterion may have limited predictive value for success in noncognitive skills. Adding situational judgment tests, such as Casper, to admissions procedures may be one strategy to strengthen decisions and address the limited predictive value of academic admission criteria. In 2021, admissions processes were modified to include Casper based on concerns identified with noncognitive skills.

OBJECTIVE

This study aims to (1) assess the incremental validity of Casper scores in predicting nursing student performance at years 1, 2, 3, and 4 and on the National Council Licensing Examination (NCLEX) performance; and (2) examine faculty members' perceptions of student performance and influences related to communication, professionalism, empathy, and problem-solving.

METHODS

We will use a multistage evaluation mixed methods design with 5 phases. At the end of each year, students will complete questionnaires related to empathy and professionalism and have their performance assessed for communication and problem-solving in psychomotor laboratory sessions. The final phase will assess graduate performance on the NCLEX. Each phase also includes qualitative data collection (ie, focus groups with faculty members). The goal of the focus groups is to help explain the quantitative findings (explanatory phase) as well as inform data collection (eg, focus group questions) in the subsequent phase (exploratory sequence). All students enrolled in the first year of the nursing program in 2021 were asked to participate (n=290). Faculty will be asked to participate in the focus groups at the end of each year of the program. Hierarchical multiple regression will be conducted for each outcome of interest (eg, communication, professionalism, empathy, and problem-solving) to determine the extent to which scores on Casper with admission grades, compared to admission grades alone, predict nursing student performance at years 1-4 of the program and success on the national exam. Thematic analysis of focus group transcripts will be conducted using interpretive description. The quantitative and qualitative data will be integrated after each phase is complete and at the end of the study.

RESULTS

This study was funded in September 2021, and data collection began in March 2022. Year 1 data collection and analysis are complete. Year 2 data collection is complete, and data analysis is in progress.

CONCLUSIONS

At the end of the study, we will provide the results of a comprehensive analysis to determine the extent to which the addition of scores on Casper compared to admission grades alone predicts nursing student performance at years 1-4 of the program and on the NCLEX exam.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR1-10.2196/48672.

Interested in situational judgment tests? Preparing pharmacy educators for potential challenges

PURPOSE

There is a growing interest in situational judgment tests (SJTs) for assessments and the authors offer their insights to help others develop and utilize these instruments while navigating common challenges.

DESCRIPTION

This manuscript provides important considerations to those interested in SJTs using the "What? So What? Now What?" reflection framework. The authors offer their experience developing, implementing, and evaluating SJTs designed to measure empathy, adaptability, and integrity.

ANALYSIS/INTERPRETATION

Effective SJT design can be organized around five key steps: attribute definition, item design, key development and scoring, test administration, and data analysis and interpretation. There are general recommendations to optimize SJT design to enhance reliability and validity of the results; however, there are often other challenges that have not been previously described.

CONCLUSIONS

SJTs offer an exciting tool for measuring social and behavioral attributes; however, their infancy in health professions education warrants caution. Like any assessment, there are several challenges to consider when designing, implementing, and evaluating SJTs in pharmacy education.

IMPLICATIONS

Individuals interested in SJTs are encouraged to ensure the construct measured is well-defined, there are clear and supported decisions around item and key design, and the appropriate psychometric analyses are conducted.