A Five-Minute Situational Judgment Test to Assess Empathy in First-Year Student Pharmacists

Exploring the Utility and Fairness of a Multi-institutional Situational Judgment Test

OBJECTIVE

To pilot an SJT developed to assess empathy, teamwork, and integrity, and to explore differences in performance between first-year (P1) and fourth-year (P4) student pharmacists, based on gender, race or ethnicity, and geographical region.

METHODS

An SJT was developed to assess empathy, teamwork, and integrity using best practices. The SJT was piloted tested with P1 and P4 student pharmacists at five institutions. Scores were adjusted for elevation and scatter to minimize subgroup differences in response styles.

RESULTS

The unadjusted mean SJT scores for P1 and P4 student pharmacists were 79.7% and 80.7%, respectively. Adjusted scores for P1 and P4 student pharmacists were 86.3% and 87.2%, respectively. Small differences were observed across institutions and likely explained by multiple factors. Mean scores were higher among students who were older, female, identified as White, and spoke English as their first language. Subgroup differences persisted even after adjusting scores for elevation and scatter.

CONCLUSIONS

Performance on an SJT designed to assess empathy, integrity, and teamwork was similar to others in the literature, but differences in performance were observed across subgroups. Additional research is needed to determine if these differences reflect test bias or other possible factors.

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

OBJECTIVE

The primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness.

METHODS

An SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics.

RESULTS

Thirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics.

CONCLUSION

An SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.

Perspectives of Professionalism, Self-Awareness, Leadership, and Innovation and Entrepreneurship From Experiential Education Preceptors

OBJECTIVE

The objectives of this study were to (1) quantify the perceived importance of and comfort with the Accreditation Council for Pharmacy Education Standard 4 (ACPE-S4) key elements, (2) describe how preceptors promote and assess key elements, and (3) describe preceptor perceptions of student key element skill demonstration.

METHODS

This study used a cross-sectional electronic survey to assess experiential education preceptors' perceptions of ACPE-S4 key elements. The survey consisted of 2 item categories, with 5 and 15 items included to collect information on preceptor demographics and ACPE-S4 key elements, respectively. Surveys were distributed to 451 introductory and advanced rotation preceptors. Frequencies and means were reported for quantitative survey items, open-ended survey responses were analyzed using a general interpretivist approach.

RESULTS

In total, 242 surveys were completed for a response rate of 54%. Most respondents identified as female (54.5%) and precepted community pharmacy rotations (28.5%). ACPE-S4 key elements, in order of most to least important, were professionalism, self-awareness, leadership, and innovation and entrepreneurship. Respondents varied in their conceptualization of key elements, with multiple activities used for skill demonstration and assessments. Qualitative analysis identified 2 themes: "Variation in preceptor fluency with S4 key elements" and "Communication as a means to demonstrate and evaluate ACPE-S4 key element skills."

CONCLUSION

Variation among preceptors in conceptualization, learning activities, and comfort in assessment for ACPE-S4 key elements highlights the need for support and resources to assist preceptors in developing activities and assessments for skills reflective of ACPE-S4 key elements. Preceptors commonly relied on vulnerable communication with students to assess key element skills, identifying a potential target for assessment and preceptor development. Pharmacy programs, in addition to the Academy, could benefit from using assessments already offered by preceptors. Finally, explicit evaluation criteria may help to guide skill acquisition and facilitate consistent assessment of key element skills.

Examining Validity for the Pharmacy Affective Domain Situational Judgment Test

Objective. With the integration of the affective domain in the pharmacy accreditation standards, it is important for programs to have methods for formatively assessing student competency in these areas. The objective of this study was to examine the validity evidence for a recently developed situational judgment test to assess the affective domain contained in the Accreditation Council for Pharmacy Education (ACPE) Standards 2016.Methods. After pilot testing items in multiple pharmacy schools, a revised 15-item situational judgment test instrument was developed measuring the affective domain as it relates to the pharmacy profession. The resulting Pharmacy Affective Domain Situational Judgment Test (PAD-S) was administered to students at three institutions and internal structure validity evidence was examined using item descriptives, Cronbach alpha, and results from a Rasch measurement model.Results. There were 559 valid responses included in the study. The items were of varying difficulty, with Rasch results confirming the hierarchy of items and suggesting that items were contributing to the measure. The Cronbach alpha was 0.70, suggesting acceptable reliability. However, the reliability items from the Rasch model were lower (0.65 and 0.66), indicating that the ranking of ability was limited, which may be due to fewer items of high difficulty.Conclusion. Overall, the PAD-S performed well as a measure of the affective domain. The PAD-S may be a useful formative instrument as part of a comprehensive assessment plan and may be less resource intensive than other measures.

Use of Situational Judgment Tests to Teach Empathy, Assertiveness, Communication, and Ethics

Objective. The objectives of this study were to develop situational judgment test (SJT) scenarios for use in teaching empathy, assertiveness, and interprofessional communication in second-year pharmacy (P2) students and ethics in third-year pharmacy (P3) students, determine whether the SJTs developed were effective at measuring students' communication skills and ethical judgment, compare the performance of individual third-year students to the performance of teams of third-year students on the ethics SJT, and evaluate student feedback about SJTs as a teaching tool.Methods. Pharmacy faculty developed five SJT scenarios related to communications and five SJT scenarios related to ethics and piloted the scenarios with P2 and P3 students, respectively. Second-year students completed SJTs individually, while P3 students completed SJTs individually and in teams. Scenarios and responses were discussed with faculty after completion of the SJTs, and students completed a questionnaire to provide feedback on the SJTs.Results. The communications SJT was completed by 59 P2 students with a mean score of 67.5%. The ethics SJT was completed by 57 P3 students with a mean score of 80.1%. The ethics SJT was also completed by 10 teams of P3 students resulting in a mean score of 93.2%. Students indicated the SJT content was realistic and the tests provided the opportunity to reflect on how to approach challenging situations.Conclusion. Situational judgment tests were useful for teaching empathy, assertiveness, interprofessional communication, and ethics in pharmacy students. Future research should focus on predictive validity of SJTs for these content areas.

Community pharmacist's perspective regarding patient-centred communication in conjunction with pharmaceutical practice: a cross-sectional survey

Effective communication between pharmacists and patients is essential for therapeutic success. The pharmacist's perspective may differ from that of the patient in terms of effective communication. Our study aimed to assess the communication efficiency in the pharmacist-patient relationship from the pharmacists’ perspective. We hypothesize that the community pharmacist's perspective can lead to relevant aspects of patient-centred communication and their profession. A cross-sectional study was conducted through an online questionnaire addressed to pharmacists. A number of 506 questionnaires were collected, evaluated, analyzed and interpreted. The questionnaire focused on the following main issues: degree of job satisfaction, essential skills of a pharmacist working in a community pharmacy, different aspects of pharmacist-patient communication, shared decision-making, patient monitoring plan and other elements related to a patient under treatment (healthy lifestyle, receptivity to counselling, loyalty and appreciation of pharmaceutical services). The pharmacist's efficiency in communication with the patient and professional education were also targeted in the study. There are no significant differences between job satisfaction in women compared to men. However, the job satisfaction increases with the average age. Caregiver, communicator and life-long learner were identified as essential skills of a pharmacist. Pharmacists participating in the study generally perceive themselves as good communicators with the patient. Nevertheless, many particular aspects of communication with patients can be greatly improved. The lack of training in the spirit of the „patient-centred communication” concept has been identified. Still, more than a third of the respondents are missing the need for professional training. A periodic evaluation of the efficiency of pharmacist-patient communication in the community pharmacy is necessary. The degree of subjectivity of the pharmacist from this perspective and self-sufficiency would be significantly diminished if the pharmacists had access to the results of the periodic evaluations.

Interested in situational judgment tests? Preparing pharmacy educators for potential challenges

PURPOSE

There is a growing interest in situational judgment tests (SJTs) for assessments and the authors offer their insights to help others develop and utilize these instruments while navigating common challenges.

DESCRIPTION

This manuscript provides important considerations to those interested in SJTs using the "What? So What? Now What?" reflection framework. The authors offer their experience developing, implementing, and evaluating SJTs designed to measure empathy, adaptability, and integrity.

ANALYSIS/INTERPRETATION

Effective SJT design can be organized around five key steps: attribute definition, item design, key development and scoring, test administration, and data analysis and interpretation. There are general recommendations to optimize SJT design to enhance reliability and validity of the results; however, there are often other challenges that have not been previously described.

CONCLUSIONS

SJTs offer an exciting tool for measuring social and behavioral attributes; however, their infancy in health professions education warrants caution. Like any assessment, there are several challenges to consider when designing, implementing, and evaluating SJTs in pharmacy education.

IMPLICATIONS

Individuals interested in SJTs are encouraged to ensure the construct measured is well-defined, there are clear and supported decisions around item and key design, and the appropriate psychometric analyses are conducted.

A Situational Judgment Test to Assess Students' Achievement of Accreditation Council for Pharmacy Education Standards 3 and 4

Objective. For pharmacy students to successfully meet competencies related to Accreditation Council for Pharmacy Education Standards 3 and 4, it is essential for pharmacy programs to assess student progression in the affective domain. The purpose of this study was to develop and assess the validity of a situational judgment test (SJT) for evaluating student mastery of Standards 3 and 4.Methods. A multi-institutional faculty team developed an 18-item SJT that consisted of scenarios asking the respondent to rank the effectiveness of four response options mapped to Standards 3 and 4. The research team systematically reviewed the literature, created items, and deliberated until consensus was achieved. Subject matter experts (SMEs) reviewed and provided feedback on the instrument. Students from two institutions were recruited to participate in cognitive interviews about the finalized instrument. Cognitive interview data were analyzed to identify themes.Results. After edits were made to the instrument based on SME feedback, students (n=18) in the cognitive interviews identified item length as a concern and commented on item/response clarity and comprehensiveness. Data from the cognitive interviews were used to modify the SJT to reduce the length and clarify items. The result was two shorter versions of the instrument, both with similar mapping to all elements in Standards 3 and 4.Conclusion. Early steps in validating the SJT suggested that the instrument may be a promising tool to assess student progression in the affective domain. The SJT instrument is intended to provide evidence of student pharmacist development that occurs in the didactic, experiential, and co-curricular portions of pharmacy education. The instrument can serve as one part of a comprehensive assessment plan.

Developing a Framework of Relationships Among Noncognitive Factors in Doctor of Pharmacy Students' Academic Performance

Objective. To develop a framework of the effects of select noncognitive factors (grit, perceived stress, internal locus of control, and select Big Five personality traits) on pharmacy students' academic performance.Methods. A survey measuring select noncognitive factors was administered to two cohorts of first professional year (P1) pharmacy students (entering classes of 2019 and 2020, n=374) during fall orientation. Demographics, pre-pharmacy GPA, and P1 fall semester GPA were collected from student records. Structural equation modeling was conducted to assess the proposed framework.Results. Academic performance was directly influenced by students' (n=367; 98.1%) perceived stress and neuroticism and indirectly influenced by internal locus of control and neuroticism (via perceived stress as mediator). Neuroticism has a complex relationship in the models as it was directly and positively associated with academic performance, and indirectly contributed to decreased academic performance via a positive association with perceived stress. Squared multiple correlations indicated 13% and 9% of the variance in academic performance in the first final model (academic performance measured by pre-pharmacy GPA and P1 fall GPA) and second final model (academic performance measured by P1 fall GPA), respectively, were explained by the predictor variables.Conclusion. Evidence provided by structural equation modeling supports the conclusion that select noncognitive factors, namely perceived stress, neuroticism, and internal locus of control, have direct and indirect effects on the academic success of P1 students. The model variances of 9% and 13% represent 36% to 52% of the predictive value of the most accepted cognitive measures used to determine students' potential for academic success.

Systematic review of noncognitive factors influence on health professions students' academic performance

Increased importance has been placed on noncognitive skills in professional development and by accrediting bodies of health professions programs in recent years. Therefore, the purpose of this study was to conduct a comprehensive systematic review of evidence examining effects of academic resilience, grit, perceived stress, locus of control, and Big Five Personality Traits on academic performance of health professions students. A literature search of peer-reviewed, English-language articles describing select noncognitive factors was performed using seven databases. Searches were performed from the earliest index date through May 2020. The following data from included studies were extracted and summarized: research design hierarchy, hierarchy of study outcomes (modified from Kirkpatrick), association between noncognitive factors and academic outcomes, and quality assessment criteria. 149 articles met inclusion criteria. Almost 80% of studies were Level III (observational). Medical students were the most frequently studied population (n = 73 articles). The most studied academic outcome was grade point average (n = 61). Perceived stress and Big Five Personality Traits accounted for greater than 50% of studies. Most studies were rated as fair to good quality. Associations between noncognitive factors and academic outcomes were largely inconsistent, although greater perceived stress was generally associated with poorer academic performance outcomes, while higher conscientiousness, academic resilience, and grit were generally associated with better outcomes. This systematic review represents a large body of evidence concerning select noncognitive factors and their association with academic performance of health professions students. Support services addressing noncognitive factors should be deliberated and tailored for specific health professions education programs and student populations.

Exploring the Role of Item Scenario Features on Situational Judgment Test Response Selections

Objective. To explore pharmacists' and pharmacy students' perceptions regarding the significance of changing the features of test item scenario (eg, switching from a health care to a non-health care context) on their situational judgment test (SJT) responses.Methods. Fifteen Doctor of Pharmacy students and 15 pharmacists completed a 12-item SJT intended to measure empathy. The test included six scenarios in a health care context and six scenarios in a non-health care context; participants had to rank potential response options in order of appropriateness and no two items could be of equal rank. Qualitative data were collected individually from participants using think-aloud and cognitive interview techniques. During the cognitive interview, participants were asked how they selected their final responses for each item and whether they would have changed their answer if features of the scenario were switched (eg, changed to a non-health care context if the original item was in a health care context). Interviews were transcribed and a thematic analysis was conducted to identify the features of the scenario for each item that were perceived to impact response selections.Results. Participants stated that they would have changed their responses on average 51.3% of the time (range 20%-100%) if the features of the scenario for an item were changed. Qualitative analysis identified four pertinent scenario features that may influence response selections, which included information about the examinee, the actors in the scenario, the relationship between examinee and actors, and details about the situation. There was no discernible pattern linking scenario features to the component of empathy being measured or participant type.Conclusion. Results from this study suggest that the features of the scenario described in an SJT item could influence response selections. These features should be considered in the SJT design process and require further research to determine the extent of their impact on SJT performance.

Exploring Multiple Perspectives on Pharmacy Students' Readiness for Advanced Pharmacy Practice Experiences

Objective. This study aimed to enrich the Academy's understanding of pharmacy students' readiness for advanced pharmacy practice experiences (APPEs) by exploring the perspectives of three primary stakeholders: APPE students, APPE preceptors, and APPE faculty site directors.Methods. A descriptive qualitative study of APPE readiness was conducted using workplace learning as a guiding conceptual framework. Data were collected between March and September 2019 through semi-structured focus groups and interviews with students (five groups), preceptors (four groups), and faculty site directors (one group, two individual interviews). The data were analyzed using directed content analysis.Results. Participants described APPE readiness as a multifaceted construct comprised of four themes: learner characteristics, participation in workplace activities, relationship-building, and workplace practices to orient and support students. While all participants addressed each category, faculty site directors and preceptors tended to focus on learner characteristics, while students emphasized their participation in the workplace and relationship building.Conclusion. Knowledge is widely recognized as a requirement for APPE readiness. This study identified learner characteristics, workplace participation, and relational skills as additional requirements. Some of these criteria are challenging to assess prior to APPEs, which makes orienting students both prior to and at the start of APPEs particularly important to support readiness. Thus, a comprehensive review of APPE readiness might also include assessing the readiness of workplaces, administrators, and preceptors for APPE students.

Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-aloud interviews

BACKGROUND

Situational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context.

METHODS

Thirty participants-15 student pharmacists and 15 practicing pharmacists-completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes.

RESULTS

The findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item.

CONCLUSIONS

This study provides additional SJT validity evidence by exploring participants' response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future.

Use of Capstone Experiences in Pharmacy Education to Synthesize and Apply Students' Knowledge and Skills

Objective. To investigate capstone experiences in pharmacy education with the broad aim of better understanding how they are designed, used, and sustained.Methods. Capstones were defined as culminating experiences that required Doctor of Pharmacy (PharmD) students to apply their knowledge and skills and demonstrate their ability to integrate, extend, and apply multiple components of their academic experience. This study was conducted in two phases. Phase one was a mapping review, which provided insight into the design of pharmacy capstone experiences, including the purpose they served and their associated outcomes. Phase two consisted of conducting follow-up interviews to explore the current status and sustainability of those capstone experiences. Qualitative methods with multiple coders were used to analyze the data.Results. Five distinct purposes for capstone experiences were identified: preparation for advanced pharmacy practice experiences (APPEs), pharmacotherapy education, research, reflection on pharmacy school experiences, and immersive learning through an external organization. Wide variability was found in the design, purpose, and outcomes of capstone experiences. Study authors described the capstone experiences at their institution in terms of the viability, feasibility, desirability, and challenges faced.Conclusion. Capstones can play an important role in assessing student progress and practice readiness in pharmacy education. The findings of this study can be used to help pharmacy schools design capstone experiences that are effective and sustainable.

Development and Validation of a Situational Judgement Test to Assess Professionalism

Objective. The purpose of this study was to develop, pilot, and validate a situational judgement test (SJT) to assess professionalism in Doctor of Pharmacy (PharmD) students. Methods. Test specifications were developed and faculty members were educated on best practices in item writing for SJTs. The faculty members then developed 75 pilot scenarios. From those, two versions of the SJT, each containing 25 scenarios, were created. The pilot population for the SJT was student pharmacists in their third professional year, just prior to starting their advanced pharmacy practice experiences. The students completed the two versions of the test on different days, approximately 48 hours apart, with 50 minutes allowed to complete each. Subsequently, students completed a questionnaire regarding the SJT at the conclusion of the second test. Results. Version 1 of the SJT was completed by 228 students, and version 2 was completed by 225 students. Mean scores were 390 (SD=20, range 318-429) and 342 (SD=21, range 263-387) on test versions 1 and 2, respectively. The reliability of the tests was appropriate (test version 1, α=0.77; test version 2, α=0.79). Students felt that the content of the tests was realistic with respect to pharmacy practice (90.1%), and that the tests gave them an opportunity to reflect on how to approach challenging situations (82.6%). Conclusion. We developed a reliable SJT to assess professionalism in PharmD students. Future research should focus on creating a personalized learning plan for students who do not meet minimum performance standards on this SJT.