Development and Validation of a Situational Judgement Test to Assess Professionalism

Developing a Situational Judgement Test to Assess Clinical Judgement in Fourth-Year Medical Students: A Pilot Study

Introduction Assessing clinical judgement objectively and economically presents a challenge in academic medicine. The authors developed a situational judgement test (SJT) to measure fourth-year medical students' clinical judgement. Methods A knowledge-based, single-best-answer SJT was developed by a panel of subject matter experts (SMEs). The SJT included 30 scenarios, each with five response options ranked ordinally from most to least appropriate. A computer-based format was used, and the SJT was piloted by two cohorts of fourth-year medical students at California University of Science and Medicine in 2022 and 2023 upon completion of an internship preparation course. Subsequently, students completed an optional survey. Evaluated scoring methods included original ordinal ranking, dichotomous, dichotomous with negative correction, distance from SME best answer, and distance from SME best answer squared. Results The SJT was completed by 142 fourth-year medical students. Cronbach's alpha ranged from 0.39 to 0.85, depending on the scoring method used. The distance-from-SME-best-answer-squared method yielded the highest internal consistency, which was considered acceptable. Using this scoring method, the mean score was 72.89 (SD = 48.32, range = 26-417), and the standard error of measurement was 18.41. Item analysis found that seven (23%) scenarios were of average difficulty, 13 (43%) had a good or satisfactory discrimination index, and nine (30%) had a distractor efficiency of at least 66%. Most students preferred the SJT to a traditional multiple-choice exam (16; 62%) and felt it was an appropriate tool to assess clinical judgement (15; 58%). Conclusions The authors developed and piloted an SJT to assess clinical judgement among medical students. Although not achieving validation, subsequent development of the SJT will focus on expanding the SME concordance panel, improving difficulty and discrimination indices, and conducting parallel forms reliability and adverse impact analyses.

Exploring the Utility and Fairness of a Multi-institutional Situational Judgment Test

OBJECTIVE

To pilot a situational judgment test (SJT) developed to assess empathy, integrity, and teamwork and explore differences in performance between first-year (P1) and fourth-year (P4) student pharmacists based on gender, race or ethnicity, and geographical region.

METHODS

An SJT was developed to assess empathy, integrity, and teamwork, using best practices. The SJT was piloted tested with P1 and P4 student pharmacists at 5 institutions. Scores were adjusted for elevation and scatter to minimize the subgroup differences in response styles.

RESULTS

The unadjusted mean SJT scores for P1 and P4 student pharmacists were 79.7 % and 80.7 %, respectively. The adjusted scores for P1 and P4 student pharmacists were 86.3 % and 87.2 %, respectively. Small differences were observed across institutions which can likely explained by multiple factors. The mean scores were higher among students who were older, female, identified as White, and spoke English as their first language. The subgroup differences persisted even after adjusting scores for elevation and scatter.

CONCLUSION

Performance on an SJT designed to assess empathy, integrity, and teamwork was similar to others in the literature; however, differences in performance were observed across subgroups. Additional research is needed to determine whether these differences reflect test bias or other possible factors.

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

OBJECTIVE

The primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness.

METHODS

An SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics.

RESULTS

Thirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics.

CONCLUSION

An SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.

Design and Evaluation of an APPE Readiness Dashboard Tool

OBJECTIVE

To describe the design and evaluation of a dashboard tool to assess the readiness for advanced pharmacy practice experiences (APPEs).

METHODS

The investigators developed a dashboard tool for the appraisal of APPE readiness consisting of indicators of pharmacy student knowledge, skills, and noncognitive attributes. For the 2018-2019 and 2019-2020 rotation years, a panel of stakeholders used the dashboard to evaluate APPE readiness. Students deemed not ready (5 or more dashboard indicators) had a remediation experience before the APPEs. The investigators monitored the students who were deemed ready but at risk for substandard performance (3 or 4 indicators or completion of remediation experience) and tracked the incidence of substandard performance for all APPE students. The outcomes were presented using descriptive statistics. Performance of the tool between years was compared using the χ2 test.

RESULTS

For 2 APPE rotation cycles, the investigators evaluated readiness in 499 students. A total of 3 students were deemed not ready, and 78 students were considered at risk for substandard performance. The tool predicted 52.1% of students who experienced a substandard performance. It had a negative predictive value of 93.6% and a positive predictive value of 38.5% for substandard performance. The predictive value was similar across the risk categories of knowledge, skills, and noncognitive attributes.

CONCLUSION

A dashboard tool facilitating holistic evaluation of APPE readiness was helpful in identifying students who were not ready for APPEs or at risk for substandard performance. The next steps include refinement of the tool through the evaluation of characteristics of students who experienced substandard performance and were not identified by the tool.

Perspectives of Professionalism, Self-Awareness, Leadership, and Innovation and Entrepreneurship From Experiential Education Preceptors

OBJECTIVE

The objectives of this study were to (1) quantify the perceived importance of and comfort with the Accreditation Council for Pharmacy Education Standard 4 (ACPE-S4) key elements, (2) describe how preceptors promote and assess key elements, and (3) describe preceptor perceptions of student key element skill demonstration.

METHODS

This study used a cross-sectional electronic survey to assess experiential education preceptors' perceptions of ACPE-S4 key elements. The survey consisted of 2 item categories, with 5 and 15 items included to collect information on preceptor demographics and ACPE-S4 key elements, respectively. Surveys were distributed to 451 introductory and advanced rotation preceptors. Frequencies and means were reported for quantitative survey items, open-ended survey responses were analyzed using a general interpretivist approach.

RESULTS

In total, 242 surveys were completed for a response rate of 54%. Most respondents identified as female (54.5%) and precepted community pharmacy rotations (28.5%). ACPE-S4 key elements, in order of most to least important, were professionalism, self-awareness, leadership, and innovation and entrepreneurship. Respondents varied in their conceptualization of key elements, with multiple activities used for skill demonstration and assessments. Qualitative analysis identified 2 themes: "Variation in preceptor fluency with S4 key elements" and "Communication as a means to demonstrate and evaluate ACPE-S4 key element skills."

CONCLUSION

Variation among preceptors in conceptualization, learning activities, and comfort in assessment for ACPE-S4 key elements highlights the need for support and resources to assist preceptors in developing activities and assessments for skills reflective of ACPE-S4 key elements. Preceptors commonly relied on vulnerable communication with students to assess key element skills, identifying a potential target for assessment and preceptor development. Pharmacy programs, in addition to the Academy, could benefit from using assessments already offered by preceptors. Finally, explicit evaluation criteria may help to guide skill acquisition and facilitate consistent assessment of key element skills.

Examining Validity for the Pharmacy Affective Domain Situational Judgment Test

Objective. With the integration of the affective domain in the pharmacy accreditation standards, it is important for programs to have methods for formatively assessing student competency in these areas. The objective of this study was to examine the validity evidence for a recently developed situational judgment test to assess the affective domain contained in the Accreditation Council for Pharmacy Education (ACPE) Standards 2016.Methods. After pilot testing items in multiple pharmacy schools, a revised 15-item situational judgment test instrument was developed measuring the affective domain as it relates to the pharmacy profession. The resulting Pharmacy Affective Domain Situational Judgment Test (PAD-S) was administered to students at three institutions and internal structure validity evidence was examined using item descriptives, Cronbach alpha, and results from a Rasch measurement model.Results. There were 559 valid responses included in the study. The items were of varying difficulty, with Rasch results confirming the hierarchy of items and suggesting that items were contributing to the measure. The Cronbach alpha was 0.70, suggesting acceptable reliability. However, the reliability items from the Rasch model were lower (0.65 and 0.66), indicating that the ranking of ability was limited, which may be due to fewer items of high difficulty.Conclusion. Overall, the PAD-S performed well as a measure of the affective domain. The PAD-S may be a useful formative instrument as part of a comprehensive assessment plan and may be less resource intensive than other measures.

Understanding Professionalism's Interplay Between the Profession's Identity and One's Professional Identity

Pharmacy schools have taught professionalism for many years but are now discussing the model of professional identity formation adopted by academic medicine. While professionalism is an outward expression of a community's norms, professional identity is an internalization of those norms such that one thinks, acts, and feels like a member of that community. Some have expressed concern that the wide-ranging roles and responsibilities of pharmacists mean there is no universal identity for the pharmacy profession, and therefore a student's professional identity cannot be intentionally developed. This commentary suggests that the profession's identity is an outward expression to patients and other health care providers of what pharmacists do, while one's professional identity is an internal acceptance that being a pharmacist is part of who one is. While a clear identity of the profession may make one's professional identity easier to develop, full agreement on the profession's identity is not required to begin guiding student pharmacists in the formation of their professional identities.

Use of Situational Judgment Tests to Teach Empathy, Assertiveness, Communication, and Ethics

Objective. The objectives of this study were to develop situational judgment test (SJT) scenarios for use in teaching empathy, assertiveness, and interprofessional communication in second-year pharmacy (P2) students and ethics in third-year pharmacy (P3) students, determine whether the SJTs developed were effective at measuring students' communication skills and ethical judgment, compare the performance of individual third-year students to the performance of teams of third-year students on the ethics SJT, and evaluate student feedback about SJTs as a teaching tool.Methods. Pharmacy faculty developed five SJT scenarios related to communications and five SJT scenarios related to ethics and piloted the scenarios with P2 and P3 students, respectively. Second-year students completed SJTs individually, while P3 students completed SJTs individually and in teams. Scenarios and responses were discussed with faculty after completion of the SJTs, and students completed a questionnaire to provide feedback on the SJTs.Results. The communications SJT was completed by 59 P2 students with a mean score of 67.5%. The ethics SJT was completed by 57 P3 students with a mean score of 80.1%. The ethics SJT was also completed by 10 teams of P3 students resulting in a mean score of 93.2%. Students indicated the SJT content was realistic and the tests provided the opportunity to reflect on how to approach challenging situations.Conclusion. Situational judgment tests were useful for teaching empathy, assertiveness, interprofessional communication, and ethics in pharmacy students. Future research should focus on predictive validity of SJTs for these content areas.

A Situational Judgment Test to Assess Students' Achievement of Accreditation Council for Pharmacy Education Standards 3 and 4

Objective. For pharmacy students to successfully meet competencies related to Accreditation Council for Pharmacy Education Standards 3 and 4, it is essential for pharmacy programs to assess student progression in the affective domain. The purpose of this study was to develop and assess the validity of a situational judgment test (SJT) for evaluating student mastery of Standards 3 and 4.Methods. A multi-institutional faculty team developed an 18-item SJT that consisted of scenarios asking the respondent to rank the effectiveness of four response options mapped to Standards 3 and 4. The research team systematically reviewed the literature, created items, and deliberated until consensus was achieved. Subject matter experts (SMEs) reviewed and provided feedback on the instrument. Students from two institutions were recruited to participate in cognitive interviews about the finalized instrument. Cognitive interview data were analyzed to identify themes.Results. After edits were made to the instrument based on SME feedback, students (n=18) in the cognitive interviews identified item length as a concern and commented on item/response clarity and comprehensiveness. Data from the cognitive interviews were used to modify the SJT to reduce the length and clarify items. The result was two shorter versions of the instrument, both with similar mapping to all elements in Standards 3 and 4.Conclusion. Early steps in validating the SJT suggested that the instrument may be a promising tool to assess student progression in the affective domain. The SJT instrument is intended to provide evidence of student pharmacist development that occurs in the didactic, experiential, and co-curricular portions of pharmacy education. The instrument can serve as one part of a comprehensive assessment plan.

The Importance of Teaching Ethics in Pharmacy Education

All pharmacy faculty members should have a general understanding of the field of ethics, regardless of whether they have received any formal training, since instruction and training in ethical decision-making is an accreditation expectation. Additionally, whether they recognize it or not, pharmacy faculty members are involved in ethical decision-making on an almost daily basis. The aims of the current commentary are to expand on a basic approach to ethical decision-making using examples involving students or faculty members in each of the triad areas of teaching, research, and service, and serve as a starting point to enable all faculty to teach students how to work through an ethical dilemma. This commentary will focus on the initial steps involved in determining whether an ethical dilemma exists, determining the facts related to the dilemma by identifying technical facts and legal constraints, and identifying the principles and values that play a role in the situation and decide which are in conflict. References are provided for more in-depth review of ethics subject matter beyond the scope of this commentary.

Candidate reactions to a postgraduate year 1 pharmacy residency supplemental application

DISCLAIMER

In an effort to expedite the publication of articles, AJHP is posting manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

PURPOSE

The purpose of this study was to gauge postgraduate year 1 (PGY1) pharmacy residency candidates' reactions to supplemental applications, as well as aspects specific to our process, including a personality test and situational judgment test (SJT).

METHODS

After rank lists were submitted, applicants to our program were invited to complete an online survey. Outcomes of interest included candidates' perceptions of relevance and fairness. Whether candidates' attitudes differed based on the receipt of an interview offer was also assessed.

RESULTS

Of 199 applicants to our program for the 2021-2022 training year, 48 applicants (24.1%) completed the survey, 15 of whom had received an interview offer. Most (64.6%) agreed that supplemental applications were useful, and nearly all (95.8%) indicated that they were willing to submit one for programs in which they were most interested. The process was seen as being fair, although ratings were higher among those who received interview offers. Most respondents believed that the personality test and SJT were relevant to the role of a resident, but attitudes towards the SJT were generally more favorable and less likely to vary according to whether candidates received an interview offer. Candidates believed that the personality test and SJT were not as representative of them as letters of reference or their curriculum vitae, but perceptions of academic performance varied.

CONCLUSION

Applicants responded positively to our supplemental application and indicated that they would be willing to complete one for programs of interest. These findings should help assuage concerns about the use of supplemental applications, particularly when short-answer or essay formats are avoided.

Assessing professionalism in health profession degree programs: A scoping review

BACKGROUND

To identify and classify methods for assessing professionalism across health profession degree programs and identify gaps in the literature regarding types of assessments.

METHODS

The authors conducted a scoping review of articles published from database inception through 24 January 2020. Included articles described an assessment approach for professionalism in health profession degree programs available in full-text in the English language. Articles were classified based on profession, timing of assessment, feedback type, assessment type, professionalism dimension, and Barr's modified Kirkpatrick hierarchy.

RESULTS

Authors classified 277 articles meeting inclusion criteria. Most articles were from medical education (62.5%) conducted during didactic (62.1%) or experiential/clinical curriculum (49.8%). Few articles (15.5%) described longitudinal assessment. Feedback type was formative (32.2%) or summative (35%), with only 8.3% using both. Assessment types frequently reported included self-administered rating scales (30%), reflections (18.8%), observed clinical encounters (17.3%), and knowledge-based tests (13.4%). Ethical practice principles (65%) and effective interactions with patients (48.4%) were the most frequently assessed dimensions of professionalism. Authors observed balanced distribution among Barr's modified Kirkpatrick model at levels of reaction (38.3%), modification of perceptions and attitudes (33.6%), acquisition of knowledge and skills (39%), and behavioral change (36.1%).

IMPLICATIONS

The classification scheme identified in current literature on professionalism assessment does not align with International Ottawa Conference Working Group on the Assessment of Professionalism recommendations. Gaps identified were limited description of professionalism assessment during admissions, infrequent longitudinal assessment, limited use of methods for both formative and summative assessment, and limited reports of assessments applicable to interprofessional education settings.

Exploring the Role of Item Scenario Features on Situational Judgment Test Response Selections

Objective. To explore pharmacists' and pharmacy students' perceptions regarding the significance of changing the features of test item scenario (eg, switching from a health care to a non-health care context) on their situational judgment test (SJT) responses.Methods. Fifteen Doctor of Pharmacy students and 15 pharmacists completed a 12-item SJT intended to measure empathy. The test included six scenarios in a health care context and six scenarios in a non-health care context; participants had to rank potential response options in order of appropriateness and no two items could be of equal rank. Qualitative data were collected individually from participants using think-aloud and cognitive interview techniques. During the cognitive interview, participants were asked how they selected their final responses for each item and whether they would have changed their answer if features of the scenario were switched (eg, changed to a non-health care context if the original item was in a health care context). Interviews were transcribed and a thematic analysis was conducted to identify the features of the scenario for each item that were perceived to impact response selections.Results. Participants stated that they would have changed their responses on average 51.3% of the time (range 20%-100%) if the features of the scenario for an item were changed. Qualitative analysis identified four pertinent scenario features that may influence response selections, which included information about the examinee, the actors in the scenario, the relationship between examinee and actors, and details about the situation. There was no discernible pattern linking scenario features to the component of empathy being measured or participant type.Conclusion. Results from this study suggest that the features of the scenario described in an SJT item could influence response selections. These features should be considered in the SJT design process and require further research to determine the extent of their impact on SJT performance.

Exploring Multiple Perspectives on Pharmacy Students' Readiness for Advanced Pharmacy Practice Experiences

Objective. This study aimed to enrich the Academy's understanding of pharmacy students' readiness for advanced pharmacy practice experiences (APPEs) by exploring the perspectives of three primary stakeholders: APPE students, APPE preceptors, and APPE faculty site directors.Methods. A descriptive qualitative study of APPE readiness was conducted using workplace learning as a guiding conceptual framework. Data were collected between March and September 2019 through semi-structured focus groups and interviews with students (five groups), preceptors (four groups), and faculty site directors (one group, two individual interviews). The data were analyzed using directed content analysis.Results. Participants described APPE readiness as a multifaceted construct comprised of four themes: learner characteristics, participation in workplace activities, relationship-building, and workplace practices to orient and support students. While all participants addressed each category, faculty site directors and preceptors tended to focus on learner characteristics, while students emphasized their participation in the workplace and relationship building.Conclusion. Knowledge is widely recognized as a requirement for APPE readiness. This study identified learner characteristics, workplace participation, and relational skills as additional requirements. Some of these criteria are challenging to assess prior to APPEs, which makes orienting students both prior to and at the start of APPEs particularly important to support readiness. Thus, a comprehensive review of APPE readiness might also include assessing the readiness of workplaces, administrators, and preceptors for APPE students.