Developing a Comprehensive First-year Capstone to Assess and Inform Student Learning and Curriculum Effectiveness

Evolution of a capstone exam for third-year doctor of pharmacy students

BACKGROUND AND PURPOSE

Pharmacy capstone exams are an example of an assessment strategy to evaluate student competency and practice-readiness. Capstone exam processes have been discussed in the literature; however, description of format, evolution of the process, and implementation in the year prior to rotations is lacking. This manuscript discusses the development, implementation, and utilization of a third professional year pharmacy capstone exam to assess advanced pharmacy practice experience (APPE) readiness.

EDUCATIONAL ACTIVITY AND SETTING

A pharmacy year three (PY3) capstone exam was initially developed prior to enrollment of the inaugural graduating class. Since the first iteration, utility has remained the same, but the overall development and implementation process has evolved due to changes in programmatic and student learning outcomes, curricular mapping, student and faculty feedback, and practice expectations.

FINDINGS

The entities responsible for exam implementation perceive faculty and students have a positive perception of the exam evolution and improvements implemented. Specifically, feedback from students and faculty mentioned clear expectations, comfort with the new capstone exam process, and an appreciation for the focus on interrater reliability. As assessment is an iterative process, quality improvement strategies continue to be implemented to address feedback regarding the PY3 capstone exam.

SUMMARY

Purposeful utilization of capstone exams in the third year is one method to assess APPE readiness. Sharing one college's experience with the evolution of such an exam, quality improvement methods, and experience in implementing reliability and validity measures may provide a model for institutions to implement similar assessment methods efficiently.

Feedback for Learning in Pharmacy Education: A Scoping Review

Feedback is an effective pedagogy aimed to create cognitive dissonance and reinforce learning as a key component of clinical training programs. Pharmacy learners receive constant feedback. However, there is limited understanding of how feedback is utilized in pharmacy education. This scoping review sought to summarize the breadth and depth of the use of feedback within pharmacy education and identify areas for future research. PubMed, Embase, Scopus, and Web of Science were searched for English articles since January 2000 to identify studies related to feedback in pharmacy education. Sixty-four articles were included for analysis, stratified by moderate and major theory talk, where moderate theory talk explicitly included feedback into study design and major theory talk included feedback into both study design and analysis. Feedback was provided in Bachelor (14%), Master (15.6%), Doctor of Pharmacy (67.2%) and post-graduate programs (4.7%) on a variety of curricular objectives including communication and patient work up in didactic, objective structured clinical examination (OSCE), and experiential settings, and career/interview preparation in the co-curriculum. Feedback comments were mostly written in didactic courses, and both written and verbal in OSCE, experiential, and co-curricular settings. The pharmacy education feedback literature lacks depth beyond student perceptions, especially with respect to assessing the effectiveness and quality of feedback for learning. While feedback has been utilized throughout pharmacy education across myriad outcomes, several areas for inquiry exist which can inform the design of faculty and preceptor development programs, ensuring provision of effective, quality feedback to pharmacy learners.

Use of Capstone Experiences in Pharmacy Education to Synthesize and Apply Students' Knowledge and Skills

Objective. To investigate capstone experiences in pharmacy education with the broad aim of better understanding how they are designed, used, and sustained.Methods. Capstones were defined as culminating experiences that required Doctor of Pharmacy (PharmD) students to apply their knowledge and skills and demonstrate their ability to integrate, extend, and apply multiple components of their academic experience. This study was conducted in two phases. Phase one was a mapping review, which provided insight into the design of pharmacy capstone experiences, including the purpose they served and their associated outcomes. Phase two consisted of conducting follow-up interviews to explore the current status and sustainability of those capstone experiences. Qualitative methods with multiple coders were used to analyze the data.Results. Five distinct purposes for capstone experiences were identified: preparation for advanced pharmacy practice experiences (APPEs), pharmacotherapy education, research, reflection on pharmacy school experiences, and immersive learning through an external organization. Wide variability was found in the design, purpose, and outcomes of capstone experiences. Study authors described the capstone experiences at their institution in terms of the viability, feasibility, desirability, and challenges faced.Conclusion. Capstones can play an important role in assessing student progress and practice readiness in pharmacy education. The findings of this study can be used to help pharmacy schools design capstone experiences that are effective and sustainable.

A Five-Minute Situational Judgment Test to Assess Empathy in First-Year Student Pharmacists

Objective. To design, implement, and evaluate the utility of a situational judgment test (SJT) to assess empathy in first-year student pharmacists as part of an end-of-year capstone experience. Methods. First-year students completed a five-minute SJT in lieu of a multiple mini interview (MMI) during the end-of-year capstone. For each SJT item, students selected the two most appropriate response options from a list of five. Various strategies to score the SJT were compared to evaluate the psychometric properties of the test. Student performance on the SJT was examined in relationship to performance on other measures, (eg, MMI stations, personality assessments, and admissions data). Results. A total of 135 first-year pharmacy students completed an average of 9.5 items. Scoring keys based on subject matter experts' and student responses demonstrated high reliability. There was a positive, weak relationship between student performance on the SJT and performance on the adaptability station used in the capstone, and an inverse, weak relationship with students' agreeableness scores. Conclusion. This study suggests that the SJT may be a feasible and efficient assessment strategy in pharmacy education. Additional research is needed to inform SJT design, implementation, and interpretation.

Assessing metacognition in the classroom: Student help-seeking behavior

INTRODUCTION

The study's purpose was to develop an assessment of students' metacognitive monitoring of help-seeking behavior.

METHODS

This study piloted an assessment of help-seeking behavior in first-year student pharmacists to answer two questions: (1) Does help-seeking behavior depend on how familiar students are with the content? and (2) When students ask for help, does their performance and metacognition differ from when they do not seek help? As part of their year-end capstone, students answered drug information questions. The drugs within these questions were chosen based on the level of emphasis during the first-year curriculum (i.e. more familiar or less familiar). For each question, students rated their confidence level for their answer's correctness and marked whether they would ask their preceptor for help. Bias scores were calculated under conditions of familiarity based on level of emphasis (more familiar vs. less familiar) and help-seeking (asked for help and did not ask for help).

RESULTS

Students performed better on more familiar material (d = 1.2), with a small difference in confidence on more familiar material (d = 0.2). When students asked for help, they scored lower (d = -2.2) and reported lower confidence (d = -3.7). Students were more likely to ask for help from their preceptors on more familiar content than less familiar (odds ratio = 1.25) and less likely to ask for help when they were overconfident (odds ratio = 0.18).

CONCLUSIONS

Overall, students were more overconfident for less familiar material and were less likely to ask for help.

Using the multiple mini interview as an assessment strategy within the first year of a health professions curriculum

BACKGROUND

The multiple mini-interview (MMI) is a common assessment strategy used in student selection. The MMI as an assessment strategy within a health professions curriculum, however, has not been previously studied. This study describes the integration of a 5-station MMI as part of an end-of-year capstone following the first year of a health professions curriculum. The goal of the capstone MMI was to assess professional competencies of students and to offer formative feedback to prepare students for their upcoming clinical practice experiences. The purpose of this study was to evaluate the psychometric properties of an MMI integrated into a health professions curriculum.

METHODS

Five capstone MMI stations were designed to each evaluate a single construct assessed by one rater. A principal component analysis (PCA) was used to evaluate the structure of the model and its ability to distinguish 5 separate constructs. A Multifaceted Rasch Measurement (MFRM) model assessed student performance and estimated the sources of measurement error attributed to 3 facets: student ability, rater stringency, and station difficulty. At the conclusion, students were surveyed about the capstone MMI experience.

RESULTS

The PCA confirmed the MMI reliably assessed 5 unique constructs and performance on each station was not strongly correlated with one another. The 3-facet MFRM analysis explained 58.79% of the total variance in student scores. Specifically, 29.98% of the variance reflected student ability, 20.25% reflected rater stringency, and 8.56% reflected station difficulty. Overall, the data demonstrated an acceptable fit to the MFRM model. The majority of students agreed the MMI allowed them to effectively demonstrate their communication (80.82%), critical thinking (78.77%), and collaboration skills (70.55%).

CONCLUSIONS

The MMI can be a valuable assessment strategy of professional competence within a health professions curriculum. These findings suggest the MMI is well-received by students and can produce reliable results. Future research should explore the impact of using the MMI as a strategy to monitor longitudinal competency development and inform feedback approaches.