Impact of high-stakes exams, incentives, and preparation on student performance: Insights from the pharmacy curriculum outcomes assessment

INTRODUCTION

Efforts to ensure success of pharmacy students in passing pharmacy standardized exams require substantial investments. Engaging students effectively can be a challenge when there are no consequences for non-participation or poor performance. This study examined how engagement reinforcement, including high-stake exam requirements, instructional strategies, and incentives, impacted student performance on the Pharmacy Curriculum Outcomes Assessment (PCOA).

METHODS

PCOA scores, milestone exams, grade point averages (GPAs), and PCOA preparedness assessments for cohorts (Co) that received high-stakes exams, incentives, and preparation (Co2019, Co2020, and Co2021) was compared with those that did not receive these interventions (Co2017 and Co2018). Students' perceptions regarding reinforcement, incentive, and preparedness were evaluated using an anonymous survey.

RESULTS

Analyzing data from 545 students over five years, mandated PCOA preparedness, high-stakes PCOA requirements, and incentives for Co2019, Co2020, and Co2021 improved scores by 11% to 18% compared to Co2017 and Co2018. This corresponded to a rise in performance from the 12th to 27th percentile for Co2017 and Co2018 to the 39th to 49th percentile for Co2019, Co2020, and Co2021. In these later cohorts, PCOA scores consistently correlated with the school's milestone exams and students' cumulative GPAs (correlation coefficients 0.47-0.70, P < .001), while no such correlation was observed in Co2017 and Co2018. Faculty-led PCOA preparation yielded better results (48.2% in Co2020, 45.8% in Co2021) than self-learning (42% in Co2019). Students using faculty-prepared assessments reported increased confidence in biomedical and pharmaceutical sciences.

CONCLUSIONS

This study highlights the importance of high-stakes requirements, incentives, and thorough preparation in improving PCOA results.

Methicillin-resistant Staphylococcus aureus and Pseudomonas aeruginosa community acquired pneumonia: Prevalence and locally derived risk factors in a single hospital system

Objectives

Current American Thoracic Society/Infectious Disease Society of America (ATS/IDSA) community-acquired pneumonia (CAP) guidelines expand the CAP definition to include infections occurring in patients with recent health care exposure. The guidelines now recommend that hospital systems determine their own local prevalence and predictors of Pseudomonas aeruginosa and methicillin-resistant Staphylococcus aureus (MRSA) among patients satisfying this new broader CAP definition. We sought to carry out these recommendations in our system, focusing on the emergency department, where CAP diagnosis and initial empiric antibiotic selection usually ooccur.

Methods

We performed a retrospective cohort study of patients admitted with CAP through any of 3 EDs in our hospital system in Northern California between November 2019 and October 2021. Inclusion criteria included an ED admission diagnosis of pneumonia or sepsis, fever or hypothermia, leukocytosis or leukopenia, and consistent chest imaging result. SARS-CoV-2-positive cases were excluded. We abstracted variables historically associated with P. aeruginosa and MRSA. Outcome measures were prevalence of P. aeruginosa and MRSA in the overall clinically defined cohort and among microbiologically confirmed cases and predictors of P. aeruginosa or MRSA isolation, as determined by univariate logistic regression, bootstrapped least absolute shrinkage and selection operator, and random forest analyses. Additionally, we describe the iterative process used and challenges encountered in carrying out the new ATS/IDSA guideline recommendations.

Results

There were 1133 unique patients who satisfied our definition of clinically defined CAP, of whom 109 (9.6%) had a bacterial pathogen isolated. There were 24 P. aeruginosa isolates and 11 MRSA isolates in 33 patients. Thus, the prevalence P. aeruginosa and MRSA was 2.9% in the overall CAP cohort, but 30.3% in the microbiologically confirmed cohort. The most important predictors of either P. aeruginosa or MRSA isolation were tracheostomy (odds ratio [OR] 22.08; 95% confidence interval [CI] 10.39-46.96) and gastrostomy tube (OR 14.7; 95% CI 7.14-30.26). Challenges included determining the suspected infection type in patients admitted simply for "sepsis"; interpreting dictated radiology reports; determining functional status, presence of indwelling lines and tubes, and long-term care facility residence from the electronic health record; and correctly attributing culture results to pneumonia.

Conclusion

Prevalence of MRSA and P. aeruginosa was low among patients admitted in our medical system with CAP - now broadly defined - but high among those with a microbiologically confirmed bacterial etiology. Our locally derived predictors of MRSA and P. aeruginosa can be used to aid our emergency physicians in empiric antibiotic selection for CAP. Findings from this project might inform efforts at other institutions.

Effectiveness of an Advanced Naloxone Training, Simulation, and Assessment of Second-Year Pharmacy Students

Background: Opioid overdoses continue to be one of the most urgent public health priorities. In 2020, reported overdose deaths in the United States reached a high of over 93,000 cases. As the COVID-19 pandemic and opioid crisis continues to be addressed, life-saving agents must be more widely accessible to those with a high overdose risk. An essential step to increasing access is to train student pharmacists to dispense naloxone. Once licensed, the number of personnel authorized to dispense naloxone can increase. Objectives: To design a training program to educate second-year pharmacy (P2) students on furnishing naloxone under a state protocol. Methods: A multi-phased curriculum-based naloxone training program was delivered to P2 students and included lecture-based education, team-based learning (TBL) applications, case-based scenarios, and summative assessments to improve student knowledge and confidence in furnishing naloxone. Students were surveyed on their knowledge and confidence with naloxone prior to training, after the in-class training and TBL applications and after three assessments. Assessments included simulated patient counseling, case-based scenarios, and proper dispensing of naloxone in a community pharmacy simulation lab. Results: A total of 185 student pharmacists completed the naloxone training program and 68 completed all three surveys. Average scores for naloxone assessments were 83% for the APPS lab patient case, 90.5% for the prescription label typed for the naloxone product, and 88.5% for patient counseling. Statistically significant increases in knowledge-based quiz-like scores (42.1% after training vs. 7.2% after assessment) and in the proportion of students affirmatively answering survey questions after training and assessment was observed. Conclusion: Multi-phase curriculum-based naloxone training program improved pharmacy student knowledge and confidence in furnishing naloxone under a state BOP protocol.

A multicenter study of gender bias in student evaluations of teaching in pharmacy programs

INTRODUCTION

Student evaluations of teaching (SET) are widely used to assess effectiveness of teaching. Studies conducted to assess the presence of gender bias in SET have produced mixed results. The purpose of this study was to evaluate the presence and degree of gender bias in SET of didactic courses in United States pharmacy programs.

METHODS

A three-year, retrospective, multi-institutional analysis of anonymous SET data were analyzed from required didactic courses. Analysis included gender, SET questions, mean scale score, and number of students responding to each question. A hierarchical linear model was used to compare the gender difference with the normalized SET scores as the outcome.

RESULTS

A total of 2114 SET scores were included from seven pharmacy schools across eight campuses. Analysis of the results revealed that the combined data were skewed secondary to one institution whose results fell significantly outside the mean. When this school was excluded, the difference between SET scores did not differ by gender, b = 0.021, t(1,702) = 0.69, P = .49, with similar SET scores for female faculty (mean = 4.41, SD = 0.35, range = 2.54-5) and male faculty (mean = 4.44, SD = 0.32, range = 2.67-5).

CONCLUSIONS

After secondary analysis, the aggregated data showed no significant difference between ratings of male and female instructors. However, there were differences within individual programs. This illustrates the importance of applying assessment principles to SET to determine the presence of bias so that continuous quality improvement strategies may be applied.

Clinical Characteristics, Outcomes and Prognostic Factors for Critical Illness in Hospitalized COVID-19 Patients in Saudi Arabia: A Retrospective Cohort Study

Background

A good understanding of the possible risk factors for coronavirus disease 19 (COVID‐19) severity could help clinicians in identifying patients who need prioritized treatment to prevent disease progression and adverse outcome. In the present study, we aimed to correlate clinical and laboratory characteristics of hospitalized COVID-19 patients to disease outcome in Saudi Arabia.

Materials and Methods

The present study included 199 COVID-19 patients admitted to King Fahd Specialist Hospital, Buraydah, Qassim, Saudi Arabia, from April to December 2020. Patients were followed-up until discharge either for recovery or death. Demographic data, clinical data and laboratory results were retrieved from electronic patient records.

Results

Critical COVID-19 cases showed higher mean of age and higher prevalence of co-morbid conditions. Fifty-five patients died during the observation period. Risk factors for in hospital death for COVID 19 patients were leukocytosis (OR 1.89, 95% CI 1.008–3.548, p = 0.081), lymphocytopenia (OR 2.152, 95% CI 1.079–4.295, p = 0.020), neutrophilia (OR 1.839, 95% CI 0.951–3.55, p = 0.047), thrombocytopenia (OR 2.152, 95% CI 0.852–5.430, p = 0.085), liver injury (OR 2.689, 95% CI 1.373–4.944, p = 0.003), acute kidney injury (OR 1.248, 95% CI 0.631–2.467 p = 0.319), pancreatic injury (OR 1.973, 95% CI 0.939–4.144, p = 0.056) and high D dimer (OR 2.635, 95% CI 0.747–9.287, p = 0.091).

Conclusion

Clinical and laboratory data of COVID-19 patients may help understanding the pathogenesis of the disease and subsequently improve of the outcome of patients by determination of the associated risk factors and recognition of high risk group who are more liable for complications and in hospital death. The present study put an eye on some parameters (laboratory and clinical) that should be alarming signs that the patient is at high risk bad prognosis.

Impact of a Prepharmacy Program on Students' Self-Awareness of Pharmacist Professional Identity: Comparison between Virtual and In-Person Settings

Ensuring an adequate preparation for undergraduate students transitioning to pharmacy school is challenging. A significant barrier is changing from a subordinate to a critical thinking mindset while self-identifying as a professional. Here, we aimed to (1) determine whether our prepharmacy program called “Professional Identity and Me” (PRIME) could enhance learners’ self-awareness of their professional identity and (2) compare the effectiveness of the in-person and online versions of PRIME. PRIME introduced prepharmacy students to aspects of pharmacists’ professional identity including community, hospital, and interprofessional work, as well as mental health, wellness, and time and stress management skills, Top-200 drugs, prerequisite foundational sciences, and calculations. Concepts of professionalism, graduate writing, and ownership were also presented. Bridging exercises were introduced to exemplify application. We used a mixed-methods approach to assess the outcomes. The average performance in knowledge-based assessments increased before and after the PRIME program from 53.8 to 74.6% and from 47.7 to 75.9%, while the difference in the test scores was statistically significant, with a 21% increase (p < 0.001, 95% CI 15−26%) and a 28% improvement (p < 0.001, 95% CI 23−34%) for face-to-face versus virtual PRIME. The results of a student perception survey revealed PRIME was equally effective as a virtual program during the COVID-19 pandemic, suggesting transferability to other pharmacy programs.

Use of End-of-Class Quizzes to Promote Pharmacy Student Self-Reflection, Motivate Students to Improve Study Habits, and to Improve Performance on Summative Examinations

Underperforming students are often unaware of deficiencies requiring improvement until after poor performance on summative exams. The goal of the current study was to determine whether inclusion of individual end-of-class formative quizzes, which comprise of higher level Bloom's questions, could encourage students to reflect on and address deficiencies and improve academic performance. Ninety-seven out of 123 first-year pharmacy students (79%) enrolled in a Biochemistry and Cell & Molecular Biology course participated in a single-blinded, randomized, controlled, crossover study. Paired t-test analyses demonstrated that that implementation of individual end-of-class formative quizzes resulted in significantly higher summative exam scores for below average students (p = 0.029). Notably, inclusion of quizzes significantly improved performance on higher Bloom's questions for these students (p = 0.006). Analysis of surveys completed by students prior to summative exam indicate that the formative end-of-class quizzes helped students identify deficiencies (89%) and making them feel compelled to study more (83%) and attend review sessions (61%). Many students indicated that quizzes increased stress levels (45%). Our collective data indicate that quizzes can improve summative exam performance for below average first year pharmacy students, and improve self-reflection and student motivation to study. However, the impact on student stress levels should be considered.

The impact of compliance to oncology pathways that include G-CSF rules for use on ER visits/hospitalizations in a multistate program

135

Background: CareFirst BCBS initiated a multistate oncology pathway program in 2008, which reduced chemotherapy and hospitalization expenditures and also increased supportive care drug utilization. This study evaluated the impact of following pathways including supportive care versus not following pathways on ER visits/hospitalizations and associated costs as a result of anemia, neutropenia, and chemotherapy-induced nausea and vomiting (CINV). Methods: The CareFirst claims database was utilized to evaluate data two years after initiation of the pathways program. Frequency of ER visits/hospitalizations for neutropenia, anemia and CINV was the primary outcome measured and was compared between compliant and noncompliant pathway utilization of granulocyte colony-stimulating factors (G-CSFs), erythropoiesis-stimulating agents (ESAs), and antiemetics, respectively. Logistic regression analyses were used to control for expenditures of ESAs, CSFs, and antiemetics in the context of compliance and the corresponding visit types. Costs associated with ER visits/hospitalizations were also compared between compliant and noncompliant pathway utilization of supportive care. Results: Overall, 46 sites with 1,586 patients were evaluated. There were a total of 12 ER visits/hospitalizations for anemia, 231 for neutropenia, and 360 for CINV. G-CSF pathway compliance resulted in a relative reduction in ER visits/hospitalizations of 67.2% (from 28.3% to 11.5%, p<0.001). Logistic regression showed consistent results when controlling for costs with a 69.5% relative reduction of ER visits/hospitalizations (p<0.001). A comparison of ER visits/hospitalization costs as a result of neutropenia demonstrated G-CSF pathway compliance producing an average savings of $1,173 per line of therapy (p=0.001). Analogous analyses did not demonstrate a significant relationship between ESA and antiemetic utilizations and their corresponding ER visits/hospitalizations. Conclusions: Adherence to oncology pathways, including G-CSF use rules, is associated with a significantly lower ER visits/hospitalization rates and costs associated with febrile neutropenia.

Second-year results of the first large-scale, multistate BCBS clinical oncology pathway program

e16522

Background: Clinical pathways (CP) are viewed as valuable practice tools leading to presumed cost savings, however none have been fully implemented on a state-wide scale. CareFirst BCBS partnering with P4 Pathways implemented a multi-state oncology CP program in Aug., 2008. The CP included both chemotherapy (chemo) and supportive care (SC). Our study goals were to determine year (yr) 2 physician (MD) compliance (compl) to CP, their effects on drug and hospitalization (hosp) cost, and their ability to curb projected cost increases. Methods: This study used a retrospective single-group, pretest-posttest design. Data representing pre-CP yr -1 (2007-08) and two years after CP initiation, yr +1 and +2 (08-10) were obtained from claims data. The dataset contained breast, colorectal and lung cancer. One claim from each of the 3 yrs was needed for site eligibility. First and 2nd yr compl was defined as 65% and 80%, respectively, use of chemo regimens according to CP by site. Compl for SC was defined as 80% use of agents per CP for each year. Drug and hosp (occurring within 30 days of chemo regimen) cost were calculated. Savings were determined by extrapolating results on a per patient basis and comparing them to projections cost increases for yr +2. Results: Overall, 46 sites representing 193 MD’s, 4713 pts and 110,203 claims met inclusion criteria. The uncorrected site compl rate for chemo was 83% and 54% for yrs +1 and +2 respectively, while SC site compl was 74% for both yrs. Compared to yr -1, per patient drug expenditures ↑ by 5.8% in yr +1 and 2.7% in yr +2. Per patient hosp costs ↓ by 4.2% in yr +1 and 8.3% in yr +2 compared to yr -1. Total hosp costs ↓ by 28.4% in yr +1 and 63.4% in yr +2 compared to yr -1. Using national data to project drug cost ↑ of 14.5% and hosp cost ↑ of 9.5% yearly over yrs +1 and +2, the CP provided an estimated $36 million in savings by yr +2, with $23 million due to drug and $13 million due to hosp. An absolute $7+ million savings for yr +2 would still have been seen if we assume no ↑ in drug & hosp spending over the 2 yr period. Conclusions: Reasonable MD compliance can be achieved on broad scale use of CP. CP reduce chemotherapy and hosp expenditures, although an increase use in supportive care was seen. Total costs savings can be substantial.