Faculty Process for Reviewing and Utilizing a School's Course Evaluation Comments

OBJECTIVE

This study aimed to explore faculty engagement with qualitative comments from course evaluations.

METHODS

Course faculty from the University of North Carolina Eshelman School of Pharmacy were recruited via email to participate in a 30-minute interview session. Previous course evaluation comments were adapted to create a de-identified mock evaluation. Six interviews were conducted via Zoom, consisting of a think-aloud protocol based on the mock course evaluation followed by a cognitive interview focused on goals and current utilization of comments, and common patterns and issues sought by faculty. Interview transcripts were manually cleaned and de-identified. Transcripts were inductively coded by 1 researcher using MAXQDA.

RESULTS

Three overarching themes were identified: general faculty process for reviewing comments (ie, how faculty perceive and analyze comments), comments utilization for course change (ie, how faculty utilize comments in making course changes), and faculty analysis strategy (ie, faculty approach to locating common patterns in evaluation comments). The most common subthemes included usefulness of comments, methods for tracking comment patterns, and challenges with the large number of comments each semester.

CONCLUSION

Faculty provided useful insight and feedback regarding the current state of the course evaluation process that can be used to improve the structure, organization, and utilization of course evaluations by faculty. These findings could inform the creation of the course evaluation comment automated analysis program in the next stage of an ongoing project.

Automated Analysis of Preceptor Comments: A Pilot Study Using Sentiment Analysis to Identify Potential Student Issues in Experiential Education

OBJECTIVE

The purpose of this paper is to describe a sentiment analysis program that aids in identifying pharmacy students at risk for progression issues by automatically scoring preceptor comments as positive or negative.

METHODS

An R-based program to analyze advanced pharmacy practice experiences and introductory pharmacy practice experiences midpoint evaluation of preceptor comments was piloted in phase 1 by comparing the sentiment analysis algorithm results to human coding. The algorithm was refined in phase 2. In phase 3, the validation phase, the final sentiment analysis algorithm analyzed all midpoint student evaluations (n = 1560). Sentiment scores were generated for each preceptor comment, and correlations were performed between sentiment scores and the quantitative scoring provided on the assessment.

RESULTS

In phase 1, agreement between faculty coders and sentiment analysis was 96%, and in phase 2, agreement between the final codes and sentiment analysis was 92.4% once keywords were added to the sentiment dictionary. In phase 3, a total of 3919 comments from 1560 evaluations were analyzed, and overall, the sentiment analysis results aligned with the quantitative data.

CONCLUSION

This sentiment analysis algorithm was accurate in capturing positive and negative comments corresponding to pharmacy student performance. Given the accuracy of this preliminary validation for flagging preceptor comments, there are numerous implications when considering the use of sentiment analysis in pharmacy education. Using a sentiment analysis program minimizes the number of qualitative preceptor comments needing review by experiential faculty, as this program can aid in identifying students at risk of progression issues.

Simulating patterns of life: More representative time-activity patterns that account for context

BACKGROUND

Complex contributions of environment to health are intimately connected to human behavior. Modeling of human behaviors and their influences helps inform important policy decisions related to critical environmental and public health challenges. A typical approach to human behavior modeling involves generating daily schedules based on time-activity patterns of individual humans, simulating 'agents' with these schedules, and interpreting patterns of life that emerge from the simulation to inform a research question. Current behavior modeling, however, rarely incorporates the context that surrounds individuals' truly broad scope of activities and influences on those activities.

OBJECTIVES

We describe in detail a range of elements involved in generating time-activity patterns and connect work in the social science field of behavior modeling with applications in exposure science and environmental health. We propose a framework for behavior modeling that takes a systems approach and considers the broad scope of activities and influences required to simulate more representative patterns of life and thus improve modeling that underlies understanding of environmental contributions to health and associated decisions to promote and protect public health.

METHODS

We describe an agent-based modeling approach reliant on generating a population's schedules, filtering the schedules, simulating behavior using the schedules, analyzing the emergent patterns, and interrogating results that leverages general empirical information in a systems context to inform fit-for-purpose action.

DISCUSSION

We propose a centralized and standardized program to codify behavior information and generate population schedules that researchers can select from to simulate human behavior and holistically characterize human-environment interactions for a variety of public health applications.

Recognition of Culturally Diverse Cuisine in Popular Nutrition Apps

Objectives

Nutrition apps have been designed to assist users in determining the nutritional value of their food in alliance with their dietary restrictions and/or health. However, many of these apps do not incorporate or recognize the cuisine of the diverse communities that they serve. In this work we assessed several of the most popular free apps and compared with the most popular dishes in Latin America, Europe, India, East Asia, West Africa, and the American South to determine the rate of recognition of culturally diverse cuisines in popular nutrition apps.

Methods

We built a consensus database of over 150 culturally popular foods via a web scraper for foods from Latin American, Europe, India, East Asia, West Africa, and the American South. Researched & logged popular dishes from International cuisines with their corresponding nutritional data. Each food item was run against Calorie Mama, Lose It, My Fitness Pal and My Plate to determine the recognition and availability of the nutritional value.

Results

European and Indian dishes were better recognized (greater than 70%) across all apps than West African and Latin American dishes. Of the dishes that were recognizable, most were due to manual entry and lacked some of the nutritional information, making the apps less useful.

Conclusions

We note a bias among popular nutrition apps in not taking into account cuisines, from both international and regional American communities, leaving many communities underserved and contributing to health inequalities gap.

Funding Sources

N/A.

Biolink Model: A universal schema for knowledge graphs in clinical, biomedical, and translational science

Within clinical, biomedical, and translational science, an increasing number of projects are adopting graphs for knowledge representation. Graph‐based data models elucidate the interconnectedness among core biomedical concepts, enable data structures to be easily updated, and support intuitive queries, visualizations, and inference algorithms. However, knowledge discovery across these “knowledge graphs” (KGs) has remained difficult. Data set heterogeneity and complexity; the proliferation of ad hoc data formats; poor compliance with guidelines on findability, accessibility, interoperability, and reusability; and, in particular, the lack of a universally accepted, open‐access model for standardization across biomedical KGs has left the task of reconciling data sources to downstream consumers. Biolink Model is an open‐source data model that can be used to formalize the relationships between data structures in translational science. It incorporates object‐oriented classification and graph‐oriented features. The core of the model is a set of hierarchical, interconnected classes (or categories) and relationships between them (or predicates) representing biomedical entities such as gene, disease, chemical, anatomic structure, and phenotype. The model provides class and edge attributes and associations that guide how entities should relate to one another. Here, we highlight the need for a standardized data model for KGs, describe Biolink Model, and compare it with other models. We demonstrate the utility of Biolink Model in various initiatives, including the Biomedical Data Translator Consortium and the Monarch Initiative, and show how it has supported easier integration and interoperability of biomedical KGs, bringing together knowledge from multiple sources and helping to realize the goals of translational science.

Interactive Narrative Simulation as a Method for Preceptor Development

(1) Background: This proof-of-concept study assessed an interactive web-based tool simulating three challenging non-academic learning situations—student professionalism, cross-cultural interactions, and student well-being—as a means of preceptor development. (2) Methods: Three scripts focused on professionalism, cross-cultural interactions, and student well-being were developed and implemented using a commercial narrative tool with branching dialog. Delivered online, this tool presented each challenge to participants. Participants had up to four response options at each turn of the conversation; the choice of response influenced the subsequent conversation, including coaching provided at the resolution of the situation. Participants were invited to complete pre-activity, immediate post-activity, and one-month follow-up questionnaires to assess satisfaction, self-efficacy, engagement, and knowledge change with the tool. Knowledge was assessed through situational judgment tests (SJTs). (3) Results: Thirty-two pharmacist preceptors participated. The frequency of participants reflecting on challenging learning situations increased significantly one-month post-simulation. Participants affirmatively responded that the tool was time-efficient, represented similar challenges they encountered in precepting, was easily navigable, and resulted in learning. Self-efficacy with skills in managing challenging learning situations increased significantly immediately post-simulation and at a one-month follow-up. Knowledge as measured through SJTs was not significantly changed. (4) Conclusions: Preceptors found an interactive narrative simulation a relevant, time-efficient approach for preceptor development for challenging non-academic learning situations. Post-simulation, preceptors more frequently reflected on challenging learning situations, implying behavior change. Self-efficacy and self-report of knowledge increased. Future research is needed regarding knowledge assessments.

A Concept Mapping Activity to Enhance Pharmacy Students' Metacognition and Comprehension of Fundamental Disease State Knowledge

Objective. To examine the impact of pre-class concept mapping activities on pharmacy students' ability to self-assess their degree of foundational disease state knowledge and predict their pre-class quiz performance.Methods. Second year pharmacy students in a problem-based learning course were responsible for self-directed learning of foundational knowledge for 14 disease states. After completing their independent pre-class reading, students worked in groups to create concept maps for which feedback was provided for four laboratory sessions, worked in groups to create concept maps but received no formal feedback for three laboratory sessions, and did not engage in any formal group activity for seven laboratory sessions. The day following each session, prior to the formal in-class discussion, students were asked to predict the number of questions they could answer correctly on a quiz covering foundational knowledge and then completed the quiz. Quiz performance was compared based on the three conditions, and bias and absolute bias were calculated to evaluate students' metacognitive skills.Results. There was no difference in pharmacy students' metacognition based on the conditions, as reflected by inaccuracy between predicted and actual quiz scores. However, when students had engaged in concept mapping the previous day, their quiz performance was significantly higher than when they had not.Conclusion. Concept mapping did not improve pharmacy students' metacognitive skills but did have a small effect on their quiz performance. More research is needed to tease apart the roles of concept mapping, group activity, and feedback in altering pharmacy students' quiz performance and metacognitive skills.

Testing of a Program to Automatically Analyze Students' Concept Maps

Concept maps are graphical representations of how various concepts relate to one another. Assessment of concept maps developed by students in the pharmacy curriculum helps to evaluate student understanding of course material. However, providing feedback on concept maps can be time-consuming and often requires the grader to be a content expert. The purpose of this study was to develop and validate a software program to provide students with feedback on their concept map performance. Student maps for four different disease states were compared against expert concept maps. The analysis of the program compared favorably to a manual assessment of student maps for the maps' complexity and content but did not correlate for their organization. The value of using a software program to quickly and efficiently analyze concept maps is discussed.

Identifying Priority Student Leadership and Professionalism Attributes Among Faculty, Preceptors, and Students via Modified Delphi

Objective. To identify and build consensus on priority leadership and professionalism attributes for pharmacy student development among faculty, preceptors, and students.Methods. One hundred individuals (27 faculty members, 30 preceptors, 43 students) were invited to participate in a three-round, modified Delphi. Published literature on leadership and professionalism informed the initial attribute list. In the first round, participants reviewed and provided feedback on this list. In the second round, participants prioritized attributes as highly important, important, or less important for pharmacy student development. Leadership and professionalism attributes that achieved an overall consensus (a priori set to ≥80.0%) of being highly important or important for pharmacy student development were retained. In the third round, participants rank ordered priorities for leadership and professionalism attributes.Results. Fifteen leadership and 20 professionalism attributes were included in round one while 21 leadership and 21 professionalism attributes were included in round two. Eleven leadership and 13 professionalism attributes advanced to round three. Consensus was reached on the top four leadership attributes (adaptability, collaboration, communication, integrity) and five professionalism attributes (accountability, communication, honor and integrity, respect for others, trust). Differences were observed for certain attributes between faculty members, preceptors, and/or students.Conclusion. The modified Delphi technique effectively identified and prioritized leadership and professionalism attributes for pharmacy student development. This process facilitated consensus building and identified gaps among stakeholders (ie, faculty, preceptors, students). Identified gaps may represent varying priorities among stakeholders and/or different opportunities for emphasis and development across classroom, experiential, and/or cocurricular settings.

Serious Assessments in Serious Games

In the past decade the serious games initiative has produced a number of games being used today as adjuncts to institutional training. Many of the games produced through the serious games initiative do not incorporate adaptive assessment to assure the acquisition of real-world skills from virtual training. Detailed performance assessment techniques are often lacking. Successful performance assessment in virtual training requires that students are placed into multiple simulated contexts and are challenged with different tasks to perform under various conditions, to specified standards. The aggregate of training situations must adequately cover the space of real-world situations. This article discusses important concepts related to virtual training performance assessment including critical tasks, performance criteria, forward recommendation, errors of commission, red screen alerts, reusable competency definitions, ill-structured domains, violent domains, and other game design templating. By integrating these concepts into serious game design, trainers can develop sufficiently varying simulated tasks to ensure that serious games meet real-world adaptive needs.

PHIT for duty, a mobile approach for psychological health intervention

The goal of this effort is to support prevention of psychological health problems through innovation in mobile personal health assessment and self-help intervention (SHI). For the U.S. military, we are developing and evaluating a field-deployable personalized application, PHIT for DutyTM, to help build resilience in healthy troops and support prevention in high-risk personnel. PHIT for Duty is delivered using any smartphone or tablet with optional nonintrusive physiological and behavioral sensors for health status monitoring. The application integrates a suite of health assessments with an intelligent advisor that recommends, tailors, and presents self-help advisories. PHIT for Duty is intended for secondary prevention of psychological health problems in persons who have been exposed to psychological trauma and may be showing some symptoms of distress, but have not been diagnosed with any psychological disease or disorder.

Deficits in behavioral inhibition predict treatment engagement in prison inmates

Many inmates do not respond favorably to standard treatments routinely offered in prison. Executive cognitive functioning and emotional regulation may play a key role in treatment responsivity. During intake into treatment, inmates (N = 224) were evaluated for executive functioning, emotional perception, stress reactivity (salivary cortisol), IQ, psychological and behavioral traits, prior drug use, child and family background, and criminal histories and institutional behavior. Outcome measures included program completion, treatment readiness, responsivity and gain, and the Novaco Reaction to Provocation Questionnaire. Relative deficits in behavioral inhibition significantly predicted treatment outcomes, more so than background, psychological, or behavioral variables, and other neurocognitive and emotional regulatory measures. Future replications of these results have potential to improve assessment and treatment of offenders who are otherwise intractable.

Virtual simulation-enhanced triage training for Iraqi medical personnel

Triage, establishing the priority of care among casualties in disaster management, is generally practiced using constructive tabletop or live exercises. Actual disasters involving multiple casualties occur rarely, offering little opportunity for gaining experience and competency assessment. When they do occur, response needs to be rapid and well-learned. In the Iraqi medical education environment where the need for triage is immediate, but the ability to stage practice is nearly impossible, blending didactic learning with simulation-based triage offers an alternative training methodology.

Mediators of the stress-substance-use relationship in urban male adolescents

Exposure to chronic or severe acute stressors throughout the lifespan has been linked with numerous negative behavioral, emotional, cognitive, and physical consequences. Adolescence is considered to be a particularly vulnerable period given that the brain is experiencing dramatic developmental change during this time. The present study examined a sample of adolescents (N=125) considered to be at high risk for stress exposures and drug use by virtue of their environment and low income levels to identify possible neurocognitive (i.e., impulsivity, delay of gratification, emotional perception, and risky decision-making) and social competency mechanisms that may mediate this relationship. Using Mplus, a mediational model was tested using full information maximum likelihood estimates. Risky decision-making and poor social competency skills were related to previous stressful experiences; however, only social competencies mediated the effect of stressors on reports of past year marijuana, alcohol, and polydrug use. As such, stress appears to exert its negative impact through alterations in abilities to generate and execute prosocial decisions and behaviors. Interventions that directly address the effects of stress on social competencies may be especially important for children who have experienced adversity including those exposed to parental divorce, parental psychopathology, neglect or abuse, parental death, and poverty.

Neurocognitive skills moderate urban male adolescents' responses to preventive intervention materials

The present experiment was designed to determine whether individual variation in neurobiological mechanisms associated with substance abuse risk moderated effects of a brief preventive intervention on social competency skills. This study was conducted in collaboration with the ongoing preventive intervention study at Johns Hopkins University Prevention Intervention Research Center (JHU PIRC) within the Baltimore City Public Schools. A subsample (N = 120) of male 9th grade students was recruited from the larger JHU study population. Approximately half of the participants had a current or lifetime diagnosis of CD while the other half had no diagnosis of CD or other reported problem behaviors. Measures of executive cognitive function (ECF), emotional perception and intelligence were administered. In a later session, participants were randomly assigned to either an experimental or control group. The experimental group underwent a facilitated session using excerpted materials from a model preventive intervention, Positive Adolescent Choices Training (PACT), and controls received no intervention. Outcomes (i.e., social competency skills) were assessed using virtual reality vignettes involving behavioral choices as well as three social cognition questionnaires. Poor cognitive and emotional performance and a diagnosis of CD predicted less favorable change in social competency skills in response to the prevention curriculum. This study provides evidence for the moderating effects of neurocognitive and emotional regulatory functions on ability of urban male youth to respond to preventive intervention materials.

The virtual pediatric standardized patient application: formative evaluation findings

This paper presents formative (i.e., not final project) evaluation data from the use of a responsive virtual human training application by medical students rotating through Pediatrics and by Pediatric medical educators. We are encouraged by the evaluation results and believe the iterative development strategies employed and the subsequent refinements in the scenarios will lead to important instructional and assessment tools for medical educators.