Multiple rubric-based assessments of student case presentations

Undergraduates' comparative assessment, utilizing a structured rubric, of hands-on versus video-supported presentation in class II cavity preparation

BACKGROUND

Dental education is considered as a complex, challenging and often stressful educational procedure. Acquisition of psychomotor skills by undergraduate students is an important step in many health professions to become a successful professional. During under graduation, class II cavity preparation exercise is of utmost important in dentistry.

OBJECTIVE

To compare class II cavities prepared by students after hands-on live demonstration and pre-recorded video demonstration using well-organised evaluation rubrics.

METHOD

Preclinical dental students (n= 50) were divided into two groups. The students in group I (n= 25) attended a hands-on live demonstration performed by one faculty while students in group II (n= 25) watched a 15-minute pre-recorded procedural video on the projector. Both groups were appealed to prepare class II cavity for amalgam involving disto-occlusal surface of mandibular second molar articulated on jaw model (TRU LON study model, Jayna industries, Ghaziabad U.P., India). Following completion of the preparations, all teeth were collected, and labelled grades of prepared cavities were given according to prespecified rubrics. The data of scores were presented as means and standard deviation. Statistical analysis of data was executed using SPSS software. A paired t-test was used to compare scores between groups.

RESULTS

The study shows that the video-supported demonstration of a cavity preparation was better than the live hands-on demonstration. A higher mean response for the procedural video group was found compared to the live demonstration group (p= 0.000133).

CONCLUSION

Pre-recorded video-supported demonstration along with guidance by a tutor may be a viable alternative to hands-on live demonstration in cavity preparation procedures during undergraduate dental training. Moreover, rubric methods can be implemented in the teaching of various preclinical exercises for conservative dentistry and endodontics.

Peer overmarking and insufficient diagnosticity: the impact of the rating method for peer assessment

The present study explores two rating methods for peer assessment (analytical rating using criteria and comparative judgement) in light of concurrent validity, reliability and insufficient diagnosticity (i.e. the degree to which substandard work is recognised by the peer raters). During a second-year undergraduate course, students wrote a one-page essay on an air pollutant. A first cohort (N = 260) relied on analytical rating using criteria to assess their peers' essays. A total of 1297 evaluations were made, and each essay received at least four peer ratings. Results indicate a small correlation between peer and teacher marks, and three essays of substandard quality were not recognised by the group of peer raters. A second cohort (N = 230) used comparative judgement. They completed 1289 comparisons, from which a rank order was calculated. Results suggest a large correlation between the university teacher marks and the peer scores and acceptable reliability of the rank order. In addition, the three essays of substandard quality were discerned as such by the group of peer raters. Although replication research is warranted, the results provide the first evidence that, when peer raters overmark and fail to identify substandard work using analytical rating with criteria, university teachers may consider changing the rating method of the peer assessment to comparative judgement.

Evaluating the impact of a "grand rounds" elective course on students' perceived motivations, comfort, and confidence in providing constructive and reinforcing feedback to guest lecturers

BACKGROUND AND PURPOSE

Providing feedback is an important skill for all healthcare professionals both within and outside of their discipline. Although student pharmacists frequently receive feedback during both didactic and experiential education, training on how to provide feedback to others is less common.

EDUCATIONAL ACTIVITY AND SETTING

An elective was designed to expose second-year pharmacy students to "grand rounds" with practicing pharmacists as the presenters. Students provided feedback to presenters on presentation style and assessment questions. The primary objective of this research project was to determine if the elective improved students' motivations, comfort, and confidence in providing constructive written feedback.

FINDINGS

Over two course offerings, 54% (19 of 35) of enrolled students completed both the pre- and post-surveys. At baseline, the majority of students self-identified as being motivated, comfortable, and confident with providing quality written feedback with the exception of two specific areas: motivation to provide quality written feedback and comfort with providing difficult or sensitive written feedback. At the end of the course, the majority of students self-identified as being motivated, comfortable, and confident across all areas queried. All students agreed or strongly agreed that the efficiency and quality of their written feedback improved during the course.

SUMMARY

The course offered several benefits to students, including learning clinical topics from a variety of presenters and developing feedback skills. The implementation of the grand rounds elective provided students an opportunity to develop their motivation, comfort, and confidence with providing quality constructive written feedback.

Effectiveness of Peer Evaluation in Learning Process: A Case from Dental Technology Students

Background:

Peer assessment or evaluation has been shown to play an important role in several learning processes. However, the effectiveness of this educational evaluation method has yet to be studied among students in different specialties.

Objective:

The study aimed to assess the effectiveness of peer evaluation as a method of student assessments in dental technology courses.

Methods:

The study sample (n=130 students) was randomly divided into 5 groups, then the effectiveness of peer evaluation was measured and correlated with student’s Grade Point Average (GPA) and gender.

Results:

The results showed the ability of students to evaluate their peers in a certain course. As per students’ perspective, scores ranged from 3.79-4.24 (out of 5), with a high degree of practice in all fields, ranked as follows: psychological stress, teamwork, self-efficacy, and, finally, the use of social media for learning. The mean score of all the categories in the peer assessment of students was 4.005 with a high degree of practice according to Scheffe's Test. Female gender was associated with significantly higher scores in the fields of psychosocial pressure, use of social media and overall scores. In addition, students achieving a GPA of 68 and more had significantly higher overall scores compared to students with lower GPAs.

Conclusion:

Peer evaluation seems to be a valid tool for students’ assessment; thus, it can be recommended as part of the course grading system in dental technology courses and can be applied to other dental or medical courses that involve practical sessions.

Experiential Education in Pharmacy Curriculum: The Lebanese International University Model

Experiential education is an essential component of pharmacy education in order to allow intern students to experience real conditions and training opportunities in different inpatient and outpatient settings. This paper provides a description of the pharmacy practice experiences (PPEs) in the Lebanese International University (LIU) 5-year bachelor of pharmacy (BPharm) and postgraduate doctor of pharmacy (PharmD) programs; focuses on the opportunities and challenges encountered; and presents a model for experiential education in Lebanon. Learning outcomes and thus students’ acquisition of predefined competencies are evaluated in actual practice settings through assessment tools. Our experiential education program aligns with the accreditation/certification criteria set by the Accreditation Council for Pharmacy Education (ACPE) and equips future pharmacists with the knowledge and skills to become major components in the healthcare team.

Evaluation of a peer- and self-grading process for clinical writing assignments

OBJECTIVES

As class sizes in pharmacy education increase, faculty must develop new assessment strategies for essay writing assignments. This study evaluated accuracy and student perceptions of an innovative grading process that utilizes both peer- and self-assessment.

METHODS

Four SOAP note sessions were evaluated. Each session included four activities: a writing workshop, assessment workshop, and reflection session. For each assessment workshop students scored their note and a blinded peer's note using a grading form, facilitated by a faculty-led discussion. In a subsequent reflection session, students reviewed their peer- and self-assigned grades and could petition for faculty review if desired. The average self-, peer-, and final-grades were compared for each of the four SOAP note sessions using ANOVA. After the fourth session, students completed an anonymous 10-question Likert-scale survey regarding their perceptions of the process and three open-response questions. Survey results were analyzed with descriptive statistics.

RESULTS

Approximately 140 students participated. No difference was found between the average self-, peer-, and final-grades for all four sessions (p > 0.05). The survey response rate was 65% (91/140). Survey questions were grouped into three themes. The majority of students either strongly agreed or agreed that sessions were well organized and effective (≥84%), assessment workshops enhanced learning (≥68%), and the scoring method was fair (≥72%). The lowest score (mean 2.53 on a 4-point scale) reflected satisfaction with peer-provided feedback. When asked what they liked most, respondents most commonly cited that faculty-led review and discussion enhanced clinical knowledge.

IMPLICATIONS

The combination of a peer- and self-assessment process was accurate, well-received, and can be used to decrease faculty workload.

Implementation of Oral Case Presentations in an Immunology Course

Implementation of oral case presentations (OCP) in the Immunology course at A.T. Still University-Kirksville College of Osteopathic Medicine has significantly improved written examination scores and student satisfaction with the course by enhancing its clinical relevance. With six faculty facilitators, an average class size of 172 students can complete the exercise in a single day. The exercise requires small group meeting rooms, each equipped with a computer and wall-mounted monitor, but no other physical resources.

Reliability of rubrics in the assessment of orthodontic oral presentation

Aims

The aim of this study was to evaluate the reliability of using rubrics in dental education, specifically for undergraduate students’ assessment in orthodontic oral presentation.

Methods

A rubric-based case presentation assessment form was introduced to three contributing instructors. In each instructor’s group, the course director, along with the assigned instructor, assessed 8 randomly selected fourth year male dental students utilizing the same assessment form (total of 24 students). The two final scorings made by the assigned instructor and the course director were then gathered for each student. The data of this prospective comparative study then was analyzed using paired t-test to look for any significant differences in the scoring of the course director and each instructor in each group.

Results

No significant statistical differences were detected in grading variables between the instructors and the course director. Furthermore, the data showed no significant correlations between the students’ final course grade, and their case presentation grades scored by instructors’/course director.

Conclusion

Despite the elaborate nature of the routine orthodontic case presentation, the use of rubrics was found to be a promising reliable assessment element.

Practical Team-Based Learning from Planning to Implementation

Team-based learning (TBL) helps instructors develop an active teaching approach for the classroom through group work. The TBL infrastructure engages students in the learning process through the Readiness Assessment Process, problem-solving through team discussions, and peer feedback to ensure accountability. This manuscript describes the benefits and barriers of TBL, and the tools necessary for developing, implementing, and critically evaluating the technique within coursework in a user-friendly method. Specifically, the manuscript describes the processes underpinning effective TBL development, preparation, implementation, assessment, and evaluation, as well as practical techniques and advice from authors' classroom experiences. The paper also highlights published articles in the area of TBL in education, with a focus on pharmacy education.

Assessment of a Revised Method for Evaluating Peer-graded Assignments in a Skills-based Course Sequence

OBJECTIVE. To evaluate the modified peer-grading process incorporated into the SOAP (subjective, objective, assessment, plan) note sessions in a skills-based pharmacy course sequence. DESIGN. Students assessed a de-identified peer's SOAP note in a faculty-led peer-grading session followed by an optional grade challenge opportunity. Using paired t tests, final session grades (peer-graded with challenge opportunity) were compared with the retrospective faculty-assigned grades. Additionally, students responded to a survey using 4-point Likert scale and open-answer items to assess their perceptions of the process. ASSESSMENT. No significant difference was found between mean scores assigned by faculty members vs those made by student peers after participation in 3 SOAP note sessions, which included a SOAP note-writing workshop, a peer-grading workshop, and a grade challenge opportunity. The survey data indicated that students generally were satisfied with the process. CONCLUSION. This study provides insight into the peer-grading process used to evaluate SOAP notes. The findings support the continued use of this assessment format in a skills-based course.

An elective course on application of clinical pharmacy principles

OBJECTIVE

To design and implement an elective therapeutics course and to assess its impact on students' attainment of course outcomes and level of confidence in applying clinical pharmacy principles and pharmacotherapy knowledge.

DESIGN

A 3-credit hour elective for third-year pharmacy students was structured to include problem-based learning (PBL), journal club and case presentations, and drug information activities.

ASSESSMENT

Student achievement of curricular outcomes was measured using performance on SOAP (subjective, objective, assessment, plan) notes, case and journal club presentations, drug information activities, and peer evaluations.

RESULTS

from a pre- and post-course survey instrument demonstrated significant improvement in students' confidence in applying clinical pharmacy principles.

CONCLUSION

Students completing the course demonstrated increased attainment of course outcomes and confidence in their abilities to evaluate a patient case and make pharmacotherapeutic recommendations.

Communications training in pharmacy education, 1995-2010

The role of the pharmacist as a "communicator" of information and advice between patients, other healthcare practitioners, and the community is recognized as a vital component of the responsibilities of a practicing pharmacist. Pharmacy education is changing to reflect this, although the difficulty is in designing a curriculum that is capable of equipping students with the necessary knowledge and skills, using activities that are effective in promoting communication competency. The objective of this review was to identify published, peer-reviewed articles concerning communication training in pharmacy education programs, and describe which communication skills the structured learning activities aimed to improve and how these learning activities were assessed. A systematic literature search was conducted and the articles found were analyzed and divided into categories based on specific communication skills taught and type of learning activity used. Oral interpersonal communication skills targeted at patients were the most common skill-type described, followed by clinical writing skills. Common teaching methods included simulated and standardized patient interactions and pharmacy practice experience courses. Most educational interventions were assessed by subjective measures. Many interventions were described as fragments, in isolation of other learning activities that took place in a course, which impedes complete analysis of study results. To succeed in communication training, integration between different learning activities and progression within pharmacy educations are important.

Peer- and self-grading compared to faculty grading

OBJECTIVES

To determine the reliability and value of peer- and self -reported evaluations in the grading of pharmacy students.

METHODS

Mean student peer- and self- reported grades were compared to faculty grades in the advanced pharmacy practice experience (APPE) and seminar presentation courses. Responses from pharmacy school alumni regarding curricular peer- and self-reported evaluations were solicited using an online survey tool.

RESULTS

Self-reported student grades were lower than the faculty-reported grade overall and for the formal presentation component of the APPE course grading rubric. Self-reported grades were no different than faculty-reported grades for the seminar course. Students graded their peers higher than did faculty members for both the seminar and APPE courses on all components of the grading rubric. The majority of pharmacy alumni conducted peer- and self-evaluations (64% and 85%, respectively) at least annually and considered peer- and self-evaluations useful in assessing students' work in group projects, oral presentations, and professional skills.

CONCLUSION

The combination of self-, peer-, and faculty-assessments using a detailed grading rubric offers an opportunity to meet accreditation standards and better prepare pharmacy students for their professional careers.

Rubric evaluation of pediatric emergency medicine fellows

OBJECTIVES

To develop and validate a rubric assessment instrument for use by pediatric emergency medicine (PEM) faculty to evaluate PEM fellows and for fellows to use to self-assess.

METHODS

This is a prospective study at a PEM fellowship program. The assessment instrument was developed through a multistep process: (1) development of rubric format items, scaled on the modified Dreyfus model proficiency levels, corresponding to the 6 Accreditation Council for Graduate Medical Education core competencies; (2) determination of content and construct validity of the items through structured input and item refinement by subject matter experts and focus group review; (3) collection of data using a 61-item form; (4) evaluation of psychometrics; (5) selection of items for use in the final instrument.

RESULTS

A total of 261 evaluations were collected from 2006 to 2007; exploratory factor analysis yielded 5 factors with Eigenvalues >1.0; each contained ≥4 items, with factor loadings >0.4 corresponding with the following competencies: (1) medical knowledge and practice-based learning and improvement, (2) patient care and systems-based practice, (3) interpersonal skills, (4) communication skills, and (5) professionalism. Cronbach α for the final 53-item instrument was 0.989. There was also significant responsiveness of the tool to the year of training.

CONCLUSION

A substantively and statistically validated rubric evaluation of PEM fellows is a reliable tool for formative and summative evaluation.

The development of a competency-based assessment rubric to measure resident milestones

BACKGROUND

The outcomes-based assessment rubric is a novel systematic instrument for documenting improvement in clinical learning.

APPROACH

This article describes the development of a rubric aimed at introducing specific performance indicators to measure the Accreditation Council for Graduate Medical Education competencies.

RESULTS

The potential benefits and implications for medical education include specifying performance indicators and outcomes, ensuring that assessment is coherent and consistent for all residents, measuring resident outcomes based on real-life criteria, providing opportunities for residents to demonstrate proficiency in a specific competency and outcome level, and improving the quality of assessment.