A comparison of grades awarded by peer assessment, faculty and a digital scanning device in a pre-clinical operative skills course

Feasibility of software-based assessment for automated evaluation of tooth preparation for dental crown by using a computational geometric algorithm

The purpose of this study was to propose the concept of software-based automated evaluation (SAE) of tooth preparation quality using computational geometric algorithms, and evaluate the feasibility of SAE in the assessment of abutment tooth preparation for single-unit anatomic contour crowns by comparing it with a human-based digitally assisted evaluation (DAE) by trained human evaluators. Thirty-five mandibular first molars were prepared for anatomical contour crown restoration by graduate students. Each prepared tooth was digitized and evaluated in terms of occlusal reduction and total occlusal convergence using SAE and DAE. Intra-rater agreement for the scores graded by the SAE and DAE and inter-rater agreement between the SAE and DAE were analyzed with the significance level (α) of 0.05. The evaluation using the SAE protocol demonstrated perfect intra-rater agreement, whereas the evaluation using the DAE protocol showed moderate-to-good intra-rater agreement. The evaluation values of the SAE and DAE protocols showed almost perfect inter-rater agreement. The SAE developed for tooth preparation evaluation can be used for dental education and clinical skill feedback. SAE may minimize possible errors in the conventional rating and provide more reliable and precise assessments than the human-based DAE.

Peer assessment: Development and delivery of the OSCE

INTRODUCTION

There is an expectation that healthcare professionals display competence in teaching, assessment and providing feedback. Development begins with formative peer-assisted learning and teaching in the undergraduate environment. Using peers or near-peers (in this case having 1 year more experience than the examination cohort) to provide assessment in summative exams remains unexplored. This study investigates how the use of near-peers compares to marking by academic staff in a summative OSCE.

MATERIALS AND METHODS

BDS4 Peer assessors (PAs) developed an OSCE question and marking schedule. Each PA (n = 3) was paired with an academic staff assessor (ASA) (n = 3). Peer and academic marked the candidates independently. Two years later, the process was repeated on the same cohort of candidates with the PA now 1-year post qualification. Statistical analysis compared the scores awarded by PA during each timeframe and against the marks awarded by the ASA.

RESULTS

During round 1, 28 students (62.2%) were awarded the same score by PA and ASA. On 17 occasions, there was a discrepancy (37.8%). Bias was skewed in favour of PA scoring higher (mean difference of differences -0.0667). During round 2, 27 students (55.1%) were awarded the same score by PA and ASA. On 22 occasions (44.9%), there was a discrepancy. Bias was skewed in favour of ASA scoring higher (mean difference of differences 0.0612).

DISCUSSION

Levels of agreement between PA and ASA are strong. Our results show PA mark more leniently as undergraduates and less leniently at 1-year post graduation.

CONCLUSIONS

Peer assessors are able to write OSCE stations, produce marking schemes and effectively assess their near-peers.

Peer-assessment ability of trainees in clinical restorative dentistry: can it be fostered?

Objectives

The study was conducted to investigate whether peer-assessment among dental students at the clinical stage can be fostered and become closer to that of experienced faculty assessors.

Methods

A prospective pilot study was conducted in 2021 at the Faculty of Dentistry, Damascus University. Sixteen 5th year clinical students volunteered to participate in the study. A modified version of the validated Peer Direct Observation of Procedural Skills (Peer-DOPS) assessment form was used together with a grading rubric. Participants undertook peer-assessment on their colleagues across three encounters. The difference between peers and faculty assessment was the main variable.

Results

The mean difference between peers and faculty assessment decreased after each encounter with a significant difference and a medium effect size between the first and third encounters (p = 0.016, d = 0.67). Peer-assessment was significantly higher than faculty, however, the overestimation declined with each encounter reducing the difference between peer- and faculty assessment. Peers’ perception of the educational benefit of conducting assessment was overwhelmingly positive, reporting improvements in their own performance.

Conclusion

This pilot study provides preliminary evidence that dental students assessment ability of their peers can be fostered and become closer to that of experienced faculty assessment with practice and assessment-specific instruction.

Effectiveness of Peer Evaluation in Learning Process: A Case from Dental Technology Students

Background:

Peer assessment or evaluation has been shown to play an important role in several learning processes. However, the effectiveness of this educational evaluation method has yet to be studied among students in different specialties.

Objective:

The study aimed to assess the effectiveness of peer evaluation as a method of student assessments in dental technology courses.

Methods:

The study sample (n=130 students) was randomly divided into 5 groups, then the effectiveness of peer evaluation was measured and correlated with student’s Grade Point Average (GPA) and gender.

Results:

The results showed the ability of students to evaluate their peers in a certain course. As per students’ perspective, scores ranged from 3.79-4.24 (out of 5), with a high degree of practice in all fields, ranked as follows: psychological stress, teamwork, self-efficacy, and, finally, the use of social media for learning. The mean score of all the categories in the peer assessment of students was 4.005 with a high degree of practice according to Scheffe's Test. Female gender was associated with significantly higher scores in the fields of psychosocial pressure, use of social media and overall scores. In addition, students achieving a GPA of 68 and more had significantly higher overall scores compared to students with lower GPAs.

Conclusion:

Peer evaluation seems to be a valid tool for students’ assessment; thus, it can be recommended as part of the course grading system in dental technology courses and can be applied to other dental or medical courses that involve practical sessions.

Digital assessment of a retentive full crown preparation-An evaluation of prepCheck in an undergraduate pre-clinical teaching environment

INTRODUCTION

Acquiring practical skills is essential for dental students. These practical skills are assessed throughout their training, both formatively and summatively. However, by means of visual inspection alone, assessment cannot always be performed objectively. A computerised evaluation system may serve as an objective tool to assist the assessor.

AIM

The aim of the study was to evaluate prepCheck as a tool to assess students' practical skills and as a means to provide feedback in dental education.

METHODS

As part of a previously scheduled practical examination, students made a preparation for a retentive crown on the maxillary right central incisor-tooth 11. Assessments were made four times by two independent assessors in two different ways: (a) conventionally and (b) assisted by prepCheck. By means of Cohen's kappa coefficient, agreements between conventional and digitally assisted assessments were compared. Questionnaires were used to assess how students experienced working with prepCheck.

RESULTS

Without the use of prepCheck, ratings given by teachers differed considerably (mean κ = 0.19), whereas the differences with prepCheck assistance were very small (mean κ = 0.96). Students found prepCheck a helpful tool for teachers to assess practical skills. Extra feedback given by prepCheck was considered useful and effective. However, some students complained about too few scanners and too little time for practice, and some believed that prepCheck is too strict.

CONCLUSION

prepCheck can be used to assist assessors in order to obtain a more objective outcome. Results showed that practicing with feedback from both prepCheck and the teacher contributes to an effective learning process. Most students appreciated prepCheck for learning practical skills, but introducing prepCheck requires enough equipment and preparation time.

Undergraduate dental students' perspective on the implementation of digital dentistry in the preclinical curriculum: a questionnaire survey

Background

Digitalisation is an expanding field in dentistry and implementation of digital teaching methods in dental education is an essential part of modern education. Therefore, two digital training modules were implemented in the preclinical curriculum at the Justus Liebig University Giessen. The aim of this study was to assess the students’ perspective on the implementation with a questionnaire survey.

Methods

Since the fall term 2017/18, students of the course of dental prosthodontics I attended the training module I, where they learned to use computer-aided learning (CAL) approaches for the digital analysis of tooth preparations. In training module II, students of the course of dental prosthodontics II learned how to manufacture a computer-aided design/computer-aided manufacturing restoration. After the completion of the training modules, all students starting with the fall term 2017/18 to the spring term 2019 were asked to fill in a questionnaire regarding the aspects of handling, didactic benefit, motivation, and overall assessment.

Results

Students rated the implementation of digital aspects in teaching as positive in terms of handling, didactic benefit, and motivation, but gave preference to the assessment of the tooth preparations by dental instructors. In addition, students assessed the feedback from the faculty regarding tips and tricks better than the digital feedback. More than 90% of the students indicated that they could imagine using an intraoral scanner for treatment of patients in the dental office in future.

Conclusions

The results of the present study revealed a positive perspective of students on the implementation of digital dentistry in the preclinical curriculum. However, difficulties with CAL systems were reported and most students preferred evaluation of preparation by dental instructors. Thus, CAL approaches offer an additional teaching method besides the traditional teaching of manual skills.

Use of Digital Technology to Improve Objective and Reliable Assessment in Dental Student Simulation Laboratories

The aim of this study was to assess inter-and intra-grader agreement with the use of digital scanning and a tooth preparation assessment software program in comparison to the current traditional visual grading method in a dental student simulation laboratory. Students' typodont teeth preparations from previous practical examinations were used (cast crown n=50; cast fixed partial denture abutments n=50). Five preclinical instructors received calibration training and evaluated each of the preparations by the traditional visual grading method using a rubric. The same preparations were assessed by the same instructors using a tooth preparation assessment software program (PrepCheck, Sirona). The results showed that intra-grader agreement was significantly higher when grades were determined by PrepCheck compared to the traditional visual grading method. The traditional method was associated with significantly greater inter-grader disagreement in comparison to grading using PrepCheck (p<0.05). When the average final grade for students' crown preparations by each grader was compared for the traditional method and PrepCheck, significant differences were found for all graders (p<0.001). In this study, the use of the PrepCheck software program greatly improved intra-and inter-grader agreement during grading in a student simulation laboratory. Digital technology may improve the objectivity and reliability of assessments by preclinical evaluators.

Study on the Interrater Reliability of an OSPE (Objective Structured Practical Examination) - Subject to the Evaluation Mode in the Phantom Course of Operative Dentistry

INTRODUCTION

The aim of the study presented here was to evaluate the reliability of an OSPE end-of-semester exam in the phantom course for operative dentistry in Frankfurt am Main taking into consideration different modes of evaluation (examiner's checklist versus instructor's manual) and number of examiners (three versus four).

METHODS

In an historic, monocentric, comparative study, two different methods of evaluation were examined in a real end-of-semester setting held in OSPE form (Group I: exclusive use of an examiner's checklist versus Group II: use of an examiner's checklist including an instructor's manual). For the analysis of interrater reliability, the generalisability theory was applied that contains a generalisation of the concept of internal consistency (Cronbach's alpha).

RESULTS

The results show that the exclusive use of the examiner's checklist led to higher interrater reliability values than the in-depth instructor's manual used in addition to the list.

CONCLUSION

In summary it can be said that the examiner's checklists used in the present study, without the instructor's manual, resulted in the highest interrater reliability in combination with three evaluators within the context of the completed OSPE.

Dental students' peer assessment: a prospective pilot study

INTRODUCTION

Peer assessment is increasingly used in health education. The aims of this study were to evaluate the reliability, accuracy, educational impact and student's perceptions of undergraduate pre-clinical and clinical dental students' structured and prospective Peer assessment and peer feedback protocol.

MATERIALS AND METHODS

Two Direct Observation of Procedural Skills (DOPS) forms were modified for use in pre-clinical and clinical peer assessment. Ten year two dental students working in a phantom-heads skills laboratory and 16-year five dental students attending a comprehensive care clinic piloted both peer DOPS forms. After training, pairs of students observed, assessed and provided immediate feedback to each other using their respective peer DOPS forms as frameworks. At the end of the 3-month study period, students anonymously provided their perceptions of the protocol.

RESULTS

Year 2 and year 5 students completed 57 and 104 peer DOPS forms, respectively. The generalizability coefficient was 0.62 for year 2 (six encounters) and 0.67 for year 5 (seven encounters). Both groups were able to differentiate amongst peer-assessed domains and so detect improvement in peers' performance over time. Peer DOPS scores of both groups showed a positive correlation with their mean end-of-year examination marks (r ≥ 0.505, P ≥ 0.051) although this was not statistically significant. There was no difference (P ≥ 0.094) between the end-of-year examination marks of the participating students and the rest of their respective classes. The vast majority of both groups expressed positive perceptions of the piloted protocol.

DISCUSSION

There are no data in the literature on the prospective use of peer assessment in the dental undergraduate setting. In the current study, both pre-clinical and clinical students demonstrated the ability to identify those domains where peers performed better, as well as those which needed improvement. Despite no observable educational impact, most students reported positive perceptions of the peer DOPS protocol.

CONCLUSIONS

The results of this pilot study support the need for and the potential benefit of a larger- and longer-term follow-up study utilising the protocol.

Peer assessment among first year medical students in anatomy

Peer assessment has been shown to be an effective tool to promote professionalism in medical students. Peer assessment may be particularly useful in anatomy dissection laboratory as the required close collaboration and long hours of anatomy laboratory provide students insights into their peers' work habits and interpersonal skills. The objective of this study was to quantitatively and qualitatively analyze the use of a validated peer assessment tool in Gross Anatomy. Students in a first year medical school class evaluated three members of their dissection group using an online survey tool. The mid-course and end-of-course evaluation included open-ended comments, as well as a five-point scale that measured three work habits, two interpersonal attributes and one overall score. All 267 students completed the assignment. The overall score and four of the five other assessed categories showed significant improvement from the mid- to end-of-course evaluations. Quantitative and qualitative data also revealed significant improvement among the students who received the lowest mid-course assessments. Seventy-six percent of the class agreed with the statement: "Based on the feedback I received, I made a change in how I worked with or taught my peers." The use of this peer assessment tool used by students in anatomy was associated with improvements in work habits and interpersonal attributes, particularly by the cohort of students who received the lowest mid-course feedback. Peer assessment offers students an opportunity to improve their interpersonal skills and work habits.

Use of peer assessment in tooth extraction competency

INTRODUCTION

Peer assessment is a widely recognised practice in higher education and is being increasingly utilised in healthcare education. It offers multiple advantages and may play a useful role in enhancing students' learning experience. This study was carried out to evaluate the utility of peer assessment in tooth extraction competency in simulated dental learning environment (SDLE).

METHODS

Year 2 dental students were invited to participate in a formative, peer assessment exercise on forceps tooth extraction in SDLE. The ratings made by peers in the formative assessment were compared with the ratings made by academic supervisors in an equivalent summative assessment.

RESULTS

One hundred and twenty-two students participated in both the formative and summative assessments. Descriptive statistics demonstrated that the majority of students were given the same rating in both assessments. However, there were a number of students who were rated as 'Competent' in one assessment and 'Not Competent' in the other assessment. Spearman's Rho correlations showed that there were weak relationships between the formative and summative assessments. There was also weak inter-rater reliability for the ratings of the peers and academic supervisors, as measured by the intraclass correlation coefficient. Feedback from the students and academic supervisors showed unanimously positive perceptions about the peer assessment experience, and they expressed the desire for incorporating peer assessment in other competencies.

CONCLUSIONS

This study shows mixed results with regard to the value of the peer assessment. Given the potential utility of peer assessment as a tool to augment and enrich students' learning experience, further work needs to be done to evaluate the use of peer assessment in operative competencies in dentistry.