Faculty Calibration: Much Ado About Something

Effectiveness of online faculty calibration: A randomized crossover study

PURPOSE

Dental hygiene faculty members must be able to provide evidence of skill calibration for clinical evaluation of students. The purpose of this study was to evaluate the effectiveness of online instructional videos as compared to in-person sessions for faculty calibration.

METHODS

A randomised crossover pretest/posttest research design was used to evaluate online and in-person faculty calibration activities. Fifteen faculty members from a baccalaureate dental hygiene programme were randomly assigned to an AB or BA sequence for calibration sessions on two different instruments. Following a 2-week washout period, the groups switched activity modalities. A pretest, posttest and retention test, administered 10 weeks following the activity, were administered to determine learning levels and the new and retained knowledge. A 7-point Likert scale questionnaire evaluated the reaction to and impact of the calibration activities. Descriptive statistics analysed demographic and Likert scale data. Paired samples t-tests were used to analyse the research questions (p ≤ 0.05).

RESULTS

Online calibration activities yielded higher posttest scores than in-person activities (p = 0.01). Findings related to feelings of confidence revealed that a greater percentage of participants agreed that online calibration activities increased their ability to evaluate student performance. Findings related to feelings of preparedness supported equal percentages of participants who agreed the online and in-person activities increased their ability to teach dental hygiene instrumentation. There was no significant difference between in-person and online retention test scores (p = 0.235).

CONCLUSION

Faculty members agreed that both online and in-person calibration activities were an effective use of their time and contributed to greater feelings of confidence and preparedness. However, the online calibration activities seemed to be more effective at increasing calibration on instrumentation. More research is needed to determine additional effective strategies for the online calibration of clinical faculty.

Periodontal diagnosis and treatment planning - An assessment of the understanding of the new classification system

OBJECTIVES

Substantial variations are seen among clinicians in the diagnosis and treatment planning of periodontal diseases. Accurate diagnosis and treatment planning are fundamental requirements for effective outcome-based patient care. The aim of this study was to evaluate the understanding of the American Academy of Periodontology and the European Federation of Periodontology 2017 periodontal disease classifications in diagnoses and treatment plans across four study groups.

METHODS

The study recruited at least 20 participants in each of the four study groups. These included 1) Periodontal faculty and residents at Indiana University School of Dentistry (IUSD-PF) 2) IUSD general practice faculty (IUSD-GPF), 3) private practice periodontists (PPP), and 4) general practitioners (GP). The participants were provided with 10 HIPPA de-identified case records and a link to a survey. The survey comprised five demographic questions and two questions on diagnosis and treatment plan for each case along with a fixed list of responses. The responses were then compared against gold standards that were determined by a group of three board-certified periodontists.

RESULTS

Overall, for diagnostic questions, GP (69%) were correct significantly less often than IUSD-PF (86%, p < 0.001), IUSD-GPF (79%, p = 0.002), and PPP (80%, p = 0.001). No significant differences (p > 0.05) in the overall correct treatment plan responses were found among the four groups (IUSD-PF: 69%, IUSD-GPF: 62%, PPP: 68%, and GP: 60%). The multi-rater kappas for with-in-group agreement on overall diagnosis ranged from 0.36 (GP) to 0.55 (IUSD-PF) and on overall treatment plan ranged from 0.32 (IUSD-GPF) to 0.42 (IUSD-PF). Overall agreement for diagnosis and treatment plans among the four groups was relatively low and none of the groups were statistically different from each other (p > 0.05).

CONCLUSION

Regular participation in calibration sessions may lead to more accurate adoption of the 2017 periodontal classification and thereby help provide consistent diagnosis and treatment.

The new periodontal disease classification: Level of agreement on diagnoses and treatment planning at various dental education levels

PURPOSE/OBJECTIVES

Using the periodontal diseases classification published in 2018, this study evaluated the level of agreement among predoctoral and postgraduate students of different education levels and specialties in the diagnosis and treatment planning of periodontal conditions.

METHODS

Second-year (D2) and fourth-year (D4) dental students, postgraduate orthodontic students (OS), and periodontology students (PS) were presented with an anonymous survey of 10 cases, each with five choices of diagnosis and seven therapeutic approaches. The cases included a patient summary, photographs, radiographs, periodontal charting, and ranged from health to periodontitis. Consensus diagnosis, used as a gold standard, was established between two experienced periodontists. Diagnosis and treatment choices for each case were compared across educational groups using Fisher's exact test. The level of agreement among educational groups was assessed using a multirater kappa coefficient.

RESULTS

The survey was completed in 2020 with a 72.4% overall response rate (57-D2, 45-D4, 17-OS, and 12-PS). The overall level of agreement on diagnosis was fair (κ = 0.24, p <.0001) with PS showing the most agreement (κ = 0.55, p <.0001). There were tendencies to prioritize stage over grade and extent in assigning diagnoses and to overdiagnose disease severity and underdiagnose localized periodontitis. Nonsurgical periodontal therapies had the highest overall agreement (κ = 0.63 for scaling/root planing and κ = 0.44 for maintenance, p <.0001). Response distribution per case and per level of education indicated that PS more consistently agreed with the gold standard than any other group.

CONCLUSIONS

OS, D2, and D4 students demonstrated inferior agreement levels compared to PS, successfully recognized health from disease, suggested appropriate treatment plans, and tended to overdiagnose milder periodontal conditions.

Effect of a Standardized Training with Digital Evaluation on the Improvement of Prosthodontic Faculty's Performance in Crown Preparation: A Pre-Post Design

PURPOSE

Standardized crown preparation is an important competency for prosthodontic faculty especially when they take on the dual role of clinician and clinical teacher. Effects of faculty training for enhancing crown preparation competency are seldom reported. This study aimed to analyze the impact of a standardized training workshop with digital evaluation on the dental faculty's performance in crown preparation.

MATERIALS AND METHODS

The digitally evaluated grades of anterior (the maxillary right central incisor) and posterior (the mandibular left first molar) tooth preparations made by 76 participants who accomplished all six training tasks were collected, including off-site and on-site exercises before the didactic lecture and live demonstration, three rounds of practices with digital feedback, and a final test. Grades of preparations performed in the on-site exercise were adopted as pre-training scores, and those in the final test as post-training scores. Total scores and marks deducted for the parameters including amount of reduction, margin line, and taper were compared among each training task.

RESULTS

The post-training scores of both anterior and posterior tooth preparations increased significantly more than the pre-training scores. The average increased score proportion was 22.95% ± 4.17% for anterior tooth preparations, and 21.78% ± 3.68% for posterior tooth preparations. For anterior tooth preparations, total scores and the parameters except taper significantly improved in the first practice and maintained the same level for the next sessions. Total scores and all parameters for posterior tooth preparations exhibited continual improvement during the training process.

CONCLUSION

Standardized training can further improve dental faculty's crown preparation performance in a moderate way. Individual design for crown preparation training can be considered based on different tooth positions. Providing such training will aid the calibration of clinical teaching behavior and the elevation of clinical operative standards for prosthodontic faculty.

Faculty calibration and students' self-assessments using an instructional rubric in preparation for a practical examination

PURPOSE

The purpose of this study was to investigate the effect of faculty calibration and students' self-assessments on students' performances in a periodontal practical examination.

METHODS

Before a new instructional rubric was implemented in the second-year periodontics course, faculty calibration was conducted with a pilot group of 32 third-year dental students. Afterwards, the new rubric was implemented in the second-year periodontics course. Second-year students used the rubric for their self-assessments before the practical examination. An intraclass correlation coefficient was used to test the reliability of the faculty members. A paired t test was used to compare scores between self-assessments of the pilot group (third-year students) and faculty evaluation. A two-way analysis of variance was performed to compare scores between self-assessments of second-year students and faculty evaluations. Chi-square tests were used to compare overall failure rates amongst four different classes.

RESULTS

The reliability amongst the faculty members was strong (the ICC = 0.75 at the first and 0.97 at the second calibration). The mean self-assessment score from the pilot group was significantly higher than the faculty evaluation. However, the mean self-assessment score of second-year students was significantly lower than the faculty evaluation. The class that practiced self-assessments with the validated instructional rubric exhibited the lowest overall failure rate compared to three past classes.

CONCLUSIONS

Using an instructional rubric and conducting faculty calibration improved the process of the periodontal practical examination. Improving the examination process and practicing self-assessments with feedback from faculty may have a positive impact on students' performances in the examination.

Periodontal Diagnosis and Treatment Planning Among Indiana Dental Faculty, Periodontists, and General Practice Dentists: A Multi-Group Comparison

Diagnosis and treatment planning for periodontal disease are fraught with challenges because of the complex and multifactorial nature of the disease as well as the inherent variability in interpretation of clinical findings. It is important for all practitioners to be accurate and consistent in formulating diagnoses based on the American Academy of Periodontology classification guidelines and to implement treatment plans to adequately address patients' needs. The aim of this study was to compare diagnoses and treatment plans among four groups of participants: full-time and part-time periodontology faculty at Indiana University School of Dentistry (IUSD), full-time and part-time IUSD general practice faculty, full-time periodontists in private practice, and full-time general practitioners in private practice. The study, conducted September 2016 to February 2017, also sought to determine if the calibrated participants had more correct diagnoses and treatment plans than those who had not received calibration training. Each of the four groups had 20 participants each. Participants evaluated ten de-identified case records and selected a diagnosis and treatment plan for each case. In the results, the 20 IUSD periodontal faculty members, most of whom had participated in calibration sessions, had overall better agreement and more correct responses for diagnoses and treatment plans than the IUSD general practice faculty members, private practice general practitioners, and private practice periodontists (only one of those 60 participants had participated in calibration sessions). The results supported the notion that periodic calibration is needed to standardize faculty criteria, facilitate better agreement and accuracy, and enhance consistency in the use of clinical criteria during training for dental students and in practice.

The Effect of Calibration on Caries Risk Assessment Performance by Students and Clinical Faculty

Caries management requires a complete oral examination and an accurate caries risk assessment (CRA). Performing Caries Management by Risk Assessment (CAMBRA) is inefficient when the caries risk level assignment is incorrect. The aim of this study was to evaluate the ability of faculty members and students at one U.S. dental school to correctly assign caries risk levels for 22 CRA cases, followed by calibration with guidelines on how to use the CRA form and a post-calibration test two months after calibration. Inter-examiner reliability to a gold standard (consensus of three experts) was assessed as poor, fair, moderate, good, and very good. Of the 162 students and 125 faculty members invited to participate, 13 students and 20 faculty members returned pre-calibration tests, for response rates of 8% and 16%, respectively. On the post-calibration test, eight students and 13 faculty members participated for response rates of 5% and 10%, respectively. Without guidelines and calibration, both faculty members and students when evaluated as one group performed only poor to fair in assigning correct caries risk levels. After calibration, levels improved to good and very good agreements with the gold standard. When faculty and students were evaluated separately, in the pre-calibration test they correctly assigned the caries risk level on average in only one-quarter of the cases (students 24.1%±13.3%; faculty 23.6%±17.5%). After calibration, both groups significantly improved their correct assignment rate. Faculty members (73.8% correct assignments) showed even significantly higher correct assignment rates than students (47.7% correct assignments). These findings suggest that calibration with a specific set of guidelines improved CRA outcomes for both the faculty members and students. Improved guidelines on how to use a CRA form should lead to improved caries risk assessment and proper treatment strategy for patients.