Assessment formats in dental medicine: An overview

Use of Multiple-Choice Items in Summative Examinations: Questionnaire Survey Among German Undergraduate Dental Training Programs

Background

Multiple-choice examinations are frequently used in German dental schools. However, details regarding the used item types and applied scoring methods are lacking.

Objective

This study aims to gain insight into the current use of multiple-choice items (ie, questions) in summative examinations in German undergraduate dental training programs.

Methods

A paper-based 10-item questionnaire regarding the used assessment methods, multiple-choice item types, and applied scoring methods was designed. The pilot-tested questionnaire was mailed to the deans of studies and to the heads of the Department of Operative/Restorative Dentistry at all 30 dental schools in Germany in February 2023. Statistical analysis was performed using the Fisher exact test (P<.05).

Results

The response rate amounted to 90% (27/30 dental schools). All respondent dental schools used multiple-choice examinations for summative assessments. Examinations were delivered electronically by 70% (19/27) of the dental schools. Almost all dental schools used single-choice Type A items (24/27, 89%), which accounted for the largest number of items in approximately half of the dental schools (13/27, 48%). Further item types (eg, conventional multiple-select items, Multiple-True-False, and Pick-N) were only used by fewer dental schools (≤67%, up to 18 out of 27 dental schools). For the multiple-select item types, the applied scoring methods varied considerably (ie, awarding [intermediate] partial credit and requirements for partial credit). Dental schools with the possibility of electronic examinations used multiple-select items slightly more often (14/19, 74% vs 4/8, 50%). However, this difference was statistically not significant (P=.38). Dental schools used items either individually or as key feature problems consisting of a clinical case scenario followed by a number of items focusing on critical treatment steps (15/27, 56%). Not a single school used alternative testing methods (eg, answer-until-correct). A formal item review process was established at about half of the dental schools (15/27, 56%).

Conclusions

Summative assessment methods among German dental schools vary widely. Especially, a large variability regarding the use and scoring of multiple-select multiple-choice items was found.

Evaluating Study Approach of Dental Students in Palestine using a Study Process Questionnaire: A Cross-Sectional Study

Background

Learning approach strategies are an important factor to obtain knowledge in any professional course. Surface approach learning and deep approach learning are two main types of learning strategies.

Aim

The aim of present study was to evaluate the study approach strategies of dental students in Palestine.

Materials and Methods

The present study follows a cross-sectional study design, which includes 250 students from first year to fifth year at Al Quds University. The present study evaluated the study approach using a questionnaire called R-SPQ-2F that was filled by all the students using Google forms. The assessment scores from the curriculum assessment examination were also compared with the scores of the R-SPQ-2F questionnaire. SPSS software was used to analyze data.

Results

The results of the ANOVA show that the students in the fifth years had significantly higher mean scores of deep learning approaches than other years (P < 0.001). The students having curriculum assessment scores above 80% showed significantly more deep learning strategies than surface learning strategies (P < 0.05).

Conclusion

Deep learning approach can provide better academic outcome. Newer teaching strategies that enhance the deep learning approach should be encouraged in the dental curriculum.

[ZApprO versus ZÄPrO: results of a first comparative study]

BACKGROUND

More than 60 years after the enactment of the first Licensing Regulations for Dentists (ZÄPrO), a new Licensing Regulations for Dentists, the ZApprO, was implemented in 2020. The aims of this study were to evaluate and compare the "Course in Technical Propaedeutics" (TPK) with the course "Dental Propaedeutics with a Focus on Dental Technology" (ZPDT), which are based on different legal foundations.

METHODS

The following parameters were compared after completion of the courses: (1) theoretical and practical knowledge, (2) regular department-internal learner evaluation, and (3) special evaluation of learning conditions by the learners and (4) by the instructors. The theoretical and practical exams and the questionnaires were analyzed in terms of their test-statistical characteristics (difficulty, discrimination index, internal consistency). Group comparisons between TPK and ZPDT were conducted using t‑tests for independent groups.

RESULTS

Significant differences were only detected in the students' evaluation of their learning conditions, with theoretical and practical knowledge transfer being rated lower in the TPK than in the ZPDT.

DISCUSSION

The comparable results of the knowledge assessments and the regular learner evaluation, triangulated with the comprehensive learner and instructor evaluation, allowed a broad assessment of both courses. The significant differences identified provide opportunities for optimization of the newly implemented ZPDT course.

Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods

BACKGROUND

Scoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data.

OBJECTIVE

This study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6).

METHODS

Items and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05).

RESULTS

We analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977).

CONCLUSIONS

Educators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees' scores and overall examination results.

Development of a competency-based clinical assessment instrument for exit level Oral Hygiene students at the University of Western Cape

Seminal to the process of a health sciences curriculum evaluation is the periodic review of clinical assessment instruments that measure competency. An assessment of quality is facilitated by using a well-structured, authentic and reliable instrument. This process rests on designing and measuring the instrument against a sound framework and validating it for scientific merit. This paper documents the pedagogy and the process taken in developing an improved formative competency-based assessment instrument for the final year students of the Bachelor of Oral Health program (BOH) at the University of the Western Cape (UWC).

Methods: A qualitative research study design employing the Nominal Group Technique (NGT) was used as a method for gaining small group consensus on the clinical assessment instrument for exit level Oral Hygiene (BOH3) students within the parameters of assessment principles. The key contributors to the instrument development process were the academic staff of the Department of Oral Hygiene, involved in clinical teaching and assessment of student competency.

Results: The domains of ethics and professionalism, patient assessment, diagnosis, treatment planning and implementation was identified as the core elements in the assessment. The principles of assessment, which include, alignment with outcomes, feedback, transparency and validity, were used to guide the instrument development. The assessment criteria were cross examined for alignment to the learning outcomes of the module and the program whilst formative feedback was foregrounded as a central feature to support student learning and progress monitoring. Transparency was obtained by providing students access to the instrument before and after the assessment including the written feedback on their performance. The instrument embodied a range of criteria to be assessed rather than on the awarding of a cumulative score. This allowed for the identification of the criteria or domain within which a student is struggling or excelling. Consensus on the instrument design was achieved using the NGT phases throughout the instrument development process including the weighting of the domains and grading. This level of engagement together with the application of scientifically sound assessment principles contributed to the validation of the instrument.

Conclusion: The development of a competency-based assessment instrument was the result of a structured, collaborative and scientifically engaged process framed around specific assessment principles. The process culminated in the development of a formative competency-based clinical assessment instrument that was fit for purpose in the Bachelor of Oral Health program.

The Nominal Group Technique served to be a valuable approach for small group consensus in developing the instrument. It served to promote individual perspectives and to generate debate and group discussion between academics that were proficient in clinical teaching and, finally to facilitate group consensus on the instrument structure and system for administration.

Assessment of Clinical Reasoning and Diagnostic Thinking among Dental Students

Introduction

This study aimed to investigate dental students' clinical reasoning and diagnostic thinking ability by key feature test and “diagnostic thinking inventory” questionnaire.

Methods

The present study was a descriptive cross-sectional study. The participants consisted of 61 senior dental students. Clinical reasoning and diagnostic thinking were assessed by key feature tests and the “diagnostic thinking inventory” “DTI” questionnaire, respectively. The “diagnostic thinking inventory” was developed by Bordage et al. in France and consisted of 41 questions on a 6-point Likert scale. The satisfaction of students was assessed through a 10-item questionnaire. Data were analyzed using SPSS 19 with descriptive tests (mean, SD, and percentage), student independent T-test, and Pearson correlation. The significance level was determined at p < 0.05.

Results

The mean scores of the key feature test were 56.55 ± 7.80. Diagnostic thinking scores of learners were reported in diagnostic thinking 136.47 ± 16.45, flexibility in thinking 72.22 ± 11.15, and structure of memory 64.24 ± 7.84. The difference in students' scores in flexibility in thinking was significantly higher among male students than female students. (p-value = 0.04). The students' satisfaction scores were 3.53 ± 0.52, which showed relative satisfaction.

Conclusion

The participants' clinical reasoning and diagnostic thinking skills were reported at a low level. This issue emphasizes the need for training to enhance diagnostic thinking and clinical reasoning in dental education. Formative evaluation and reform of the educational programs of this course should be considered.

An innovative interprofessional dental clinical learning environment using entrustable professional activities

INTRODUCTION

Changes in society, new public demands for dental care and contemporary educational insights have influenced dental education worldwide and demand a renewed perspective. Following this perspective, an innovative interprofessional dental Master curriculum was developed at Radboud University Nijmegen in the Netherlands.

EDUCATIONAL PRINCIPLES

The curriculum is based on five contemporary educational principles and the core of the curriculum consists of a Student Run Dental Clinic that is fully run by students under professional supervision.

THE NEW CURRICULUM

In the Student Run Dental Clinic, Master dental students and Bachelor oral hygiene students are responsible for the care of approximately 750 patients. The students work within the same clinic for 3 years and patients receive oral health care from the same students over a long period. The clinic is a longitudinal cross-disciplinary clinic with different dental subdisciplines. Entrustable professional activities (also known as EPAs), to our knowledge not yet widely used in dental education, were introduced to facilitate learning and assessment. Fourteen EPAs have been developed to stimulate interprofessional education and learning. Of these, five EPAs are identical for the dental and oral hygiene curriculum, leading to extended interprofessional education and learning in the Student Run Dental Clinic.

DISCUSSION

Preliminary results show that EPAs are generally well received by supervisors and students.

CONCLUSION

To monitor and investigate the exact effect of the interventions and underlying mechanisms, a research programme on interprofessional learning, practice-based learning and EPAs and entrustment in dental education was recently set up.

Mental burden and perception of the study situation among undergraduate students during the COVID-19 pandemic: a cross-sectional study and comparison of dental and medical students

OBJECTIVES

To investigate levels of distress, depression, anxiety, stress and perception of their current study situation during the COVID-19 pandemic among undergraduate dental and medical students.

DESIGN

Observational, cross-sectional study including two consecutive surveys (May and July 2020).

SETTING

A large medical school in Germany.

PARTICIPANTS

All first year dental and medical students were invited. 132 participating first year students (44 dental, 88 medical) from the first survey and 150 students (50 dental, 100 medical) from the second were included in our analyses.

PRIMARY AND SECONDARY OUTCOME MEASURES

Mental burden (distress thermometer, Patient Health Questionnaire-4, Perceived Stress Scale-4) and self-reported changes in mental health and perception of study situation during the COVID-19 pandemic (self-developed items) were compared. Open-ended questions were analysed by conventional content analyses.

RESULTS

A considerable proportion of students (t1: May 2020: 84.1%; t2: July 2020: 77.3%) reported distress levels above cut-off. In July 2020, dental students reported significantly higher distress scores than medical students (dental: M=7.0, SD=2.3; medical: M=5.7; SD=2.1; p<0.001). More dental than medical students reported mild, moderate and severe levels of anxiety and depression symptoms. The majority stated that their mental health and study motivation had not changed during the pandemic. Logistic regression showed that being a dental student was significantly associated with a higher likelihood for serious worries regarding the study situation during COVID-19 at t1 (OR 4.0; 95% CI 1.1 to 14.2). At t2 higher distress was significantly associated with a higher likelihood for experiencing serious worries (OR 1.8; 95% CI 1.3 to 2.5). Regarding current concerns related to the pandemic, students most frequently reported difficulties with self-regulated learning (15.2%), study-related worries and uncertainty (14.4%), missing feedback of students and lecturers (11.4%) and lack of practical training (9.8%).

CONCLUSION

The results suggest that high mental burden and the lack of practical training among medical and dental students is an increasing problem, with a possibly even higher urgency in dental students. Tailored psychological and educational support offers during and after the COVID-19 pandemic might help them as they progress through (medical and) dental school.

The utility of a formative one-station objective structured clinical examination for Substance use disorders in a dental curriculum

Substance use disorders (SUD) are chronic relapsing medical conditions characterised by compulsive substance seeking and use. They constitute a substantial disease burden globally. Labelling of persons with SUD has created barriers to treatment but there are effective management strategies. The dental profession has embraced reforms designed to address the SUD epidemic by promoting continuing education for practitioners and initiating curriculum changes in dental schools. Screening, Brief Intervention and Referral to Treatment (SBIRT) is an evidence-based model for managing patients with SUD. The use of a formative 1-station Objective Structured Clinical Examination (OSCE) for learning and assessment in SBIRT, operationalised with the MD3 rating scale is presented in this study. In 3 years of implementation, the SBIRT OSCE successfully integrated into the curriculum of the College of Dental Medicine, Columbia University. Mean score of total adherent behaviours was 11.80 (SD =4.23) (range: 2 - 24) and Cronbach's coefficient alpha for across-items reliability in adherent behaviours was 0.66. Adherent behaviours correlated with the global ratings (r = 0.66). Mean of global rating scores were 2.90 (SD =1.01) for collaboration and 2.97 (SD =1.00) for empathy and the global rating scores correlated with each other (r = 0.85). Histograms of global rating scores resembled normal distribution. The 1-station OSCE is a good model for learning about SBIRT. Psychometric analysis was useful in understanding the underlying construct of the MD3 rating scale and supported its reliability, validity and utility in dental education.

Special Teaching Formats during the COVID-19 Pandemic-A Survey with Implications for a Crisis-Proof Education

Modern teaching formats have not been considered necessary during the COVID-19 pandemic with uncertain acceptance by students. The study’s aim was to describe and evaluate all measures undertaken for theoretical and practical knowledge/skill transfer, which included objective structured practical examinations (OSPEs) covering a communication skills training. The students’ performance in the OSPE as well as the theoretical knowledge level were assessed, of which the latter was compared with previous terms. In conservative dentistry and periodontology (4th and 5th year courses), theoretical teaching formats were provided online and completed by a multiple-choice test. Practical education continued without patients in small groups using the phantom-head, 3D printed teeth, and objective structured practical examinations (OSPEs) including communication skills training. Formats were evaluated by a questionnaire. The organization was rated as very good/good (88.6%), besides poor Internet connection (22.8%) and Zoom^® (14.2%) causing problems. Lectures with audio were best approved (1.48), followed by practical videos (1.54), live stream lectures (1.81), treatment checklists (1.81), and virtual problem-based learning (2.1). Lectures such as .pdf files without audio, articles, or scripts were rated worse (2.15–2.30). Phantom-heads were considered the best substitute for patient treatment (59.5%), while additional methodical efforts for more realistic settings led to increased appraisal. However, students performed significantly worse in the multiple-choice test compared to the previous terms (p < 0.0001) and the OSPEs revealed deficits in the students’ communication skills. In the future, permanent available lectures with audio and efforts toward realistic treatment settings in the case of suspended patient treatment will be pursued.

Experiences from Two Ways of Integrating Pre- and Post-course Multiple-choice Assessment Questions in Educational Events for Surgeons

To examine how to optimise the integration of multiple-choice questions (MCQs) for learning in continuing professional development (CPD) events in surgery, we implemented and evaluated two methods in two subspecialities over multiple years. The same 12 MCQs were administered pre- and post-event in 66 facial trauma courses. Two different sets of 10 MCQs were administered pre- and post-event in 21 small animal fracture courses. We performed standard psychometric tests on responses from participants who completed both the pre- and post-event assessment. The average difficulty index pre-course was 57% with a discrimination index of 0.20 for small animal fractures and 53% with a discrimination index of 0.15 for facial trauma. For the majority of the individual MCQs, the scores were between 30%-70% and the discrimination index was >0.10. The difficulty index post-course increased in both groups (to 75% and 62%). The pre-course MCQs resulted in an average score in the expected range for both formats suggesting they were appropriate for the intended level of difficulty and an appropriate pre-course learning activity. Post-course completion resulted in increased scores with both formats. Both delivery methods worked well in all regions and overall quality depends on applying a solid item development and validation process.

Assessment methods in medical specialist assessments in the DACH region - overview, critical examination and recommendations for further development

Introduction: Specialist medical assessments fulfil the task of ensuring that physicians have the clinical competence to independently represent their field and provide the best possible care to patients, taking into account the current state of knowledge. To date, there are no comprehensive reports on the status of specialist assessments in the German-speaking countries (DACH). For that reason, the assessment methods used in the DACH region are compiled and critically evaluated in this article, and recommendations for further development are described. Methods: The websites of the following institutions were searched for information regarding testing methods used and the organisation of specialist examinations: Homepage of the Swiss Institute for Medical Continuing Education (SIWF), Homepage of the Academy of Physicians (Austria) and Homepage of the German Federal Medical Association (BAEK). Further links were considered and the results were presented in tabular form. The assessment methods used in the specialist assessments are critically examined with regard to established quality criteria and recommendations for the further development of the specialist assessments are derived from these. Results: The following assessment methods are already used in Switzerland and Austria: written examinations with multiple choice and short answer questions, structured oral examinations, the Script Concordance Test (SCT) and the Objective Structured Clinical Examination (OSCE). In some cases, these assessment methods are combined (triangulation). In Germany, on the other hand, the oral examination has so far been conducted in an unstructured manner in the form of a 'collegial content discussion'. In order to test knowledge, practical and communicative competences equally, it is recommended to implement a triangulation of methods and follow the further recommendations described in this article. Conclusion: While there are already accepted approaches for quality-assured and competence-based specialist assessments in Switzerland and Austria at present, there is still a long way to go in Germany. Following the recommendations presented in this article, a contribution could be made to improving the specialist assessments in the DACH region according to the specialist assessments objectives.