Reliability and Predictive Validity of a Comprehensive Preclinical OSCE in Dental Education

Evaluating student performance assessment methods in Objective Structured Clinical Exam: perspectives and comparisons

PURPOSE

This study aims to evaluate how student performance and perspectives changed when the Objective Structured Clinical Exam (OSCE) assessment system was changed from a composite score to discipline-specific grading at the Harvard School of Dental Medicine.

METHODS

The retrospective study population consisted of all students (n = 349) who completed three OSCEs (OSCE 1, 2, and 3) as part of the predoctoral program during the years 2014-2023. Data on the students' OSCE scores were obtained from the Office of Dental Education, and data on students' race/ethnicity and gender were obtained from their admissions data.

RESULTS

The likelihood of a student failing the OSCE after the assessment system change significantly increased with an adjusted odds ratio of 20.12. After the change, the number of failed subjects per student decreased with an adjusted mean ratio of 0.48. Students perceived the OSCE as being less useful after the change. Independent of the grading change, OSCEs 1 and 2 were seen as more useful compared to OSCE 3, which is administered in the last year of the Doctor of Dental Medicine program.

CONCLUSION

The discipline-specific nature of the new assessment system helps focus on specific areas of remediation, rather than blanket remediation used previously, in order to isolate the actual areas of deficiency and to focus remediation efforts so that students can align their learning needs appropriately. Therefore, although the actual number of fails identified increased for the course, the assessment change has allowed for more directed, actionable information to be gained from the OSCE to prepare students to work toward competency standards.

Evaluation of student readiness for clinical dental education at the end of preclinical education

OBJECTIVES

This study examined results of a summative objective structured clinical examination (OSCE) at the end of preclinical periodontal education to identify deficient areas for dental students in patient care and to explore factors affecting the probability of passing the OSCE.

METHODS

The summative OSCE was administered to two consecutive cohorts, Classes A (class of 2024; n1 = 134) and B (class of 2025; n2 = 129). The questions for each station in the OSCEs were available to both classes 1 week before the OSCEs. Descriptive statistics were used to identify deficient areas. The multiple logistic regression model was built to predict the probability of passing the OSCE based on the cohort, gender, and the practical and written examination scores.

RESULTS

Fifty-one (38%) students in Class A and 66 (51%) students in Class B completed the OSCE by passing all stations. Students undergoing remediation showed deficiencies in demonstrating how to detect tooth mobility, performing periodontal probing, drawing the healthy positive bony architecture and the mucogingival junction, and using a universal and a Gracey 13/14 curette. The probability of passing the OSCE was significantly correlated with Class B (p = 0.035) and the practical examination score (p = 0.03) while not associated with gender (p = 0.53) and the written examination score (p = 0.11).

CONCLUSION

Students showed deficiencies in assessment skills at the end of preclinical education. The study findings suggest that the implementation of the OSCE at the conclusion of preclinical education would be beneficial since the written examination score might not accurately reflect student readiness for clinical patient care.

Early formative objective structured clinical examinations for students in the pre-clinical years of medical education: A non-randomized controlled prospective pilot study

BACKGROUND

The value of formative objective structured clinical examinations (OSCEs) during the pre-clinical years of medical education remains unclear. We aimed to assess the effectiveness of a formative OSCE program for medical students in their pre-clinical years on subsequent performance in summative OSCE.

METHODS

We conducted a non-randomized controlled prospective pilot study that included all medical students from the last year of the pre-clinical cycle of the Université Paris-Cité Medical School, France, in 2021. The intervention group received the formative OSCE program, which consisted of four OSCE sessions, followed by debriefing and feedback, whereas the control group received the standard teaching program. The main objective of this formative OSCE program was to develop skills in taking a structured medical history and communication. All participants took a final summative OSCE. The primary endpoint was the summative OSCE mark in each group. A questionnaire was also administered to the intervention-group students to collect their feedback. A qualitative analysis, using a convenience sample, was conducted by gathering data pertaining to the process through on-site participative observation of the formative OSCE program.

RESULTS

Twenty students were included in the intervention group; 776 in the control group. We observed a significant improvement with each successive formative OSCE session in communication skills and in taking a structured medical history (p<0.0001 for both skills). Students from the intervention group performed better in a summative OSCE that assessed the structuring of a medical history (median mark 16/20, IQR [15; 17] versus 14/20, [13; 16], respectively, p = 0.012). Adjusted analyses gave similar results. The students from the intervention group reported a feeling of improved competence and a reduced level of stress at the time of the evaluation, supported by the qualitative data showing the benefits of the formative sessions.

CONCLUSION

Our findings suggest that an early formative OSCE program is suitable for the pre-clinical years of medical education and is associated with improved student performance in domains targeted by the program.

The Objective Structured Clinical Examination (OSCE) in Periodontology with Simulated Patient: The Most Realistic Approach to Clinical Practice in Dentistry

The objective structured clinical examination (OSCE) is becoming an increasingly established assessment test in dental schools. The use of simulated patients in the OSCE makes the stations more similar to clinical practice. Therefore, the student can show their technical and clinical knowledge, and certainly, their ability to manage the patient. These sorts of tests, in which simulated patients can be included, would be used before the student started clinical practice with patients and/or at the end of the degree. The objective of this work was to describe how the periodontology station was developed using a simulated patient for students of a fifth year dentistry degree taking an OSCE test. Furthermore, a questionnaire was created to learn the perception of the students about this station and its characteristics. The fifth year students at the European University of Madrid positively evaluated this station in their examination. In addition, it was recorded that they preferred a simulated patient in their tests, rather than stations with clinical cases, images, X-rays, and presentations. It is essential that once the OSCE has been completed, the student receives a feedback to learn where they have failed and, therefore, be able to improve any of the aspects evaluated in the station.

Dental hygiene educators' knowledge and implementation of objective structured clinical examination in United States dental hygiene programs

PURPOSE

In 2021, the American Dental Association (ADA) announced the intention to develop a standardized dental hygiene licensure objective structured clinical examination (DHLOSCE). The purpose of this study was to measure the United States (US) dental hygiene (DH) educators' foundational knowledge of OSCE development and delivery in light of the impending development of the DHLOSCE by the ADA's Testing Services.

METHODS

The study was determined to be exempt from Institutional Review Board oversight. A 21-question survey was developed, pilot tested, and electronically disseminated through Qualtrics. The survey recruitment was emailed to the directors of all entry-level DH education programs in the US (n = 328), asking them to participate in the survey and to forward it to the clinical faculty in their institutions. Descriptive and inferential statistics were utilized to analyze the data.

RESULTS

There were 143 study participants, for a completion rate of 45%. Over two-thirds of respondents (64%) were unaware of the plans to develop the DHLOSCE, while 13% reported utilizing OSCEs to meet accreditation standards. Only 3% reported receiving a formal education in OSCE development compared to 29% who learned through a colleague or peer. Nearly half reported a lack of faculty experience as a barrier to OSCE implementation. Over three-quarters, 76% reported a lack of OCSE development committees within their program and only 14% had experience developing an OSCE station.

CONCLUSION

The study results suggest an urgent need for the development of OSCE training resources specific to DH education, as programs across the US prepare for the impending DHLOSCE.

The performance of the two-year postgraduate year dentist training program in Taiwan

Background/purpose

The postgraduate year dentist training program (PGYD) officially implemented in 2010. This study aimed to assess PGYD trainees’ subjective satisfaction perception and objective competence performance according to different training institutions (either dental clinics or hospitals).

Materials and methods

A nationwide cross-sectional survey was conducted in 2018. Subjective satisfaction questionnaires from 222 PGYD trainees and 166 PGYD trainees’ scores of objective structured clinical examinations (OSCEs) were collected for analysis. The t-test and logistic regression were used to compare differences between two groups.

Results

In subjective satisfaction, PGYD trainees in hospitals revealed less positive perceptions on teachers (odds ratios [ORs] range: 0.33–0.7) and on training plans and auxiliary facilities (ORs range: 0.23–0.69), but they had more opportunities to attend and present at professional meetings than those in clinics. In PGYD trainees’ opinion, the optimal training period allocation was 25% in hospitals and 75% in clinics. Overall, trainees in hospitals had better OSCE scores than those in clinics (OR [95% confidence interval, CI]: 3.12 [1.68–5.79]), except for the item of “the relation between physical condition and dental treatment outcome” (0.3 [0.1–0.97]).

Conclusion

PGYD trainees in dental clinics have more positive perceptions on teachers and on training plans and auxiliary facilities, but PGYD trainees in hospitals have better OSCE scores. PGYD trainees prefer to be trained in clinics for a better connection with future careers, while hospitals can offer better training for PGYD trainees to become independent dentists because they have better training environment and more educational resources.

Comparison of Digital Self-Assessment Systems and Faculty Feedback for Tooth Preparation in a Preclinical Simulation

Background: Regarding the new dental licensing regulations in Germany (AOZ), this study evaluated the effectiveness of two different digital tooth preparation validation systems in comparison to traditional faculty feedback. Methods: Participants were randomly divided into groups: Faculty Feedback (FF: n = 33), PrepCheck^® (PC: n = 32) and Dental Teacher™ (n = 32). Students had the task to prepare tooth 16 for a retentive full-cast crown. Preparations could be repeated as often as desired. Feedback was provided either by faculty staff or by digital validation systems only. Exams were conducted and graded by two independent and experienced examiners. A survey was performed to evaluate the assessment concepts. Results: No statistical difference in examination performance between groups could be observed. Nevertheless, the survey showed participants preferred consulting the faculty staff rather than the digital validation systems. Students preferred practising with DT rather than with PC. Conclusions: Although both classical and digital methods showed comparable results regarding the preparation examination performance, direct feedback by the faculty staff is still appreciated by the students. A combination of both methods is mandatory since demonstration and advice by the teacher is needed. However, digital tooth preparation validation systems are predestined for free practice sessions, providing self-assessment.

Factors associated with student performance on the medical residency test

OBJECTIVE

To determine whether the scores of the Progress test, the Skills and Attitude test, and the medical internship are correlated with the medical residency exam performance of students who started medical school at the Federal University of São Paulo in 2009.

METHODS

The scores of 684 Progress tests from years 1-6 of medical school, 111 Skills and Attitude exams (5th year), 228 performance coefficients for the 5th and 6th years of internship, and 211 scores on the medical residency exam were analyzed longitudinally. Correlations between scores were assessed by Pearson's correlation. Factors associated with medical residency scores were analyzed by linear regression.

RESULTS

Scores of Progress tests from years 1-6 and the Skills and Attitude test showed at least one moderate and significant correlation with each other. The theoretical exam and final exam scores in the medical residency had a moderate correlation with performance in the internship. The score of the theoretical medical residency exam was associated with performance in internship year 6 (β=0.833; p<0.001), and the final medical residency exam score was associated with the Skills and Attitude score (β=0.587; p<0.001), 5th-year internship score, (β=0.060; p=0.025), and 6th-year Progress test score (β=0.038; p=0.061).

CONCLUSIONS

The scores of these tests showed significant correlations. The medical residency exam scores were positively associated with the student's performance in the internship and on the Skills test, with a tendency for the final medical residency exam score to be associated with the 6th-year Progress test.

Rethinking Assessment Concepts in Dental Education

Introduction

Dental education involves teaching and assessing the acquisition of verifiable domains that require superior psychomotor, communication, and cognitive skills. Evolving technologies and methods of assessment could enhance student's learning environment and improve tutor assessment experience. The aim of this study was to review the current body of research and evaluate the effectiveness of various methods of assessments in improving learning and performance in preclinical and clinical dental practice.

Materials and Methods

A search strategy was implemented using electronic search in major databases. The following key terms, clinical skills, preclinical, dental students, and assessment, were included in the search. Two reviewers independently screened all the articles retrieved following very specific inclusion criteria.

Results

The initial search generated 5371 articles and 24 articles were selected for review and data extraction. Cohen's kappa coefficient was used to measure interrater agreement and a score of 94.7% was obtained.

Conclusion

Preclinical assessment is an effective tool for promoting skills transfer to clinical phase. Early psychomotor skills assessment is valuable. It allows early intervention in the learning process and assists in effective utilization of learning resources. Technology-enhanced assessment systems allow better patient simulation, enhance learning and self-assessment experiences, and improve performance evaluation. However, these systems serve as an adjunct to conventional assessment methods. Further research should aim at calibrating and integrating these systems to optimize students learning and performance.

Stress, anxiety, self-efficacy, and the meanings that physical therapy students attribute to their experience with an objective structured clinical examination

BACKGROUND

Excessive stress and anxiety can impair learning. The objective structured clinical examination (OSCE) is a valuable tool to assess and promote the acquisition of clinical skills. However, significant OSCE-related stress and anxiety are frequently reported. The aim of this study was to investigate the relationships between physiological stress, self-reported levels of anxiety due to an OSCE, self-efficacy, and the meanings that physical therapy students attribute to their experience with the exam.

DESIGN

Concurrent mixed methods study.

METHODS

A total of 32 students took part in this study. All were enrolled in the third semester of a 10-semester Physical Therapy Bachelor Program. Salivary cortisol levels, self-reported anxiety (State-Trait Anxiety Inventory, STAI) were measured before the OSCE. Exam scores and self-efficacy ratings were also recorded. Correlations between variables were tested with the Pearson correlation, with ɑ at 0.05. Semi-structured interviews were used to explore the personal perspectives of students. Thematic analysis was used to investigate emergent themes.

RESULTS

Trait anxiety scores were significantly higher than normative values (p < 0.001). A high proportion of students showed high (STAI> 49) state anxiety (37.5%) and trait anxiety (65.6%). Salivary cortisol was not associated anxiety (p > 0.05). Neither stress nor anxiety correlated with OSCE scores. A moderate and significant direct correlation was found for self-efficacy scores and OSCE scores (r = 0.475, p = 0.007). Students reported that confidence had a calming effect and led to better self-perceived performance. They also reported that the OSCE can provide meaningful learning experiences despite being stressful.

CONCLUSIONS

A high proportion of our students reported a stable/lingering negative affect. However, neither stress nor anxiety related to OSCE scores. Students' confidence in their capabilities was correlated with their performance. Their subjective reports suggest that self-confidence may have protected them from the negative effects of stress and anxiety on academic performance.

Human disease/clinical medical sciences in dentistry: Current state and future development of undergraduate assessments in the UK and Ireland

INTRODUCTION

The United Kingdom and Ireland teachers of Human Disease/Clinical Medical Science for Dentistry (HD/CMSD) group continue to work together and most recently collaborated to review current and future assessments.

MATERIALS AND METHODS

The first part of the review of assessments in HD/CMSD took place at a face-to-face meeting with presentations from delegates on assessments in their home institutions. The second and larger part comprised an online survey where all eighteen schools in the UK and Ireland participated.

RESULTS

All schools had some element of formative assessment, and the majority had a stand-alone summative assessment at the end of the HD/CMSD teaching block. Most schools had a written paper and practical elements to their assessments, most commonly a combination of a multiple-choice type question combined with an objective structured clinical examination (OSCE). There was a trend towards the use of single best answer (SBA) questions and a willingness amongst participants to share a question bank. All schools incorporated elements of HD/CMSD in their final examinations.

DISCUSSION AND CONCLUSION

This collaboration promoted the sharing of developments in assessment for HD/CMSD and demonstrated a willingness to cooperate between institutions. Assessment in HD/CMSD in the UK and Ireland continues to be refined by those responsible for its content and delivery, and assessment methods are progressing following evidence-based best practice.

Why So Stressed? A Descriptive Thematic Analysis of Physical Therapy Students' Descriptions of Causes of Anxiety during Objective Structured Clinical Exams

Purpose: The purpose of this study was to collect and conduct a descriptive content analysis of the primary triggers of practical exam anxiety in Master of Physical Therapy (MPT) students in a Canadian university programme. Method: First and second-year MPT students were invited to reflect upon their top 5 sources or triggers of OSCE exam anxiety, collected in written format during a low-stress, low-examination period of their programme. All participants had participated in at least 3 OSCEs before providing data. The emergent themes were member-checked with 10 of the original participants to improve trustworthiness of the results. Results: 56 of a possible 105 students provided 224 triggers of OSCE anxiety. Thematic content analysis revealed 6 emergent meta-themes that adequately captured all triggers. They were: social performance anxiety, fear of lacking competence, overvaluing the outcome, fear of the unknown, impaired personal health/coping resources, and operational/procedural influences. These meta-themes were endorsed by the participant sub-group. Conclusions: OSCEs are common forms of evaluation in MPT training programmes, but are also highly anxiogenic. The first step toward mitigating exam anxiety, thereby ensuring exam performance is less confounded by anxiety, is to identify the common triggers. Confidence in results will be strengthened by replication in other programmes.

Assessing Dental Hygiene Students' Test Anxiety in Examiner-Administered vs. Camera-Supervised OSCEs: A Pilot Study

The aim of this pilot study was to measure and compare dental hygiene students' level of text anxiety and performance when being tested on an objective structured clinical examination (OSCE) using camera supervision versus direct examiner observation. This randomized, controlled trial, conducted in October 2016, assessed students' anxiety levels and performance using the Zung Self-Rating Anxiety Scale and performance on an OSCE. All 32 first-year students in one U.S. dental hygiene program were invited to participate, and all 32 participated for a 100% response rate. Participants were randomly assigned to the direct observation (N=16) and camera-supervised (N=16) groups. The comparison of pre-OSCE anxiety levels and performance scores evaluated with direct observation (36.4 and 20.4) versus camera-supervised (37.5 and 20.9) found no statistically significant difference between the two groups (p=0.496, p=0.403). The difference in post-OSCE anxiety levels between the direct observation and camera-supervised groups (33.8 and 37.8) approached significance (p=0.051). This study found that student anxiety levels were similar between the two groups except for the post-OSCE direct observation anxiety levels, which were lower. No significant difference was found between the two groups' performance scores. Due to similar performance scores and anxiety levels for the direct examiner and camera-supervised methods, the camera-supervised OSCE warrants further exploration as an alternative approach to direct observation for evaluation of students' performance.

Application of OSCE for Stage Assessment in Standardized Training for Oral Residents

The aim of this study was to establish and implement an objective structured clinical examination (OSCE) for stage assessment in standardized training for oral residents, evaluating its validity and suitability. An OSCE was established for stage assessment of 158 residents in a program in Jiangsu Province, China. Its validity and suitability were evaluated using indices such as reliability and discrimination coefficients of assessment results. The established OSCE had eight stations: interpretation of auxiliary examination results, basic knowledge, doctor-patient communication and clinical examination, clinical case analysis, medical record and prescription writing, public skills, first aid skills, and specialist's operation skills. The mean overall score and consistency coefficient (Cronbach's alpha) for the test subjects were 83.64±3.69 points and 0.732, respectively. The developed OSCE was reasonably established, with credible assessment results, and indices such as mean and discrimination coefficient of test scores from each station were appropriate. Therefore, the proposed protocol was found to be reliable and suitable.

Do coursework summative assessments predict clinical performance? A systematic review

BACKGROUND

Two goals of summative assessment in health profession education programs are to ensure the robustness of high stakes decisions such as progression and licensing, and predict future performance. This systematic and critical review aims to investigate the ability of specific modes of summative assessment to predict the clinical performance of health profession education students.

METHODS

PubMed, CINAHL, SPORTDiscus, ERIC and EMBASE databases were searched using key terms with articles collected subjected to dedicated inclusion criteria. Rigorous exclusion criteria were applied to ensure a consistent interpretation of 'summative assessment' and 'clinical performance'. Data were extracted using a pre-determined format and papers were critically appraised by two independent reviewers using a modified Downs and Black checklist with level of agreement between reviewers determined through a Kappa analysis.

RESULTS

Of the 4783 studies retrieved from the search strategy, 18 studies were included in the final review. Twelve were from the medical profession and there was one from each of physiotherapy, pharmacy, dietetics, speech pathology, dentistry and dental hygiene. Objective Structured Clinical Examinations featured in 15 papers, written assessments in four and problem based learning evaluations, case based learning evaluations and student portfolios each featured in one paper. Sixteen different measures of clinical performance were used. Two papers were identified as 'poor' quality and the remainder categorised as 'fair' with an almost perfect (k = 0.852) level of agreement between raters. Objective Structured Clinical Examination scores accounted for 1.4-39.7% of the variance in student performance; multiple choice/extended matching questions and short answer written examinations accounted for 3.2-29.2%; problem based or case based learning evaluations accounted for 4.4-16.6%; and student portfolios accounted for 12.1%.

CONCLUSIONS

Objective structured clinical examinations and written examinations consisting of multiple choice/extended matching questions and short answer questions do have significant relationships with the clinical performance of health professional students. However, caution should be applied if using these assessments as predictive measures for clinical performance due to a small body of evidence and large variations in the predictive strength of the relationships identified. Based on the current evidence, the Objective Structured Clinical Examination may be the most appropriate summative assessment for educators to use to identify students that may be at risk of poor performance in a clinical workplace environment. Further research on this topic is needed to improve the strength of the predictive relationship.

Developing doctors: what are the attitudes and perceptions of year 1 and 2 medical students towards a new integrated formative objective structured clinical examination?

BACKGROUND

Objective Structured Clinical Examination (OSCE) is a core component of undergraduate medical student assessment. With increased emphasis on integrated programmes, more courses include OSCEs in the traditionally 'preclinical' years. The acceptability and impact of such assessment methods at this stage of training is unknown.

METHODS

In 2013 and 2014 University College London Medical School piloted a formative, integrated OSCE in years 1 and 2 of the undergraduate medical degree programme. This study with a sequential exploratory design aimed to explore the acceptability and impact of such an OSCE in the early years of medical school. 1280 students completed the OSCE and were invited to complete a questionnaire exploring their views of the OSCE (response rate 96.6%). Four focus groups, each with five or six participants (22 in total) were subsequently held to further explore themes. Data was independently transcribed and coded using thematic analysis.

RESULTS

Students were strongly in favour of the introduction of an OSCE in addition to existing assessments. Six overarching themes were identified: application of knowledge and skills; OSCE as an experience; OSCE as a process; a learning curve; becoming a doctor; and creating an effective OSCE.

CONCLUSIONS

Results strongly support the role of OSCE early in the medical course with many benefits reported. An OSCE at this stage aligns with the vision of integrated medical education which includes early patient contact and introduction of clinical and professional skills. It also fosters feelings of responsibility and professional identity amongst students.

The assessment of undergraduate medical students' satisfaction levels with the objective structured clinical examination

Background:

The objective structured clinical examination (OSCE) has been introduced as an efficient method for the assessment of medical students.

Objectives:

The aim of the present study was to determine the satisfaction level of undergraduate medical students of internal medicine department with the OSCE.

Materials and Methods:

This was a descriptive cross-sectional study, performed on all available undergraduate students at the end of their internal medicine training period in Mashhad University of Medical Sciences. The students responded to 15 multiple-choice questions with confirmed validity and reliability.

Results:

The majority of the students (94.5%) had a positive attitude toward the OSCE and mentioned that the OSCE format was a more appropriate type of exam than other methods of testing; however, 79.1% thought that the OSCE format was stressful. In addition, the participants’ sex had no effect on their level of satisfaction with the examination. Likewise, there was no significant correlation between their level of satisfaction and their age, marital status, or lack of previous experience with this type of exam.

Conclusions:

If the exam standards are met and a uniform dispersion of the scientific content is maintained, the OSCE method of assessment can be recommended as an efficient and applicable method for assessing medical students.