Assessors' interpretations of narrative data on communication skills in a summative OSCE

Longitudinal assessment of undergraduate dental students: Building evidence for validity

PURPOSE

To investigate the content and criterion validity, and reliability of longitudinal clinical assessment of undergraduate dental student clinical competence by determining patterns of clinical performance and comparing them with validated standalone undergraduate examinations.

METHODS

Group-based trajectory models tracking students' clinical performance over time were produced from LIFTUPP© data for three dental student cohorts (2017-19; n = 235) using threshold models based on the Bayesian information criterion. Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Criterion validity was investigated using performance indicator 5 to create distinct trajectories of performance before linking and cross-tabulating trajectory group memberships with a 'top 20%' performance in the final Bachelor of Dental Surgery (BDS) examinations. Reliability was calculated using Cronbach's alpha.

RESULTS

Threshold 4 models showed all students followed a single upward trajectory in all three cohorts, showing clear progression in competence over three clinical BDS years. A threshold 5 model produced two distinct trajectories, and in each cohort a 'better performing' trajectory was identified. Students allocated to the 'better performing' trajectories scored higher on average in the final examinations for cohort 2 (29% vs 18% (BDS4); 33% vs. 15% (BDS5)) and cohort 3 (19% vs. 16% (BDS4); 21% vs. 16% (BDS5)). Reliability for the undergraduate examinations was high for all three cohorts (≥0.8815) and did not change appreciably when longitudinal assessment was included.

CONCLUSIONS

There is some evidence to support that longitudinal data have a degree of content and criterion validity for assessing the development of clinical competence in undergraduate dental students, which should increase confidence in decisions based on these data. The findings also provide a good foundation for subsequent research.

Cognitive load theory: Implications for assessment in pharmacy education

The concept of mental workload is well studied from a learner's perspective but has yet to be better understood from the perspective of an assessor. Mental workload is largely associated with cognitive load theory, which describes three different types of load. Intrinsic load deals with the complexity of the task, extraneous load describes distractors to the task at hand, and germane load focuses on the development of schemas in working memory for future recall. Studies from medical education show that all three types of load are relevant when considering rater -based assessment (e.g. Objective Structured Clinical Examinations (OSCEs), or experiential training). Assessments with high intrinsic and extraneous load may interfere with assessors' attention and working memory and result in poorer quality assessment. Reducing these loads within assessment tasks should therefore be a priority for pharmacy educators. This commentary aims to provide a theoretical overview of mental workload in assessment, outline research findings from the medical education context, and propose strategies to be considered for reducing mental workload in rater-based assessments relevant to pharmacy education. Suggestions for future research are also addressed.

Assessing Advanced Communication Skills via Objective Structured Clinical Examination: A Comparison of Faculty Versus Self, Peer, and Standardized Patient Assessors

Construct: The construct addressed in this study is assessment of advanced communication skills among senior medical students. Background: The question of who should assess participants during objective structured clinical examinations (OSCEs) has been debated, and options discussed in the literature have included peer, self, standardized patient, and faculty assessment models. What is not known is whether same-level peer assisted learning can be utilized for formative assessment of advanced communication skills when no faculty, standardized patients, or other trained assessors are involved in providing feedback. If successful, such an educational model would optimize resource utilization and broaden the scope of topics that could be covered in formative OSCEs. Approach: The investigators developed a 4-station formative OSCE focused on advanced communication skills for senior medical students, and evaluated the concordance of assessment done by same-level peers, self, standardized patients, and faculty for 45 students. After each station, examinees completed a self-assessment checklist and received checklist-based assessment and verbal feedback from same-level peers only. Standardized patients completed checklist-based assessments outside the room, and faculty did so after the OSCE via video review; neither group provided direct feedback to examinees. The investigators assessed inter-rater agreement and mean difference scores on the checklists using faculty score as the gold standard. Findings: There was fair to good overall agreement among self, same-level peer, standardized patient, and faculty-assessment of advanced communication skills. Relative to faculty, peer and standardized patient assessors overestimated advanced communication skills, while self-assessments underestimated skills. Conclusions: Self and same-level peer-assessment may be a viable alternative to faculty assessment for a formative OSCE on advanced communication skills for senior medical students.