Cognitive diagnostic modelling in healthcare professions education: an eye-opener

Reliability, validity and acceptability of an online clinical reasoning simulator for medical students: An international pilot

INTRODUCTION

Clinical reasoning skills are essential for decision-making. Current assessment methods are limited when testing clinical reasoning and management of uncertainty. This study evaluates the reliability, validity and acceptability of Practicum Script, an online simulation-based programme, for developing medical students' clinical reasoning skills using real-life cases.

METHODS

In 2020, we conducted an international, multicentre pilot study using 20 clinical cases with 2457 final-year medical students from 21 schools worldwide. Psychometric analysis was performed (n = 1502 students completing at least 80% of cases). Classical estimates of reliability for three test domains (hypothesis generation, hypothesis argumentation and knowledge application) were calculated using Cronbach's alpha and McDonald's omega coefficients. Validity evidence was obtained by confirmatory factor analysis (CFA) and measurement alignment (MA). Items from the knowledge application domain were analysed using cognitive diagnostic modelling (CDM). Acceptability was evaluated by an anonymous student survey.

RESULTS

Reliability estimates were high with narrow confidence intervals. CFA revealed acceptable goodness-of-fit indices for the proposed three-factor model. CDM analysis demonstrated good absolute test fit and high classification accuracy estimates. Student survey responses showed high levels of acceptability.

CONCLUSION

Our findings suggest that Practicum Script is a useful resource for strengthening students' clinical reasoning skills and ability to manage uncertainty.

Measurement of uncertainty tolerance revisited

INTRODUCTION

Uncertainty tolerance (UT) is attracting increasing attention in medical education due to the numerous challenges associated with uncertainty in professional life. Inconsistencies in analysing the relationship between UT and moderators may arise from inadequate measurement methods. Most instruments were formulated before the most widely accepted framework was published. Our aim was to investigate the validity of an UT scale using an actual framework to corroborate with better and accurate instruments.

METHODS

A total of 1052 students were invited. Various psychometric methods were used to explore validity of the TAMSAD scale in light of actual framework. Classic exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were performed. Secondly, content item classification was triangulated with exploratory graph analysis (EGA), and the new EFA, CFA, and cognitive diagnostic modelling (CDM) analysis were conducted. The reliability was calculated using Cronbach's alpha and McDonald's omega.

RESULTS

A total of 694 students (65.9%) responded to the questionnaire. The reliability of the TAMSAD scale was 0.782. The initial EFA revealed no clear interpretable dimensions. The TAMSAD scale items can be classified into sources of uncertainty. The EGA has three dimensions, and the new EFA led to a 17-item TAMSAD scale with the following three dimensions: ambiguity, complexity, and probability. These dimensions lead to better adjustment fit indices in the new CFA and CDM analyses.

CONCLUSION

We found evidence that the TAMSAD scale can be considered a multidimensional scale, organised in terms of sources of uncertainty.