Simulation-based clinical assessment identifies threshold competence to practise physiotherapy in Australia: a crossover trial

BACKGROUND

Although evidence exists for the efficacy of high-fidelity simulation as an educational tool, there is limited evidence for its application in high-stakes professional threshold competency assessment. An alternative model of simulation-based assessment was developed by the Australian Physiotherapy Council (APC), using purpose-written standardised patients, mapped to the appropriate threshold level. The aim of this two-phase study was to investigate whether simulation-based clinical assessments resulted in equivalent outcomes to standard, real-life assessments for overseas-trained physiotherapists seeking registration to practice in Australia.

METHODS

A randomised crossover trial comparing simulation-based assessment to real-life assessment was completed. Participants were internationally trained physiotherapists applying for registration to practice in Australia, voluntarily recruited from the Australian Physiotherapy Council (APC) assessment waiting list: study 1 n = 25, study 2 n = 144. Study 1 participants completed usual APC real-life assessments in 3 practice areas, completed on different days at APC partner healthcare facilities. Participants also underwent 3 practice area-matched simulation-based assessments, completed on the same day at purpose-designed simulation facilities. Study 2 participants completed 3 simulation-based assessments and 1 real-life assessment that was randomly allocated for order and practice area. Assessment of competency followed the standard APC procedure of 90-minute examinations using The Moderated Assessment Form (MAF).

RESULTS

The overall pass rate was higher for real-life assessments in both studies: study 1, 50% versus 42.7%; study 2, 55.6% versus 44.4%. Chi-square analysis showed a high to moderate level of exact matching of pass/fail grades across all assessments: study 1, 73.4% (p < 0.001); study 2, 58.3% (p = 0.027). Binary logistic regression showed that the best predictors of real-life pass/fail grade were simulation-based MAF pass/fail grade (study 1, OR 7.86 p < 0.001; study 2, OR 2.037, p = 0.038) and simulation-based total MAF score (study 1, OR 1.464 p < 0.001; study 2, OR 1.234, p = 0.001).

CONCLUSION

Simulation-based assessment is a significant predictor of clinical performance and can be used to successfully identify high stakes threshold competence to practice physiotherapy in Australia.

Development and inclusion of an entrustable professional activity (EPA) scale in a simulation-based medicine dispensing assessment

BACKGROUND AND PURPOSE

Effective, safe, and patient-centred dispensing is a core task of community pharmacists. Entrustable professional activities (EPAs) offer a way of defining and assessing these daily practice activities. Although EPAs have become popular within competency-based medical education programs, their use is new to pharmacy education and assessment.

EDUCATIONAL ACTIVITY AND SETTING

A simulation-based assessment framework containing a scale of entrustment was developed to evaluate the readiness of Year 4 undergraduate pharmacy students to safely manage the supply of prescribed medicine(s) in a community pharmacy. The assessment framework was piloted in a fourth year "Transition to Practice" course with 28 simulation-based assessments conducted.

FINDINGS

An entrustment framework was developed and implemented successfully with Year 4 undergraduate pharmacy students. The EPA for medicine dispensing integrates competency domains that include information gathering, providing patient-centred care, clinical reasoning, medicine dispensing, and professional communications. On a scale ranging from level 1 to level 5, the majority (73%) of entrustment ratings were level 2 or level 3; and of the students who achieved different ratings between clinical scenarios, 75% of students improved on their second simulation attempt. There was a strong correlation between the global EPA ratings with the total score achieved across the domains.

SUMMARY

Using simulation-based assessment, entrustment decision making can be incorporated in "entry to profession" undergraduate and postgraduate pharmacy courses to assess students' readiness to transition between learning and professional practice.

Validation of time to task performance assessment method in simulation: A comparative design study

BACKGROUND

There is a lack of objective and valid measures for assessing nursing clinical competence which could adversely impact patient safety. Therefore, we evaluated an objective assessment of clinical competence, Time to Task (ability to perform specific, critical nursing care activities within 5 min), and compared it to two subjective measures, (Lasater Clinical Judgement Rubric [LCJR] and common "pass/fail" assessment).

DESIGN/METHODS

Using a prospective, "Known Groups" (Expert vs. Novice nurses) comparative design, Expert nurses (ICU nurses with >5 years of ICU experience) and Novice nurses (senior prelicensure nursing students) participated individually in a simulation of a patient in decompensated heart failure. Fourteen nursing instructors or preceptors, blinded to group assignment, reviewed 28 simulation videos (15 Expert and 13 Novice) and scored them using the LCJR and pass/fail assessments. Time to Task assessment was scored based on time thresholds for specific nursing actions prospectively set by an expert clinical panel. Statistical analysis consisted of Medians Test and sensitivity and specificity analyses.

RESULTS

The LCJR total score was significantly different between Experts and Novices (p < 0.01) and revealed adequate sensitivity (ability to correctly identify "Expert" nurses; 0.72) but had a low specificity (ability to correctly identify "Novice" nurses; 0.40). For the subjective measure 'pass/fail', sensitivity was high (0.90) but specificity was low (0.47). The Time to Task measure had statistical significance between Expert and Novice groups (p < 0.01) and sensitivity (0.80) and specificity (0.85) were good.

CONCLUSION

Commonly used subjective measures of clinical nursing competence have difficulties with achieving acceptable specificity. However, an objective measure, Time to Task, had good sensitivity and specificity in differentiating between groups. While more than one assessment instrument should be used to determine nurse competency, an objective measure, such as Time to Task, warrants further study.

Cognitive load predicts point-of-care ultrasound simulator performance

INTRODUCTION

The ability to maintain good performance with low cognitive load is an important marker of expertise. Incorporating cognitive load measurements in the context of simulation training may help to inform judgements of competence. This exploratory study investigated relationships between demographic markers of expertise, cognitive load measures, and simulator performance in the context of point-of-care ultrasonography.

METHODS

Twenty-nine medical trainees and clinicians at the University of Toronto with a range of clinical ultrasound experience were recruited. Participants answered a demographic questionnaire then used an ultrasound simulator to perform targeted scanning tasks based on clinical vignettes. Participants were scored on their ability to both acquire and interpret ultrasound images. Cognitive load measures included participant self-report, eye-based physiological indices, and behavioural measures. Data were analyzed using a multilevel linear modelling approach, wherein observations were clustered by participants.

RESULTS

Experienced participants outperformed novice participants on ultrasound image acquisition. Ultrasound image interpretation was comparable between the two groups. Ultrasound image acquisition performance was predicted by level of training, prior ultrasound training, and cognitive load. There was significant convergence between cognitive load measurement techniques. A marginal model of ultrasound image acquisition performance including prior ultrasound training and cognitive load as fixed effects provided the best overall fit for the observed data.

DISCUSSION

In this proof-of-principle study, the combination of demographic and cognitive load measures provided more sensitive metrics to predict ultrasound simulator performance. Performance assessments which include cognitive load can help differentiate between levels of expertise in simulation environments, and may serve as better predictors of skill transfer to clinical practice.