Assessing the assessment in emergency care training

The Effects of Reflective Pauses on Performance in Simulation Training

INTRODUCTION

The reflective pause, taking a pause during performance to reflect, is an important practice in simulation-based learning. However, for novice learners, it is a highly complex self-regulatory skill that cannot stand alone without guidance. Using educational theories, we propose how to design cognitive and metacognitive aids to guide learners with the reflective pause and investigate its effects on performance in a simulation training environment.

METHODS

These effects are examined in four aspects of performance: cognitive load, primary performance, secondary performance, and encapsulation. Medical students ( N = 72) performed tasks in simulation training for emergency medicine, under 2 conditions: reflection condition ( n = 36) where reflection was prompted and guided, and control condition ( n = 36) without such reflection.

RESULTS

The effects of reflective pauses emerged for 2 aspects of performance: cognitive load decreased and secondary performance improved. However, primary performance and encapsulation did not show significant difference.

CONCLUSIONS

The results demonstrate that reflective pauses with cognitive and metacognitive aids implemented can enhance some aspects of performance. We suggest that to secure these effects, feedback during reflection and an adaptation period should be provided.

Development and national consensus finding on patient-centred high stakes communication skills assessments for the Swiss Federal Licensing Examination in Medicine

OBJECTIVE

To describe and evaluate a consensus finding and expert validation process for the development of patient-centred communication assessments for a national Licensing Exam in Medicine.

METHODS

A multi-professional team of clinicians and experts in communication, assessment and role-play developed communication assessments for the Swiss Federal Licensing Examination. The six-month process, informed by a preceding national needs-assessment, an expert symposium and a critical literature review covered the application of patient-centred communication frameworks, the development of assessment guides, concrete assessments and pilot-tests. The participants evaluated the process.

RESULTS

The multiple-step consensus process, based on expert validation of the medical and communication content, led to six high-stakes patient-centred communication OSCE-assessments. The process evaluation revealed areas of challenge such as calibrating rating-scales and case difficulty to the graduates' competencies and integrating differing opinions. Main success factors were attributed to the outcome-oriented process and the multi-professional exchange of expertise. A model for developing high stakes patient-centred communication OSCE-assessments was derived.

CONCLUSIONS

Consensus finding was facilitated by using well-established communication frameworks, by ensuring outcome-orientated knowledge exchange among multi-professional experts, and collaborative validation of content through experts.

PRACTICE IMPLICATIONS

We propose developing high-stakes communication assessments in a multi-professional expert consensus and provide a conceptual model.

Global versus task-specific postoperative feedback in surgical procedure learning

BACKGROUND

Task-specific checklists and global rating scales are both recommended assessment tools to provide constructive feedback on surgical performance. This study evaluated the most effective feedback tool by comparing the effects of the Observational Clinical Human Reliability Analysis (OCHRA) and the Objective Structured Assessment of Technical Skills (OSATS) on surgical performance in relation to the visual-spatial ability of the learners.

METHODS

In a randomized controlled trial, medical students were allocated to either the OCHRA (n = 25) or OSATS (n = 25) feedback group. Visual-spatial ability was measured by a Mental Rotation Test. Participants performed an open inguinal hernia repair procedure on a simulation model twice. Feedback was provided after the first procedure. Improvement in performance was evaluated blindly using a global rating scale (performance score) and hand-motion analysis (time and path length).

RESULTS

Mean improvement in performance score was not significantly different between the OCHRA and OSATS feedback groups (P = .100). However, mean improvement in time (371.0 ± 223.4 vs 274.6 ± 341.6; P = .027) and path length (53.5 ± 42.4 vs 34.7 ± 39.0; P = .046) was significantly greater in the OCHRA feedback group. When stratified by mental rotation test scores, the greater improvement in time (P = .032) and path length (P = .053) was observed only among individuals with low visual-spatial abilities.

CONCLUSION

A task-specific (OCHRA) feedback is more effective in improving surgical skills in terms of time and path length in novices compared to a global rating scale (OSATS). The effects of a task-specific feedback are present mostly in individuals with lower visual-spatial abilities.

Translation and further validation of a global rating scale for the assessment of clinical competence in prehospital emergency care

Global rating scales are useful to assess clinical competence at a general level based on specific word dimensions. The aim of this study was to translate and culturally adapt the Paramedic Global Rating Scale, and to contribute validity evidence and instrument usefulness in training results and clinical competence assessments of students undergoing training to become ambulance nurses and paramedics at Swedish and Finnish universities. The study included translation, expert review and inter-rater reliability (IRR) tests. The scale was translated and culturally adapted to clinical and educational settings in both countries. A content validity index (CVI) was calculated using eight experts. IRR tests were performed with five registered nurses working as university lecturers, and with six clinicians working as ambulance nurses. They individually rated the same simulated ambulance assignment. Based on the ratings IRR was calculated with intra-class correlation (ICC). The scale showed excellent CVI for items and scale. The ICC indicated substantial agreement in the group of lecturers and a high degree of agreement in the group of clinicians. This study provides validity evidence for a Swedish version of the scale, supporting its use in measuring clinical competence among students undergoing training to become ambulance nurses and paramedics.

Performance Assessment of Emergency Teams and Communication in Trauma Care (PERFECT checklist)-Explorative analysis, development and validation of the PERFECT checklist: Part of the prospective longitudinal mixed-methods EPPTC trial

BACKGROUND

Trainings in emergency medicine are well structured, but examinations are rarely validated. We are evaluating the impact of pre-hospital emergency trainings on participants and patient care and developed and validated a checklist to assess emergency trainings.

METHODS

We used videos recorded at the time points directly before (t0), directly after (t1), and one year after (t2) training to develop the PERFECT checklist (Performance Assessment of Emergency Teams and Communication in Trauma Care). The videos were assessed using semi-qualitative/linguistic analysis as well as expert panel appraisal and recommendations using the Delphi method. The checklist was tested for validity and reliability.

RESULTS

The inter-rater reliability (ICC = 0.99) and internal consistency (α = 0.99) were high. Concurrent validity was moderate to high (r = 0.65 -r = 0.93 (p<0.001)). We included scales for procedures, non-technical skills, technical skills and global performance. The procedures were done faster in the mean over the timeline (t0: 2:29, 95%CI 1:54-3:03 min., t1: 1:11, 95%C 0:53-1:30 min, t2: 1:14, 95%CI 0:56-1:31 min.). All experts rated the recorded scenarios at t0 with the lowest sum score (mean 31±8), with a significantly better performance of the teams at t1 (mean 69±7). The performance at t2 (mean 66 ± 13) was slightly lower than at t1, but still better than at t0. At t1 and t2, linguistic analysis showed a change in the team leaders communication behaviour, which can be interpreted as a surrogate parameter for reduced stress.

CONCLUSION

The PERFECT checklist has a good validity and high reliability for assessing trauma procedures and teamwork.

Preparing Residents Effectively in Emergency Skills Training With a Serious Game

Supplemental digital content is available in the text.

Introduction

Training emergency care skills is critical for patient safety but cost intensive. Serious games have been proposed as an engaging self-directed learning tool for complex skills. The objective of this study was to compare the cognitive skills and motivation of medical residents who only used a course manual as preparation for classroom training on emergency care with residents who used an additional serious game.

Methods

This was a quasi-experimental study with residents preparing for a rotation in the emergency department. The “reading” group received a course manual before classroom training; the “reading and game” group received this manual plus the game as preparation for the same training. Emergency skills were assessed before training (with residents who agreed to participate in an extra pretraining assessment), using validated competency scales and a global performance scale. We also measured motivation.

Results

All groups had comparable important characteristics (eg, experience with acute care). Before training, the reading and game group felt motivated to play the game and spent more self-study time (+2.5 hours) than the reading group. Game-playing residents showed higher scores on objectively measured and self-assessed clinical competencies but equal scores on the global performance scale and were equally motivated for training, compared with the reading group. After the 2-week training, no differences between groups existed.

Conclusions

After preparing training with an additional serious game, residents showed improved clinical competencies, compared with residents who only studied course material. After a 2-week training, this advantage disappeared. Future research should study the retention of game effects in blended designs.

Serious games and blended learning; effects on performance and motivation in medical education

INTRODUCTION

More efficient, flexible training models are needed in medical education. Information technology offers the tools to design and develop effective and more efficient training. The aims of this thesis were: 1) Compare the effectiveness of blended versus classroom training for the acquisition of knowledge; 2) Investigate the effectiveness and critical design features of serious games for performance improvement and motivation.

METHODS

Five empirical studies were conducted to answer the research questions and a descriptive study on an evaluation framework to assess serious games was performed.

RESULTS

The results of the research studies indicated that: 1) For knowledge acquisition, blended learning is equally effective and attractive for learners as classroom learning; 2) A serious game with realistic, interactive cases improved complex cognitive skills for residents, with limited self-study time. Although the same game was motivating for inexperienced medical students and stimulated them to study longer, it did not improve their cognitive skills, compared with what they learned from an instructional e‑module. This indicates an 'expertise reversal effect', where a rich learning environment is effective for experts, but may be contra-productive for novices (interaction of prior knowledge and complexity of format).

DISCUSSION

A blended design is equally effective and attractive as classroom training. Blended learning facilitates adaptation to the learners' knowledge level, flexibility in time and scalability of learning. Games may support skills learning, provided task complexity matches the learner's competency level. More design-based research is needed on the effects of task complexity and other design features on performance improvement, for both novices and experts.

An experimental study on the effects of a simulation game on students' clinical cognitive skills and motivation

Simulation games are becoming increasingly popular in education, but more insight in their critical design features is needed. This study investigated the effects of fidelity of open patient cases in adjunct to an instructional e-module on students' cognitive skills and motivation. We set up a three-group randomized post-test-only design: a control group working on an e-module; a cases group, combining the e-module with low-fidelity text-based patient cases, and a game group, combining the e-module with a high-fidelity simulation game with the same cases. Participants completed questionnaires on cognitive load and motivation. After a 4-week study period, blinded assessors rated students' cognitive emergency care skills in two mannequin-based scenarios. In total 61 students participated and were assessed; 16 control group students, 20 cases students and 25 game students. Learning time was 2 h longer for the cases and game groups than for the control group. Acquired cognitive skills did not differ between groups. The game group experienced higher intrinsic and germane cognitive load than the cases group (p = 0.03 and 0.01) and felt more engaged (p < 0.001). Students did not profit from working on open cases (in adjunct to an e-module), which nonetheless challenged them to study longer. The e-module appeared to be very effective, while the high-fidelity game, although engaging, probably distracted students and impeded learning. Medical educators designing motivating and effective skills training for novices should align case complexity and fidelity with students' proficiency level. The relation between case-fidelity, motivation and skills development is an important field for further study.