Development of a novel global rating scale for objective structured assessment of technical skills in an emergency medical simulation training

BACKGROUND

Medical simulation trainings lead to an improvement in patient care by increasing technical and non-technical skills, procedural confidence and medical knowledge. For structured simulation-based trainings, objective assessment tools are needed to evaluate the performance during simulation and the learning progress. In surgical education, objective structured assessment of technical skills (OSATS) are widely used and validated. However, in emergency medicine and anesthesia there is a lack of validated assessment tools for technical skills. Thus, the aim of the present study was to develop and validate a novel Global Rating Scale (GRS) for emergency medical simulation trainings.

METHODS

Following the development of the GRS, 12 teams of different experience in emergency medicine (4th year medical students, paramedics, emergency physicians) were involved in a pre-hospital emergency medicine simulation scenario and assessed by four independent raters. Subsequently, interrater reliability and construct validity of the GRS were analyzed. Moreover, the results of the GRS were cross-checked with a task specific check list. Data are presented as median (minimum; maximum).

RESULTS

The GRS consists of ten items each scored on a 5-point Likert scale yielding a maximum of 50 points. The median score achieved by novice teams was 22.75 points (17;30), while experts scored 39.00 points (32;47). The GRS overall scores significantly discriminated between student-guided teams and expert teams of emergency physicians (p = 0.005). Interrater reliability for the GRS was high with a Kendall's coefficient of concordance W ranging from 0.64 to 0.90 in 9 of 10 items and 0.88 in the overall score.

CONCLUSION

The GRS represents a promising novel tool to objectively assess technical skills in simulation training with high construct validity and interrater reliability in this pilot study.

The desktop, or the top of the desk? The relative usefulness of household features for personal health information management

Sixty percent of the US population manages at least one chronic illness. For these patients, personal health information management (PHIM) is an integral part of daily life, and largely occurs within the home. However, the way in which the home supports PHIM has not been systematically investigated. The present study examined how members of the diabetic population use features of the home environment to support PHIM. Participants (N = 60) explored a simulated home environment, the VR CAVE, and identified the most useful features for performing three examples of PHIM tasks. The computer was perceived as the most useful feature for PHIM. However, perceived usefulness of features varied based on the PHIM task performed and the rooms in which features appeared. We conclude that a detailed study of the affordances of features is necessary to ease the burden of managing chronic illness, particularly diabetes mellitus, in the sociotechnical system of the home.

Registered nurses psychophysiological stress and confidence during high-fidelity emergency simulation: Effects on performance

INTRODUCTION

Simulation has been used extensively to train students and health professionals in the assessment and early intervention of patients with acutely deteriorating conditions. These simulations evoke psychophysiological stress in learners which may affect performance. We examined the relationship between stress variables, confidence, and performance during repeated scenarios in clinically-based emergency simulations.

METHODS

Twenty-six registered nurses completed three simulation scenarios focussing on life-threatening clinical events in a single group pre-test/post-test study design. Trait anxiety was measured at baseline. Visual analogue ratings of anxiety and stress were measured before ('pre'), recalled 'during', and immediately following ('post') each simulation scenario, with a self-rating of confidence completed after each simulation scenario. Heart rate was measured continuously throughout the simulation program. Participants self-rated their clinical performance prior to and following the simulation program ('pre' and 'post').

RESULTS

Participants' trait anxiety was not elevated at baseline (mean: 39.6, SD 6.1). Across the three simulation scenarios, anxiety and stress was elevated 'during' simulation compared to 'pre' and 'post' time points. However, the magnitude of elevation of stress and anxiety during all time points ('pre', 'during' and 'post' simulation) decreased significantly (p < 0.05) with progressive simulations. Heart rate increased significantly during all simulations compared to 'pre'-levels but returned to similar levels following the simulation. The amount of increase in heart rate over progressive simulations was attenuated during simulation 3 compared with 1 and 2 (Sim 1: 103.6 bpm (SD 22.1), Sim 2: 101.9 bpm (SD 18.9), and Sim 3: 99.5 bpm (SD 23.4)). Confidence increased across the three simulations (p < 0.001), with most of the increase observed after the first two simulations. Performance scores increased by 19.0% 'pre-post' simulation program (p < 0.001) and were not confounded by previous ALS or simulation experience.

DISCUSSION

We observed temporal-dependent changes in psychophysiological stress variables across the simulation scenarios, with decreased magnitudes of elevations of psychological (self-reported anxiety and stress) and physiological (heart rate) stress variables during successive simulation scenarios. This study has shown that simulation increased stress, especially before and during scenarios; however, the learning effect decreased the magnitude of the stress response with repeated simulation scenarios. Simulation educators need to create simulations that change stress in a purposeful manner to enhance learning.

An Analysis of the Efficacy of Different Teaching Modalities in Imparting Adult Cardiopulmonary Resuscitation Skills among First-year Medical Students: A Pilot Study

Introduction

Our current medical curriculum devotes a large percentage of time to knowledge acquisition by means of didactic lectures. Psychomotor skill acquisition takes a back seat. Certain lifesaving skills like basic life support skill training have not even made an appearance in the current curriculum. Equal time distribution to cognitive and psychomotor skills should be allotted for MBBS trainees, which is a very practical subject. Simulation can prove to be a valuable tool in imparting skill training. The present study aims to evaluate the efficacy of different teaching modalities in imparting lifesaving skills among first-year MBBS students.

Materials and methods

This cross-sectional study was conducted among 33 first-year students who consented to participate. Approval was obtained from the institutional ethics committee. The students were divided into three groups, each undergoing either didactic lecture or animation-based videos or simulation studies. Pretest, posttest, and skills tests were administered to them. One-way analysis of variance (ANOVA) and paired t test were the statistical tests employed using SPSS version 21.

Results

The pretest and posttest scores were comparable in the three groups while the improvement in the posttest scores in all the three groups was significant. The skills test was significantly better in the group undergoing simulation training compared to the other groups.

Conclusion

Didactic, animation, and simulation are all good methods in imparting cognitive knowledge, but simulation is the method of choice in imparting psychomotor skills.

Clinical significance

An overhauling of the medical curriculum to include more skills training to the budding doctors using simulation-based techniques is recommended.

How to cite this article

Suseel A, Panchu P, Abraham SV, Varghese S, George T, Joy L. An Analysis of the Efficacy of Different Teaching Modalities in Imparting Adult Cardiopulmonary Resuscitation Skills among First-year Medical Students: A Pilot Study. IJCCM 2019;23(11): 509–512.

Home is where the head is: a distributed cognition account of personal health information management in the home among those with chronic illness

Managing chronic illness requires personal health information management (PHIM) to be performed by lay individuals. Paramount to understanding the PHIM process is understanding the sociotechnical system in which it frequently occurs: the home environment. We combined distributed cognition theory and the patient work system model to investigate how characteristics of the home interact with the cognitive work of PHIM. We used a 3D virtual reality CAVE that enabled participants who had been diagnosed with diabetes (N = 20) to describe how they would perform PHIM in the home context. We found that PHIM is distinctly cognitive work, and rarely performed 'in the head'. Rather, features of the physical environment, tasks, people, and tools and technologies present, continuously shape and are shaped by the PHIM process. We suggest that approaches in which the individual (sans context) is considered the relevant unit of analysis overlook the pivotal role of the environment in shaping PHIM. Practitioner Summary: We examined how Personal Health Information Management (PHIM) is performed in the homes of diabetic patients. We found that approaches to studying cognition that focus on the individual, to the exclusion of their context, overlook the pivotal role of environmental, social, and technological features in shaping PHIM.

Comparison between Simulation-based Training and Lecture-based Education in Teaching Situation Awareness. A Randomized Controlled Study

RATIONALE

Situation awareness has been defined as the perception of the elements in the environment within volumes of time and space, the comprehension of their meaning, and the projection of their status in the near future. Intensivists often make time-sensitive critical decisions, and loss of situation awareness can lead to errors. It has been shown that simulation-based training is superior to lecture-based training for some critical scenarios. Because the methods of training to improve situation awareness have not been well studied in the medical field, we compared the impact of simulation vs. lecture training using the Situation Awareness Global Assessment Technique (SAGAT) score.

OBJECTIVES

To identify an effective method for teaching situation awareness.

METHODS

We randomly assigned 17 critical care fellows to simulation vs. lecture training. Training consisted of eight cases on airway management, including topics such as elevated intracranial pressure, difficult airway, arrhythmia, and shock. During the testing scenario, at random times between 4 and 6 minutes into the simulation, the scenario was frozen, and the screens were blanked. Respondents then completed the 28 questions on the SAGAT scale. Sample items were categorized as Perception, Projection, and Comprehension of the situation. Results were analyzed using SPSS Version 21.

RESULTS

Eight fellows from the simulation group and nine from the lecture group underwent simulation testing. Sixty-four SAGAT scores were recorded for the simulation group and 48 scores were recorded for the lecture group. The mean simulation vs. lecture group SAGAT score was 64.3 ± 10.1 (SD) vs. 59.7 ± 10.8 (SD) (P = 0.02). There was also a difference in the median Perception ability between the simulation vs. lecture groups (61.1 vs. 55.5, P = 0.01). There was no difference in the median Projection and Comprehension scores between the two groups (50.0 vs. 50.0, P = 0.92, and 83.3 vs. 83.3, P = 0.27).

CONCLUSIONS

We found a significant, albeit modest, difference between simulation training and lecture training on the total SAGAT score of situation awareness mainly because of the improvement in perception ability. Simulation may be a superior method of teaching situation awareness.

Virtual reality-based assessment of basic laparoscopic skills using the Leap Motion controller

BACKGROUND

The majority of the current surgical simulators employ specialized sensory equipment for instrument tracking. The Leap Motion controller is a new device able to track linear objects with sub-millimeter accuracy. The aim of this study was to investigate the potential of a virtual reality (VR) simulator for assessment of basic laparoscopic skills, based on the low-cost Leap Motion controller.

METHODS

A simple interface was constructed to simulate the insertion point of the instruments into the abdominal cavity. The controller provided information about the position and orientation of the instruments. Custom tools were constructed to simulate the laparoscopic setup. Three basic VR tasks were developed: camera navigation (CN), instrument navigation (IN), and bimanual operation (BO). The experiments were carried out in two simulation centers: MPLSC (Athens, Greece) and CRESENT (Riyadh, Kingdom of Saudi Arabia). Two groups of surgeons (28 experts and 21 novices) participated in the study by performing the VR tasks. Skills assessment metrics included time, pathlength, and two task-specific errors. The face validity of the training scenarios was also investigated via a questionnaire completed by the participants.

RESULTS

Expert surgeons significantly outperformed novices in all assessment metrics for IN and BO (p < 0.05). For CN, a significant difference was found in one error metric (p < 0.05). The greatest difference between the performances of the two groups occurred for BO. Qualitative analysis of the instrument trajectory revealed that experts performed more delicate movements compared to novices. Subjects' ratings on the feedback questionnaire highlighted the training value of the system.

CONCLUSIONS

This study provides evidence regarding the potential use of the Leap Motion controller for assessment of basic laparoscopic skills. The proposed system allowed the evaluation of dexterity of the hand movements. Future work will involve comparison studies with validated simulators and development of advanced training scenarios on current Leap Motion controller.

Missing mediated interruptions in manual assembly: Critical aspects of breakpoint selection

The factory of the future aims to make manufacturing more effective and easily customisable, using advanced sensors and communications to support information management. In this paper, we examine how breakpoint selection during interruption management can fail, even when using recommendations for interruption management from existing research. We present an experiment based on prior work where mediated interruptions (i.e. smart interruptions that should interrupt at opportune moments) were missed by participants when sent at one of two pre-defined breakpoints. These breakpoints were selected based on existing research to minimise the cost of interruption, which can involve longer times to complete tasks as well as making errors on tasks. Missing mediated interruptions in this way was unexpected, and the prior study was not configured to measure this effect, which has led to the experiment detailed here. We strive to explore whether there is a risk of missing notifications when mediated interruptions are used, and how this is affected by breakpoint selection. This was investigated through an experiment that uses tasks and environments that simulate a manufacturing assembly facility. The results indicate that the effect exists, i.e. that participants miss significantly more notifications when interrupted at fine breakpoints than when interrupted at coarse breakpoints. An embodied cognition perspective was used for analysis of the tasks to understand the cause of the effect. This analysis shows that an overlap between "action" and "anticipation of action" can account for why participants miss notifications at fine breakpoints. Based on these findings, recommendations were developed for designing interruption systems that minimise the costs (errors and time) imposed by interruptions during assembly tasks in manufacturing.

Coordinating the interruption of assembly workers in manufacturing

This paper examines how interruptions from information and communications technology systems affect errors and the time to complete tasks for assembly workers. Interruptions have previously been examined in laboratory experiments and office environments, but not much work has been performed in other authentic environments. This paper contains the results of an experiment that was performed in a simulated manufacturing assembly environment, which tested the effects of interruptions on a manual assembly task. The experiment used existing interruption coordination methods as a basis, and the results showed a difference in the effect of interruptions and interruption coordination between cognitively complex laboratory tasks and manual assembly tasks in an authentic environment. Most notably, the negative effects of interruptions delivered without consideration were smaller in this experiment. Based on these findings, recommendations were developed for designing interruption systems for minimizing the costs (errors and time) imposed by interruptions during assembly tasks in manufacturing.

Achieving Interface and Environment Fidelity in the Virtual Basic Laparoscopic Surgical Trainer

Virtual reality trainers are educational tools with great potential for laparoscopic surgery. They can provide basic skills training in a controlled environment and free of risks for patients. They can also offer objective performance assessment without the need for proctors. However, designing effective user interfaces that allow the acquisition of the appropriate technical skills on these systems remains a challenge. This paper aims to examine a process for achieving interface and environment fidelity during the development of the Virtual Basic Laparoscopic Surgical Trainer (VBLaST). Two iterations of the design process were conducted and evaluated. For that purpose, a total of 42 subjects participated in two experimental studies in which two versions of the VBLaST were compared to the accepted standard in the surgical community for training and assessing basic laparoscopic skills in North America, the FLS box-trainer. Participants performed 10 trials of the peg transfer task on each trainer. The assessment of task performance was based on the validated FLS scoring method. Moreover, a subjective evaluation questionnaire was used to assess the fidelity aspects of the VBLaST relative to the FLS trainer. Finally, a focus group session with expert surgeons was conducted as a comparative situated evaluation after the first design iteration. This session aimed to assess the fidelity aspects of the early VBLaST prototype as compared to the FLS trainer. The results indicate that user performance on the earlier version of the VBLaST resulting from the first design iteration was significantly lower than the performance on the standard FLS box-trainer. The comparative situated evaluation with domain experts permitted us to identify some issues related to the visual, haptic and interface fidelity on this early prototype. Results of the second experiment indicate that the performance on the second generation VBLaST was significantly improved as compared to the first generation and not significantly different from that of the standard FLS box-trainer. Furthermore, the subjects rated the fidelity features of the modified VBLaST version higher than the early version. These findings demonstrate the value of the comparative situated evaluation sessions entailing hands on reflection by domain experts to achieve the environment and interface fidelity and training objectives when designing a virtual reality laparoscopic trainer. This suggests that this method could be used successfully in the future to enhance the value of VR systems as an alternative to physical trainers for laparoscopic surgery skills. Some recommendations on how to use this method to achieve the environment and interface fidelity of a VR laparoscopic surgical trainer are identified.