Validity of Cognitive Load Measures in Simulation-Based Training: A Systematic Review

Estimating Cognitive Workload Using Task-Related Pupillary Responses in Simulated Drilling in Cochlear Implantation

OBJECTIVES

Training of temporal bone drilling requires more than mastering technical skills with the drill. Skills such as visual imagery, bimanual dexterity, and stress management need to be mastered along with precise knowledge of anatomy. In otorhinolaryngology, these psychomotor skills underlie performance in the drilling of the temporal bone for access to the inner ear in cochlear implant surgery. However, little is known about how psychomotor skills and workload management impact the practitioners' continuous and overall performance.

METHODS

To understand how the practitioner's workload and performance unfolds over time, we examine task-evoked pupillary responses (TEPR) of 22 medical students who performed transmastoid-posterior tympanotomy (TMPT) and removal of the bony overhang of the round window niche in a 3D-printed model of the temporal bone. We investigate how students' TEPR metrics (Average Pupil Size [APS], Index of Pupil Activity [IPA], and Low/High Index of Pupillary Activity [LHIPA]) and time spent in drilling phases correspond to the performance in key drilling phases.

RESULTS

All TEPR measures revealed significant differences between key drilling phases that corresponded to the anticipated workload. Enlarging the facial recess lasted significantly longer than other phases. IPA captured significant increase of workload in thinning of the posterior canal wall, while APS revealed increased workload during the drilling of the bony overhang.

CONCLUSION

Our findings contribute to the contemporary competency-based medical residency programs where objective and continuous monitoring of participants' progress allows to track progress in expertise acquisition. Laryngoscope, 2024.

Understanding The Role of Cognitive Load In Paramedical Contexts: A Systematic Review

Objectives: Cognitive load refers to the working memory resources required during a task. When the load is too high or too low this has implications for an individual's task performance. In the context of paramedicine and emergency medical services (EMS) broadly, high cognitive load could potentially put patient and personnel safety at risk. This systematic review aimed to determine the current understanding of the role of cognitive load in paramedical contexts.Methods: To do this, five databases were searched (Elsevier Embase, ProQuest Psychology, CINAHL, Ovid Medline, and Ovid PsychINFO) using synonyms of cognitive load and paramedical contexts. Included articles were full text, peer reviewed empirical research, with a focus on cognitive load and EMS work. Two reviewers screened titles, abstracts, and full text using a traffic light system against the inclusion and exclusion criteria. The quality of evidence was assessed using the GRADE framework. This study was registered on PROSPERO (CRD42022384246). No funding was received for this research.Results: The searches identified 73 unique articles and after title/abstract and full text screening, 25 articles were included in the final review. Synthesis of the research revealed 10 categories of findings in the area. These are clinical performance, cognitive processes, emotional responses, physical expenditure, physiological responses, equipment and ergonomics, expertise and experience, multiple loads, cognitive load measures, and task complexity.Conclusions: From these findings it was determined that there is agreement in terms of what factors influence cognitive load in paramedical contexts, such as cognitive processes, task complexity, physical expenditure, level of experience, multiple types of loads, and the use of equipment. Cognitive load influences clinical task performance and has a bi-directional relationship with emotion. However, the literature is mixed regarding physiological responses to cognitive load, and how they are best measured. These findings highlight potential intervention points where cognitive load can be managed or reduced to improve working conditions for EMS clinicians and safety for their patients.

Simulated skill complexity and perceived cognitive load during preclinical dental training

INTRODUCTION

Cognitive Load theory (CLT) focuses on the information processing aspect of learning and how the working memory handles the mental effort associated with new task. The aim of this study is to investigate the association between the perceived cognitive load and performance amongst dental students during preclinical simulation training at various levels of procedural task complexity. Additionally, some cognitive load-modifying factors were examined.

MATERIALS AND METHODS

This cross-sectional study evaluated the perceived cognitive load amongst second-year dental students (n = 34), using the validated National Aeronautics and Space Administration's Task Load Index (NASA TLX index) after training on four dental tasks at two levels of complexity, in addition to structured online anonymous questionnaire about demographics, feedback and performance. The NASA TLX raw scores and the weighted global score were calculated for each exercise. Descriptive statistics and Pearson's correlations between performance and the corresponding NASA TLX-weighted score were calculated. Mean differences in the perceived cognitive load across the exercise levels were assessed using RM-ANOVA with Bonferroni corrections at p < .05.

RESULTS

Reduced performance was significantly associated with higher cognitive load particularly in high complexity dental task (class II-mirror vision). Simulated exercise complexity significantly influenced the students' perceived mental demand, physical demand and temporal demand; all were significantly higher for class II- mirror vision task than for direct vision tasks. The majority of participants (82.1%) preferred detailed feedback from instructors, and more than half of the participants (60.7%) preferred continuous feedback throughout the training session.

CONCLUSION

Complex dental tasks are associated with higher cognitive load in novice dental students during preclinical training. The NASA TLX index is a useful instrument to explore the level of perceived cognitive load associated with performance of simulated complex dental skills. Cognitive load theory is relevant to simulation-based dental education to improve the preclinical instructional efficiency and to enhance students learning.

The Evidence Basis for Learning Theory and Technology in Surgical Skills Training

Orthopaedic trainees face a complex and challenging training environment that is currently becoming more competency driven. Associated with these changes are an increasing introduction and use of a variety of technologically driven surgical training augments. Although these new learning resources can positively transform the educational environment, they must be used appropriately by both learners and educators. To aid in this, we review learning theories because they apply to surgical skills training and highlight recent surgical training evidence that demonstrates how technology use can be optimized to promote surgical learning, with an emphasis on procedural learning theory and cognitive load theory. Specifically, we review the evidence demonstrating the importance of targeting technology to a learner's experience level and methods to optimize cognitive load by managing intrinsic load, minimizing extraneous load, and maximizing germane load.

Mastoidectomy Training: Is Anatomical Variation Needed? A Randomized, Controlled Trial on Performance and Skills Transfer From Virtual Reality to a Three-Dimensional Printed Model

OBJECTIVE

Virtual reality (VR) simulation-based training effectively improves novices' mastoidectomy skills. Unfortunately, learning plateaus at an insufficient level and knowledge on optimizing mastoidectomy training to overcome this plateau is needed. In this study, we aim to investigate how training on anatomically different temporal bone cases affects learning, including the effect on retention and transfer of skills.

STUDY DESIGN

Randomized controlled trial of an educational intervention.

SETTING

The Simulation Center at Copenhagen Academy for Medical Education and Simulation.

PARTICIPANTS

Twenty-four medical students from the University of Copenhagen.

INTERVENTION

Participants were randomized to practice mastoidectomy on either 12 anatomically varying (intervention group) or 12 identical (control group) cases in a VR simulator. At the end of training and again ~ 3 weeks after training (retention), learners were tested on a new VR patient case and a three-dimensional printed model.

MAIN OUTCOME MEASURE

Mastoidectomy performance evaluated by blinded expert raters using a 26-item modified Welling Scale.

RESULTS

The intervention and control groups' performance results were comparable at the end of training. Likewise, retention and transfer performances were similar between groups. The overall mean score at the end of training corresponded to approximately 70% of the possible maximum score.

CONCLUSIONS

Simulation-based training using anatomical variation was equivalent to training on a single case with respect to acquisition, retention, and transfer of mastoidectomy skills. This suggests that efforts to expose novices to variation during initial training are unnecessary as this variation has limited effect, and-conversely-that educators can expose novices to naturally different anatomical variations without worry of hindered learning.

Objective Measurement of Learners' Cognitive Load During Simulation-Based Trauma Team Training: A Pilot Study

INTRODUCTION

Literature has shown cognitive overload which can negatively impact learning and clinical performance in surgery. We investigated learners' cognitive load during simulation-based trauma team training using an objective digital biomarker.

METHODS

A cross-sectional study was carried out in a simulation center where a 3-h simulation-based interprofessional trauma team training program was conducted. A session included three scenarios each followed by a debriefing session. One scenario involved multiple patients. Learners wore a heart rate sensor that detects interbeat intervals in real-time. Low-frequency/high-frequency (LF/HF) ratio was used as a validated proxy for cognitive load. Learners' LF/HF ratio was tracked through different phases of simulation.

RESULTS

Ten subjects participated in 12 simulations. LF/HF ratios during scenario versus debriefing were compared for each simulation. These were 3.75 versus 2.40, P < 0.001 for scenario 1; 4.18 versus 2.77, P < 0.001 for scenario 2; and 4.79 versus 2.68, P < 0.001 for scenario 3. Compared to single-patient scenarios, multiple-patient scenarios posed a higher cognitive load, with LF/HF ratios of 3.88 and 4.79, P < 0.001, respectively.

CONCLUSIONS

LF/HF ratio, a proxy for cognitive load, was increased during all three scenarios compared to debriefings and reached the highest levels in a multiple-patient scenario. Using heart rate variability as an objective marker of cognitive load is feasible and this metric is able to detect cognitive load fluctuations during different simulation phases and varying scenario difficulties.

State of the evidence on simulation-based electronic health records training: A scoping review

This study synthesized the available evidence of simulation-based electronic health records (EHRs) training in educational and clinical environments for healthcare providers in the literature. The Arksey and O'Malley methodological framework was employed. A systematic search was carried out in relevant databases from inception to January 2020, identifying 24 studies for inclusion. Three themes emerged: (a) role of simulation-based EHR training in evaluating improvement interventions, (b) debriefing and feedback methods used, and (c) challenges of evaluating simulation-based EHR training. The majority of the studies aimed to emphasize the practical skills of individual medical trainees and employed post-simulation feedback as the feedback method. Future research should focus on (a) using simulation-based EHR training to achieve specific learning goals, (b) investigating aspects of clinical performance that are susceptible to skill decay, and (c) examining the influence of simulation-based EHR training on team dynamics.

Cognitive load in academic clinical simulation activities. Cross-sectional study

Introduction: Cognitive load determines working memory ability to store and retain information in long-term memory, thus conditioning learning. Objective: To compare cognitive loads among different simulation activities, including anesthesia and surgery simulation workshops in medical students. Methods: Cross-sectional analytical observational study. Two cognitive load measurement scales (Paas and NASA-TLX) were given to the students after each simulation workshop. Comparisons were made based on the scores derived from the scales. Results: Relevant differences were found in terms of the mental effort assessed by means of the Paas scale, as relates to student rotation order in the airway management workshop, with a greater effort being found in the group that rotated initially in surgery (6.19 vs. 5.53; p = 0.029). The workshop with the highest associated rate of frustration was the airway management workshop. Higher scores were obtained for this workshop in all the items of the NASA-TLX scale, reflecting a higher cognitive load when compared to the others. Conclusion: It was not possible to determine whether higher scores in some of the activities were associated with the inherent difficulty of airway management or the specific workshop design. Consequently, further studies are required to distinguish between those components in order to improve the way learning activities are designed.

Methods and Measures for Mental Stress Assessment in Surgery: A Systematic Review of 20 Years of Literature

Real-time mental stress monitoring from surgeons and surgical staff in operating rooms may reduce surgical injuries, improve performance and quality of medical care, and accelerate implementation of stress-management strategies. Motivated by the increase in usage of objective and subjective metrics for cognitive monitoring and by the gap in reviews of experimental design setups and data analytics, a systematic review of 71 studies on mental stress and workload measurement in surgical settings, published in 2001-2020, is presented. Almost 61% of selected papers used both objective and subjective measures, followed by 25% that only administered subjective tools - mostly consisting of validated instruments and customized surveys. An overall increase in the total number of publications on intraoperative stress assessment was observed from mid-2010 s along with a momentum in the use of both subjective and real-time objective measures. Cardiac activity, including heart-rate variability metrics, stress hormones, and eye-tracking metrics were the most frequently and electroencephalography (EEG) was the least frequently used objective measures. Around 40% of selected papers collected at least two objective measures, 41% used wearable devices, 23% performed synchronization and annotation, and 76% conducted baseline or multi-point data acquisition. Furthermore, 93% used a variety of statistical techniques, 14% applied regression models, and only one study released a public, anonymized dataset. This review of data modalities, experimental setups, and analysis techniques for intraoperative stress monitoring highlights the initiatives of surgical data science and motivates research on computational techniques for mental and surgical skills assessment and cognition-guided surgery.

Effects of a virtual reality serious game training program on the cognitive function of people diagnosed with schizophrenia: A randomized controlled trial

BACKGROUND

Cognitive impairment persists through the course of schizophrenia and affects patients' activities of daily living.

AIM

This study aims to investigate the effects of a virtual reality (VR) serious game training program on the cognitive function of people diagnosed with schizophrenia.

MATERIALS AND METHODS

Sixty-four eligible people diagnosed with schizophrenia were recruited and randomly assigned to the intervention group (n = 31) and the control group (n = 33). The control group received standard psychiatric care. The intervention group was trained with an additional VR game twice a day for at least 10 days during hospitalization. Cognitive function was measured at enrollment and before discharge using the Brief Cognitive Assessment Tool for Schizophrenia.

RESULTS

Compared with those of the control group, the results of the working memory (t = 3.463, Cohen's d = 0.87, p = 0.001) and executive function (TMTA: Z = -2.272, Cohen's d = 0.59, p = 0.023; TMTB:Z = -2.365, Cohen's d = 0.62, p = 0.018) of the intervention group after intervention were significantly better. However, there was no significant difference in the results of social cognition (Z = -1.394, Cohen's d = 0.35, p = 0.163) between the two groups.

CONCLUSION

Intensive active virtual reality serious game training in addition to standard psychiatric care can significantly improve working memory and executive function in people diagnosed with schizophrenia.

IMPLICATIONS FOR PRACTICE

When helping improve the cognitive function of people diagnosed with schizophrenia, mental health professionals should identify cognitive domains to be enhanced and develop corresponding serious game training strategies.

Automated summative feedback improves performance and retention in simulation training of mastoidectomy: a randomised controlled trial

OBJECTIVE

This study aimed to investigate the effects of automated metrics-based summative feedback on performance, retention and cognitive load in distributed virtual reality simulation training of mastoidectomy.

METHOD

Twenty-four medical students were randomised in two groups and performed 15 mastoidectomies on a distributed virtual reality simulator as practice. The intervention group received additional summative metrics-based feedback; the control group followed standard instructions. Two to three months after training, participants performed a retention test without learning supports.

RESULTS

The intervention group had a better final-product score (mean difference = 1.0 points; p = 0.001) and metrics-based score (mean difference = 12.7; p < 0.001). At retention, the metrics-based score for the intervention group remained superior (mean difference = 6.9 per cent; p = 0.02). Also at the retention, cognitive load was higher in the intervention group (mean difference = 10.0 per cent; p < 0.001).

CONCLUSION

Summative metrics-based feedback improved performance and lead to a safer and faster performance compared with standard instructions and seems a valuable educational tool in the early acquisition of temporal bone skills.

Effects of Neuro-Cognitive Load on Learning Transfer Using a Virtual Reality-Based Driving System

Understanding the ways different people perceive and apply acquired knowledge, especially when driving, is an important area of study. This study introduced a novel virtual reality (VR)-based driving system to determine the effects of neuro-cognitive load on learning transfer. In the experiment, easy and difficult routes were introduced to the participants, and the VR system is capable of recording eye-gaze, pupil dilation, heart rate, as well as driving performance data. So, the main purpose here is to apply multimodal data fusion, several machine learning algorithms, and strategic analytic methods to measure neurocognitive load for user classification. A total of ninety-eight (98) university students participated in the experiment, in which forty-nine (49) were male participants and forty-nine (49) were female participants. The results showed that data fusion methods achieved higher accuracy compared to other classification methods. These findings highlight the importance of physiological monitoring to measure mental workload during the process of learning transfer.

Impact of nonclinical factors on intensive care unit admission decisions: a vignette-based randomized trial (V-TRIAGE)

OBJECTIVE

To assess the impact of intensive care unit bed availability, distractors and choice framing on intensive care unit admission decisions.

METHODS

This study was a randomized factorial trial using patient-based vignettes. The vignettes were deemed archetypical for intensive care unit admission or refusal, as judged by a group of experts. Intensive care unit physicians were randomized to 1) an increased distraction (intervention) or a control group, 2) an intensive care unit bed scarcity or nonscarcity (availability) setting, and 3) a multiple-choice or omission (status quo) vignette scenario. The primary outcome was the proportion of appropriate intensive care unit allocations, defined as concordance with the allocation decision made by the group of experts.

RESULTS

We analyzed 125 physicians. Overall, distractors had no impact on the outcome; however, there was a differential drop-out rate, with fewer physicians in the intervention arm completing the questionnaire. Intensive care unit bed availability was associated with an inappropriate allocation of vignettes deemed inappropriate for intensive care unit admission (OR = 2.47; 95%CI 1.19 - 5.11) but not of vignettes appropriate for intensive care unit admission. There was a significant interaction with the presence of distractors (p = 0.007), with intensive care unit bed availability being associated with increased intensive care unit admission of vignettes inappropriate for intensive care unit admission in the distractor (intervention) arm (OR = 9.82; 95%CI 2.68 - 25.93) but not in the control group (OR = 1.02; 95%CI 0.38 - 2.72). Multiple choices were associated with increased inappropriate allocation in comparison to the omission group (OR = 5.18; 95%CI 1.37 - 19.61).

CONCLUSION

Intensive care unit bed availability and cognitive biases were associated with inappropriate intensive care unit allocation decisions. These findings may have implications for intensive care unit admission policies.

The future of simulation-based medical education: Adaptive simulation utilizing a deep multitask neural network

BACKGROUND

In resuscitation medicine, effectively managing cognitive load in high-stakes environments has important implications for education and expertise development. There exists the potential to tailor educational experiences to an individual's cognitive processes via real-time physiologic measurement of cognitive load in simulation environments.

OBJECTIVE

The goal of this research was to test a novel simulation platform that utilized artificial intelligence to deliver a medical simulation that was adaptable to a participant's measured cognitive load.

METHODS

The research was conducted in 2019. Two board-certified emergency physicians and two medical students participated in a 10-minute pilot trial of a novel simulation platform. The system utilized artificial intelligence algorithms to measure cognitive load in real time via electrocardiography and galvanic skin response. In turn, modulation of simulation difficulty, determined by a participant's cognitive load, was facilitated through symptom severity changes of an augmented reality (AR) patient. A postsimulation survey assessed the participants' experience.

RESULTS

Participants completed a simulation that successfully measured cognitive load in real time through physiological signals. The simulation difficulty was adapted to the participant's cognitive load, which was reflected in changes in the AR patient's symptoms. Participants found the novel adaptive simulation platform to be valuable in supporting their learning.

CONCLUSION

Our research team created a simulation platform that adapts to a participant's cognitive load in real-time. The ability to customize a medical simulation to a participant's cognitive state has potential implications for the development of expertise in resuscitation medicine.

Effect of simulation on cognitive load in health care professionals and students: protocol for a systematic review and meta-analysis

OBJECTIVE

The objective of this review is to assess the effect of simulation activities and their design features on cognitive load in health care professionals and students.

INTRODUCTION

Simulation activities are now widely implemented in health care professionals' education. However, the mechanisms by which simulations and their design features lead to health care professionals' and students' learning remains unclear. Still, because of their high interactivity and complexity, simulation activities have the potential to impact the cognitive load of learners. Synthesizing evidence regarding this phenomenon could help simulation educators identify the design features that affect learners' cognitive load, and explain why some simulation activities are more effective than others.

INCLUSION CRITERIA

This review will consider experimental and quasi-experimental studies in which the effect of a simulation activity on cognitive load in health care professionals or students from any discipline or level of practice is evaluated. All academic and health settings will be included.

METHODS

Following the guidelines of the JBI methods for systematic reviews of effectiveness, CINAHL, Embase, ERIC, MEDLINE, PsycINFO, and Web of Science will be searched for studies published in English or French, without a date limit. Retrieved studies will be independently screened for inclusion, then critically appraised for methodological quality by two reviewers using standardized JBI tools. Data extraction will be done independently using adapted tools from JBI. Where possible, data will be pooled using meta-analytical methods.

SYSTEMATIC REVIEW REGISTRATION NUMBER

PROSPERO CRD42020187723.

Low versus high level of physical resemblance in simulation for the acquisition of basic surgical skill: a meta-analysis

Background

Many studies explored the use of simulation in basic surgical education, with a variety of devices, contexts and outcomes, with sometimes contradictory results.

Objectives

The objectives of this meta-analysis were to focus the effect that the level of physical resemblance in a simulation has on the development of basic surgical skill in undergraduate medical students and to provide a foundation for the design and implementation of a simulation, with respect to its effectiveness and alignment with the learning outcomes.

Study selection

We searched PubMed and Scopus database for comparative randomised studies between simulations with a different level of resemblance. The result was synthesised as the standardised mean difference, under a random effect model.

Findings

We selected 12 out of 2091 retrieved studies, reporting on 373 undergraduate students (mean of subjects 15.54±6.89). The outcomes were the performance of simple skills and the time to complete a task. Two studies reported a scoring system; seven studies reported time for a task; and three studies reported both. The total number of measures included in the meta-analysis was 456 for score and 504 for time. The pooled effect size did not show any significant advantage in a simulation of a high level of physical resemblance over a lower level, both for the scoring system (−0.19, 95% CI −0.44 to 0.06) and for time (−0.14, 95% CI −0.54 to 0.27).

Conclusion

Simulations with a low level of physical resemblance showed the same effect as the simulation using a higher level of resemblance on the development of basic surgical skills in undergraduate students.

Examiners' decision-making processes in observation-based clinical examinations

BACKGROUND

Objective structured clinical examinations (OSCEs) are commonly used to assess the clinical skills of health professional students. Examiner judgement is one acknowledged source of variation in candidate marks. This paper reports an exploration of examiner decision making to better characterise the cognitive processes and workload associated with making judgements of clinical performance in exit-level OSCEs.

METHODS

Fifty-five examiners for exit-level OSCEs at five Australian medical schools completed a NASA Task Load Index (TLX) measure of cognitive load and participated in focus group interviews immediately after the OSCE session. Discussions focused on how decisions were made for borderline and clear pass candidates. Interviews were transcribed, coded and thematically analysed. NASA TLX results were quantitatively analysed.

RESULTS

Examiners self-reported higher cognitive workload levels when assessing a borderline candidate in comparison with a clear pass candidate. Further analysis revealed five major themes considered by examiners when marking candidate performance in an OSCE: (a) use of marking criteria as a source of reassurance; (b) difficulty adhering to the marking sheet under certain conditions; (c) demeanour of candidates; (d) patient safety, and (e) calibration using a mental construct of the 'mythical [prototypical] intern'. Examiners demonstrated particularly higher mental demand when assessing borderline compared to clear pass candidates.

CONCLUSIONS

Examiners demonstrate that judging candidate performance is a complex, cognitively difficult task, particularly when performance is of borderline or lower standard. At programme exit level, examiners intuitively want to rate candidates against a construct of a prototypical graduate when marking criteria appear not to describe both what and how a passing candidate should demonstrate when completing clinical tasks. This construct should be shared, agreed upon and aligned with marking criteria to best guide examiner training and calibration. Achieving this integration may improve the accuracy and consistency of examiner judgements and reduce cognitive workload.

Workload of learners during simulated paediatric cardiopulmonary resuscitation

Introduction

Learner workload during simulated team-based resuscitations is not well understood. In this descriptive study, we measured the workload of learners in different team roles during simulated paediatric cardiopulmonary resuscitation.

Methods

Paediatric emergency nurses and paediatric and emergency medicine residents formed teams of four to eight and randomised into roles to participate in simulation-based, paediatric resuscitation. Participant workload was measured using the NASA Task Load Index, which provides an average workload score (from 0 to 100) across six subscores: mental demand, physical demand, temporal demand, performance, frustration and mental effort. Workload is considered low if less than 40, moderate if between 40 and 60 and high if greater than 60.

Results

There were 210 participants representing 40 simulation teams. 138 residents (66%) and 72 nurses (34%) participated. Team lead reported the highest workload at 65.2±10.0 (p=0.001), while the airway reported the lowest at 53.9±10.8 (p=0.001); team lead had higher scores for all subscores except physical demand. Team lead reported the highest mental demand (p<0.001), while airway reported the lowest. Cardiopulmonary resuscitation coach and first responder reported the highest physical demands (p<0.001), while team lead and nurse recorder reported the lowest (p<0.001).

Conclusions

Workload for learners in paediatric simulated resuscitation teams was moderate to high and varied significantly based on team role. Composition of workload varied significantly by team role. Measuring learner workload during simulated resuscitations allows improved processes and choreography to optimise workload distribution.

How do attending physicians describe cognitive overload among their workplace learners?

OBJECTIVES

Cognitive load theory (CLT) focuses on the limited bandwidth of working memory. Core to CLT is the concept of cognitive overload, which occurs when working memory demands exceed working memory capacity, and learning and performance suffer. Within health professions education (HPE), workplace learning settings are very complex, placing learners at high risk of cognitive overload. Although continuous monitoring of physiologic parameters can indicate states of high cognitive load, how to practically identify cognitively overloaded learners within everyday workplace settings is not well understood. We sought to characterise how attending physicians described their perceiving of cognitive overload among learners in two different workplace settings: the gastrointestinal endoscopy suite and the intensive care unit.

METHODS

We performed a secondary qualitative analysis of transcripts of interviews with workplace teachers that had been carried out during two previous studies. These studies had addressed different objectives but both were informed by CLT. Each included questions that prompted participants to reflect on how they perceived cognitive overload to manifest among learners in the workplace. To investigate the phenomenon of cognitive overload, we developed a new codebook and performed content analysis.

RESULTS

We analysed 42 interview transcripts (22 endoscopists, 12 hospitalists, eight intensivists). Participants described four behaviours they had witnessed among learners they thought were cognitively overloaded: poor performance on workplace tasks; non-verbal physical manifestations (including posture, eye and body movements and autonomic functions); verbal utterances (words and sounds), and interpersonal interactions with team members. Endoscopists often described individually oriented examples, whereas intensivists and hospitalists tended to frame examples within an interpersonal context.

CONCLUSIONS

We identified four overarching ways in which HPE workplace teachers perceived learners as appearing to be cognitively overloaded. Workplace teachers and learners should be mindful of and watch for these signs, which may signal states of cognitive overload. Earlier recognition of cognitive overload may facilitate timely action to reduce cognitive overload and promote learning.

Using Cognitive Load Theory to Improve Teaching in the Clinical Workplace

INTRODUCTION

Cognitive load theory (CLT) views working memory as the primary bottleneck for learning, as it is limited in both capacity and retention. CLT delineates three types of activities that impose on working memory: intrinsic load, germane load, and extraneous load. These three constructs have practical ramifications for direct teaching, learning environments, and curricular design. CLT could help educators across health professions improve quality of teaching, especially in demanding and unpredictable workplace environments. However, few educational resources exist to familiarize clinical workplace educators with CLT.

METHODS

We developed a 2-hour workshop focused on CLT's core concepts and practical applications, targeted at health professions' workplace educators. It featured large-group, small-group, and individual reflective activities. An end-of-workshop survey was administered, and a follow-up survey was sent to participants 2 months after the workshop.

RESULTS

A total of 134 educators attended the first two offerings of the workshop in two different states. Participants considered CLT as relevant to a variety of workplace teaching settings and activities. Participants' self-assessed familiarity with CLT on a 0-100 scale increased from a mean of 36 (SD = 26) before the workshop to 59 (SD = 17) after the workshop. At follow-up, participants scored an average of 85% on content knowledge questions. Approximately half of respondents to the follow-up survey stated they had made or planned to make specific changes to their workplace teaching leveraging tenets of CLT.

DISCUSSION

The workshop conveyed CLT concepts and primed participants to independently craft CLT-based interventions for their own teaching practices.

Reliability and Validity of Pupillary Response During Dual-Task Balance in Parkinson Disease

OBJECTIVE

To investigate the reliability and validity of pupillary response during dual-task balance conditions in individuals with Parkinson disease (PD).

DESIGN

Cross-sectional study.

SETTING

University of Kansas Medical Center Parkinson's Disease and Movement Disorder Center.

PARTICIPANTS

Participants (N=68) included individuals with PD (n=33) and healthy controls (n=35).

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Pupillary response was the main outcome measure that was measured during the following conditions: single-task balance eyes open, single-task balance eyes occluded, dual-task eyes open, and dual-task eyes occluded. After each condition, the National Aeronautics and Space Administration-Task Load Index (NASA-TLX) was administered to assess self-reported cognitive workload. To examine the test-retest reliability of the pupillary response, the conditions were administered twice for each individual within 2 hours. Intraclass correlation coefficients (ICC) were used to analyze the test-retest reliability of pupillary response in each condition for both groups. Pearson's r correlation was used to assess the convergent validity of pupillary response against the NASA-TLX.

RESULTS

The test-retest reliability was excellent for both groups in almost all conditions (ICC>0.75). There were no correlations between pupillary response and the NASA-TLX. However, increased mental demand (a subitem of the NASA-TLX) significantly correlated with increased pupillary response in individuals with PD (r=0.38; P=.03).

CONCLUSIONS

Pupillary response showed excellent test-retest reliability and validity during dual-task balance for individuals with PD and healthy controls. Overall, these results suggest that pupillary response represents a stable index of cognitive workload during dual-task balance in individuals with PD.

Psychometric properties of the Chinese version of the instrument for measuring different types of cognitive load (MDT‐CL)

Aim

Background

Methods

Results

Conclusions

Implications for nursing management

Ultra-high-fidelity virtual reality mastoidectomy simulation training: a randomized, controlled trial

PURPOSE

Ultra-high-fidelity (UHF) graphics in virtual reality (VR) simulation might improve surgical skill acquisition in temporal bone training. This study aims to compare UHF VR simulation training with conventional, screen-based VR simulation training (cVR) with respect to performance and cognitive load (CL).

METHODS

In a randomized trial with a cross-over design, 24 students completed a total of four mastoidectomies in a VR temporal bone surgical simulator: two performances under UHF conditions using a digital microscope and two performances under conventional conditions using screen-based VR simulation. Performances were assessed by two blinded raters using an established assessment tool. In addition, CL was estimated as the relative change in secondary-task reaction time during simulation when compared with individual baseline measurements. Data were analyzed using linear mixed model analysis for repeated measurements.

RESULTS

The mean final-product performance score was significantly lower in UHF VR simulation compared to cVR simulation [mean difference 1.0 points out of 17 points, 95% CI (0.2-1.7), p = 0.02]. The most important factor for performance during UHF simulation was the ability to achieve stereovision (mean difference = 3.4 points, p < 0.001). Under the UHF VR condition, CL was significantly higher than during cVR (28% vs. 18%, respectively, p < 0.001).

CONCLUSION

UHF graphics in VR simulation training reduced performance and induced a higher CL in novices than conventional, screen-based VR simulation training. Consequently, UHF VR simulation training should be preceded by cVR training and might be better suited for the training of intermediates or experienced surgeons.

Understanding the effects of structured self-assessment in directed, self-regulated simulation-based training of mastoidectomy: A mixed methods study

Objective

Self-directed training represents a challenge in simulation-based training as low cognitive effort can occur when learners overrate their own level of performance. This study aims to explore the mechanisms underlying the positive effects of a structured self-assessment intervention during simulation-based training of mastoidectomy.

Methods

A prospective, educational cohort study of a novice training program consisting of directed, self-regulated learning with distributed practice (5x3 procedures) in a virtual reality temporal bone simulator. The intervention consisted of structured self-assessment after each procedure using a rating form supported by small videos. Semi-structured telephone interviews upon completion of training were conducted with 13 out of 15 participants. Interviews were analysed using directed content analysis and triangulated with quantitative data on secondary task reaction time for cognitive load estimation and participants’ self-assessment scores.

Results

Six major themes were identified in the interviews: goal-directed behaviour, use of learning supports for scaffolding of the training, cognitive engagement, motivation from self-assessment, self-assessment bias, and feedback on self-assessment (validation). Participants seemed to self-regulate their learning by forming individual sub-goals and strategies within the overall goal of the procedure. They scaffolded their learning through the available learning supports. Finally, structured self-assessment was reported to increase the participants’ cognitive engagement, which was further supported by a quantitative increase in cognitive load.

Conclusions

Structured self-assessment in simulation-based surgical training of mastoidectomy seems to promote cognitive engagement and motivation in the learning task and to facilitate self-regulated learning.

The mental workload of conducting research in assessor cognition

None

Optimizing assessors' mental workload in rater-based assessment: a critical narrative review

INTRODUCTION

Rater-based assessment has resulted in high cognitive demands for assessors within the education of health professionals. Rating quality may be influenced by the mental workload required of assessors to complete rating tasks. The objective of this review was to explore interventions or strategies aimed at measuring and reducing mental workload for improvement in assessment outcomes in health professions education.

METHODS

A critical narrative review was conducted for English-language articles using the databases PubMed, EMBASE, and Google Scholar from conception until November 2018. To be included, articles were eligible if they reported results of interventions aimed at measuring or reducing mental workload in rater-based assessment.

RESULTS

A total of six articles were included in the review. All studies were conducted in simulation settings (OSCEs or videotaped interactions). Of the four studies that measured mental workload, none found any reduction in mental workload as demonstrated by objective secondary task performance after interventions of assessor training or reductions in competency dimension assessment. Reductions in competency dimensions, however, did result in improvements in assessment quality across three studies.

DISCUSSION

The concept of mental workload in assessment in medical education needs further exploration, including investigation into valid measures of assessors' mental workload. It appears that adjusting raters' focus may be a valid strategy to improve assessment outcomes. Future research should be designed to inform how to best reduce load in assessments to improve quality, while balancing the type and quantity of data needed for judgments.

Failure affects subjective estimates of cognitive load through a negative carry-over effect in virtual reality simulation of hip fracture surgery

BACKGROUND

Cognitive overload can impair learning, and different factors might affect cognitive load during simulation-based training. In this study, we investigate the role of failure in repeated practice of virtual reality (VR) simulation of hip fracture surgery on cognitive load (CL) estimated by secondary-task reaction time test and two questionnaires.

METHODS

The VR simulation training program consisted of three competency levels of increasing complexity starting with the placement of a Kirschner wire in a fractured hip of one patient, adding clinical variability at the intermediate level, and performing the entire dynamic hip screw procedure in 24 different patients at the highest level. Thirteen consecutive passed simulations were required to advance to the next level. Performance was measured as passing/failing a procedure and the number of failed procedures within the latest three and five simulations. CL was measured objectively using reaction time testing during simulation and subjectively using the NASA-TLX and PAAS questionnaires. The study was carried out at a simulation center from November 2016 to March 2019. Forty-two first-year orthopedic surgery residents from the Central Denmark Region and the North Denmark Region participated in the training program.

RESULTS

A failing performance in the simulated procedure was associated with a higher CL than passing a procedure. The subjective CL estimates were affected by the number of failures during last three and five procedures with a higher number of failures being associated with a higher CL. In contrast, relative reaction time estimates of CL were not affected by previous failures.

CONCLUSIONS

Questionnaires for estimation of CL seem to be affected by participant frustration after failure-a meta-cognitive "carry-over" effect. This could be a general limitation of the subjective questionnaire approach to estimate CL. Reducing CL through instructional design and handling of participant frustration might improve the learning outcome of simulation training programs.

An observational study using eye tracking to assess resident and senior anesthetists' situation awareness and visual perception in postpartum hemorrhage high fidelity simulation

Background

The postpartum hemorrhage (PPH) is the leading cause of maternal mortality in the world. Human factors and especially situation awareness has primarily responsibility to explain suboptimal cares. Based on eye tracking and behavior analysis in high fidelity simulation of PPH management, the goal of this study is to identify perceptual and cognitive key parameters of the expertise.

Methods

Two groups of fifteen anesthetists (residents and experienced anesthetists) watched the beginning of a severe simulated PPH management. During this first experimental phase, situation awareness was assessed using SAGAT (Situation Awareness Global Assessment Technique) questionnaire and visual behavior was analyzed with eye tracking. In the continuity of the video sequence, they have to step in the PPH situation and to provide care to the simulated patient. Performance of cares was evaluated and self-assessed as well as cognitive load.

Results

No statistical difference between the residents and experienced anesthetists was observed on performance of simulated PPH management. The mean expected practice score was 76.9 ± 13.9%). Assessment of situation awareness (65 ± 7%), cognitive load (74.4 ± 11.3%) and theoretical knowledge of PPH (52.4 ± 3.5%) were also not statistically different between the two groups. Only results of self-assessed performance (respectively 66.1 ± 16.6 and 47.0 ± 20.8 for experts and residents) and eye-tracking data revealed that experts tended to get accurate evaluation of their performance and to monitor more the blood loss of the patient. Experts have in average 8.28% more fixating points than Novices and gazed the blood loss region longer (865 ms ± 439 vs. 717 ms ± 362).

Conclusions

This study pointed out the limits of classical assessment of performance, and human factors based on questionnaires to identify expertise in simulated PPH care. A neuroscientific approach with new technology like eye tracking could provide new objective and more sensitive insights on human factors in simulated medical emergency situations.

Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: a randomized trial

BACKGROUND

Virtual reality simulators combined with head-mounted displays enable highly immersive virtual reality (VR) for surgical skills training, potentially bridging the gap between the simulation environment and real-life operating room conditions. However, the increased complexity of the learning situation in immersive VR could potentially induce high cognitive load thereby inhibiting performance and learning. This study aims to compare cognitive load and performance in immersive VR and conventional VR simulation training.

METHODS

A randomized controlled trial of residents (n = 31) performing laparoscopic salpingectomies with an ectopic pregnancy in either immersive VR or conventional VR simulation. Cognitive load was estimated by secondary-task reaction time at baseline, and during nonstressor and stressor phases of the procedure. Simulator metrics were used to evaluate performance.

RESULTS

Cognitive load was increased by 66% and 58% during immersive VR and conventional VR simulation, respectively (p < 0.001), compared to baseline. A light stressor induced a further increase in cognitive load by 15.2% and a severe stressor by 43.1% in the immersive VR group compared to 23% (severe stressor) in the conventional VR group. Immersive VR also caused a significantly worse performance on most simulator metrics.

CONCLUSION

Immersive VR simulation training induces a higher cognitive load and results in a poorer performance than conventional VR simulation training in laparoscopy. High extraneous load and element interactivity in the immersive VR are suggested as mechanisms explaining this finding. However, immersive VR offers some potential advantages over conventional VR such as more real-life conditions but we only recommend introducing immersive VR in surgical skills training after initial training in conventional VR.

Learners' Experiences During Rapid Cycle Deliberate Practice Simulations: A Qualitative Analysis

INTRODUCTION

Rapid Cycle Deliberate Practice (RCDP) is a team-based simulation method, emphasizing repetitive practice over reflective debriefing, with progressively more challenging rounds, frequent starts and stops, and direct coaching. Although some studies have shown improved performance, no study has evaluated learners' perceptions. We aimed to explore learners' experiences during RCDP.

METHODS

This was a qualitative study of participants' perceptions regarding RCDP during their pediatric emergency medicine rotation. Participants completed surveys about RCDP learning. A purposive sample of residents and nurses were interviewed. Two coders analyzed all interview transcripts to identify emerging themes. Constant comparison analysis was performed until thematic saturation was achieved.

RESULTS

Forty-four participants completed surveys. Participants found RCDP interruptions beneficial and felt that they learned most during interruptions. Participants who were able to compare traditional and RCDP simulations felt that RCDP allowed more focused correction of mistakes, acquisition of new skills, and timely practice of team behaviors. Fourteen participants were interviewed. Three themes emerged. (1) The real-time corrections during RCDP allowed for learning and creation of new skills through repetition and practice. (2) The RCDP allowed learners to gain confidence, decrease anxiety, and learn in a safe environment. (3) By introducing new information in smaller chunks, participants maximized learning without cognitive overload.

CONCLUSIONS

Rapid Cycle Deliberate Practice is well received by learners. Because of frequent interruptions, learners noted early error correction, a safe learning environment, and skill improvement during RCDP. Learners recognized that the progressive advancement of RCDP helped prevent cognitive overload. Future studies should measure cognitive load and skills retention.

Cognitive load theory for training health professionals in the workplace: A BEME review of studies among diverse professions: BEME Guide No. 53

AIM

Cognitive load theory (CLT) is of increasing interest to health professions education researchers. CLT has intuitive applicability to workplace settings, yet how CLT should inform teaching, learning, and research in health professions workplaces is unclear.

METHOD

To map the existing literature, we performed a scoping review of studies involving cognitive load, mental effort and/or mental workload in professional workplace settings within and outside of the health professions. We included actual and simulated workplaces and workplace tasks.

RESULT

Searching eight databases, we identified 4571 citations, of which 116 met inclusion criteria. Studies were most often quantitative. Methods to measure cognitive load included psychometric, physiologic, and secondary task approaches. Few covariates of cognitive load or performance were studied. Overall cognitive load and intrinsic load were consistently negatively associated with the level of experience and performance. Studies consistently found distractions and other aspects of workplace environments as contributing to extraneous load. Studies outside the health professions documented similar findings to those within the health professions, supporting relevance of CLT to workplace learning.

CONCLUSION

The authors discuss implications for workplace teaching, curricular design, learning environment, and metacognition. To advance workplace learning, the authors suggest future CLT research should address higher-level questions and integrate other learning frameworks.

Temporal pattern of emotions and cognitive load during simulation training and debriefing

BACKGROUND

In the simulated clinical environment, there is a perceived benefit to the emotional activation experienced by learners; however, potential harm of excessive and/or negative emotions has also been hypothesized. An improved understanding of the emotional experiences of learners during each phase of the simulation session will inform instructional design.

METHODS

In this observational study, we asked 174 first-year medical students about their emotional state upon arrival to the simulation lab (t1). They were then trained on a standard simulation scenario, after which they rated their emotional state and perceived cognitive load (t2). After debriefing, we then asked them to again rate their emotions and cognitive load (t3).

RESULTS

Students reported that their experience of tranquility (a positive and low-arousal state) dropped from pre-scenario (t1) to post-scenario (t2), and returned to baseline levels after debriefing (t3), from 0.69 (0.87) to 0.14 (0.78) to 0.62 (0.78). Post scenario cognitive load was rated to be moderately high at 6.62 (1.12) and scores increased after debriefing to 6.90 (1.05) d = 0.26, p < 0.001. Cognitive load was associated with the simultaneous measures of emotions at both t2 and t3.

CONCLUSIONS

Participant emotions are significantly altered through the experience of medical simulation and emotions are associated with subjective ratings of cognitive load.

The effect of distributed virtual reality simulation training on cognitive load during subsequent dissection training

BACKGROUND

Complex tasks such as surgical procedures can induce excessive cognitive load (CL), which can have a negative effect on learning, especially for novices.

AIM

To investigate if repeated and distributed virtual reality (VR) simulation practice induces a lower CL and higher performance in subsequent cadaveric dissection training.

METHODS

In a prospective, controlled cohort study, 37 residents in otorhinolaryngology received VR simulation training either as additional distributed practice prior to course participation (intervention) (9 participants) or as standard practice during the course (control) (28 participants). Cognitive load was estimated as the relative change in secondary-task reaction time during VR simulation and cadaveric procedures.

RESULTS

Structured distributed VR simulation practice resulted in lower mean reaction times (32% vs. 47% for the intervention and control group, respectively, p < 0.01) as well as a superior final-product performance during subsequent cadaveric dissection training.

CONCLUSIONS

Repeated and distributed VR simulation causes a lower CL to be induced when the learning situation is increased in complexity. A suggested mechanism is the formation of mental schemas and reduction of the intrinsic CL. This has potential implications for surgical skills training and suggests that structured, distributed training be systematically implemented in surgical training curricula.

Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload

BACKGROUND

Surgeons in the operating theatre deal constantly with high-demand tasks that require simultaneous processing of a large amount of information. In certain situations, high cognitive load occurs, which may impact negatively on a surgeon's performance. This systematic review aims to provide a comprehensive understanding of the different methods used to assess surgeons' cognitive load, and a critique of the reliability and validity of current assessment metrics.

METHODS

A search strategy encompassing MEDLINE, Embase, Web of Science, PsycINFO, ACM Digital Library, IEEE Xplore, PROSPERO and the Cochrane database was developed to identify peer-reviewed articles published from inception to November 2016. Quality was assessed by using the Medical Education Research Study Quality Instrument (MERSQI). A summary table was created to describe study design, setting, specialty, participants, cognitive load measures and MERSQI score.

RESULTS

Of 391 articles retrieved, 84 met the inclusion criteria, totalling 2053 unique participants. Most studies were carried out in a simulated setting (59 studies, 70 per cent). Sixty studies (71 per cent) used self-reporting methods, of which the NASA Task Load Index (NASA-TLX) was the most commonly applied tool (44 studies, 52 per cent). Heart rate variability analysis was the most used real-time method (11 studies, 13 per cent).

CONCLUSION

Self-report instruments are valuable when the aim is to assess the overall cognitive load in different surgical procedures and assess learning curves within competence-based surgical education. When the aim is to assess cognitive load related to specific operative stages, real-time tools should be used, as they allow capture of cognitive load fluctuation. A combination of both subjective and objective methods might provide optimal measurement of surgeons' cognition.

Ward round simulation in final year medical students: Does it promote students learning?

INTRODUCTION

Ward round skills are essential for doctors in hospital settings. Literature shows medical students' deficiencies in these skills. Simulation has been used to train these skills. However, exposing learners to simulation at an early stage may be associated with a high cognitive load and limited learning. This study aims to determine how students experience this load and its interplay with performance and which factors promote and impair learning.

METHODS

Fifty-six final year medical students participated in a simulated ward round training exercise. Both students' performance and cognitive load were measured to determine if there was any correlation and interviews were carried out to understand which factors support and impair learning.

RESULTS

Performance scores revealed deficiencies in ward round skills. Students experienced a cognitive load that weakly correlated with performance. Qualitative findings provided important insights into simulated ward-based learning. It is clear that well-designed clinical scenarios, prioritization tasks, teamwork and feedback support students' learning process whereas distractions impair learning.

CONCLUSIONS

WRS proved to be a good teaching method to improve clinical skills at this stage as the cognitive load is not too high to impair learning. Hence, including tasks in the simulation design can enhance the learning process.

Cognitive load predicts point-of-care ultrasound simulator performance

INTRODUCTION

The ability to maintain good performance with low cognitive load is an important marker of expertise. Incorporating cognitive load measurements in the context of simulation training may help to inform judgements of competence. This exploratory study investigated relationships between demographic markers of expertise, cognitive load measures, and simulator performance in the context of point-of-care ultrasonography.

METHODS

Twenty-nine medical trainees and clinicians at the University of Toronto with a range of clinical ultrasound experience were recruited. Participants answered a demographic questionnaire then used an ultrasound simulator to perform targeted scanning tasks based on clinical vignettes. Participants were scored on their ability to both acquire and interpret ultrasound images. Cognitive load measures included participant self-report, eye-based physiological indices, and behavioural measures. Data were analyzed using a multilevel linear modelling approach, wherein observations were clustered by participants.

RESULTS

Experienced participants outperformed novice participants on ultrasound image acquisition. Ultrasound image interpretation was comparable between the two groups. Ultrasound image acquisition performance was predicted by level of training, prior ultrasound training, and cognitive load. There was significant convergence between cognitive load measurement techniques. A marginal model of ultrasound image acquisition performance including prior ultrasound training and cognitive load as fixed effects provided the best overall fit for the observed data.

DISCUSSION

In this proof-of-principle study, the combination of demographic and cognitive load measures provided more sensitive metrics to predict ultrasound simulator performance. Performance assessments which include cognitive load can help differentiate between levels of expertise in simulation environments, and may serve as better predictors of skill transfer to clinical practice.

Personalized Video Feedback and Repeated Task Practice Improve Laparoscopic Knot-Tying Skills: Two Controlled Trials

PURPOSE

Compare the effect of personalized feedback (PF) vs. task demonstration (TD), both delivered via video, on laparoscopic knot-tying skills and perceived workload; and evaluate the effect of repeated practice.

METHOD

General surgery interns and research fellows completed four repetitions of a simulated laparoscopic knot-tying task at one-month intervals. Midway between repetitions, participants received via e-mail either a TD video (demonstration by an expert) or a PF video (video of their own performance with voiceover from a blinded senior surgeon). Each participant received at least one video per format, with sequence randomly assigned. Outcomes included performance scores and NASA Task Load Index (NASA-TLX) scores. To evaluate the effectiveness of repeated practice, scores from these trainees on a separate delayed retention test were compared against historical controls who did not have scheduled repetitions.

RESULTS

Twenty-one trainees completed the randomized study. Mean change in performance scores was significantly greater for those receiving PF (difference = 23.1 of 150 [95% confidence interval (CI): 0, 46.2], P = .05). Perceived workload was also significantly reduced (difference = -3.0 of 20 [95% CI: -5.8, -0.3], P = .04). Compared with historical controls (N = 93), the 21 with scheduled repeated practice had higher scores on the laparoscopic knot-tying assessment two weeks after the final repetition (difference = 1.5 of 10 [95% CI: 0.2, 2.8], P = .02).

CONCLUSIONS

Personalized video feedback improves trainees' procedural performance and perceived workload compared with a task demonstration video. Brief monthly practice sessions support skill acquisition and retention.

Measuring achievement goal motivation, mindsets and cognitive load: validation of three instruments' scores

OBJECTIVE

Measurement of motivation and cognitive load has potential value in health professions education. Our objective was to evaluate the validity of scores from Dweck's Implicit Theories of Intelligence Scale (ITIS), Elliot's Achievement Goal Questionnaire-Revised (AGQ-R) and Leppink's cognitive load index (CLI).

METHODS

This was a validity study evaluating internal structure using reliability and factor analysis, and relationships with other variables using the multitrait-multimethod matrix. Two hundred and thirty-two secondary school students participated in a medical simulation-based training activity at an academic medical center. Pre-activity ITIS (implicit theory [mindset] domains: incremental, entity) and AGQ-R (achievement goal domains: mastery-approach, mastery-avoidance, performance-approach, performance-avoidance), post-activity CLI (cognitive load domains: intrinsic, extrinsic, germane) and task persistence (self-directed repetitions on a laparoscopic surgery task) were measured.

RESULTS

Internal consistency reliability (Cronbach's alpha) was > 0.70 for all domain scores except AGQ-R performance-avoidance (alpha 0.68) and CLI extrinsic load (alpha 0.64). Confirmatory factor analysis of ITIS and CLI scores demonstrated acceptable model fit. Confirmatory factor analysis of AGQ-R scores demonstrated borderline fit, and exploratory factor analysis suggested a three-domain model for achievement goals (mastery-approach, performance and avoidance). Correlations among scores from conceptually-related domains generally aligned with expectations, as follows: ITIS incremental and entity, r = -0.52; AGQ-R mastery-avoidance and performance-avoidance, r = 0.71; mastery-approach and performance-approach, r = 0.55; performance-approach and performance-avoidance, r = 0.43; mastery-approach and mastery-avoidance, r = 0.36; CLI germane and extrinsic, r = -0.35; ITIS incremental and AGQ-R mastery-approach, r = 0.34; ITIS incremental and CLI germane, r = 0.44; AGQ-R mastery-approach and CLI germane, r = 0.48 (all p < 0.001). We found no correlation between the number of task repetitions (i.e. persistence) and mastery-approach scores, r = -0.01.

CONCLUSIONS

ITIS and CLI scores had appropriate internal structures and relationships with other variables. AGQ-R scores fit a three-factor (not four-factor) model that collapsed avoidance into one domain, although relationships of other variables with the original four domain scores generally aligned with expectations. Mastery goals are positively correlated with germane cognitive load.

Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool

In general, researchers attempt to quantify cognitive load using physiologic and psychometric measures. Although the construct measured by both of these metrics is thought to represent overall cognitive load, there is a paucity of studies that compares these techniques to one another. The authors compared data obtained from one physiologic tool (pupillometry) to one psychometric tool (Paas scale) to explore whether they actually measured the construct of cognitive load as purported. Thirty-two participants with a range of resuscitation medicine experience and expertise completed resuscitation-medicine based multiple-choice-questions as well as arithmetic questions. Cognitive load, as measured by both tools, was found to be higher for the more difficult questions as well as for questions that were answered incorrectly (p < 0.001). The group with the least medical experience had higher cognitive load than both the intermediate and experienced groups when answering domain-specific questions (p = 0.023 and p = 0.003 respectively for the physiologic tool; p = 0.006 and p < 0.001 respectively for the psychometric tool). There was a strong positive correlation (Spearman's ρ = 0.827, p < 0.001 for arithmetic questions; Spearman's ρ = 0.606, p < 0.001 for medical questions) between the two cognitive load measurement tools. These findings support the validity argument that both physiologic and psychometric metrics measure the construct of cognitive load.

Non-technical skills of surgeons and anaesthetists in simulated operating theatre crises

Background

Deficiencies in non-technical skills (NTS) have been increasingly implicated in avoidable operating theatre errors. Accordingly, this study sought to characterize the impact of surgeon and anaesthetist non-technical skills on time to crisis resolution in a simulated operating theatre.

Methods

Non-technical skills were assessed during 26 simulated crises (haemorrhage and airway emergency) performed by surgical teams. Teams consisted of surgeons, anaesthetists and nurses. Behaviour was assessed by four trained raters using the Non-Technical Skills for Surgeons (NOTSS) and Anaesthetists' Non-Technical Skills (ANTS) rating scales before and during the crisis phase of each scenario. The primary endpoint was time to crisis resolution; secondary endpoints included NTS scores before and during the crisis. A cross-classified linear mixed-effects model was used for the final analysis.

Results

Thirteen different surgical teams were assessed. Higher NTS ratings resulted in significantly faster crisis resolution. For anaesthetists, every 1-point increase in ANTS score was associated with a decrease of 53·50 (95 per cent c.i. 31·13 to 75·87) s in time to crisis resolution (P &lt; 0·001). Similarly, for surgeons, every 1-point increase in NOTSS score was associated with a decrease of 64·81 (26·01 to 103·60) s in time to crisis resolution in the haemorrhage scenario (P = 0·001); however, this did not apply to the difficult airway scenario. Non-technical skills scores were lower during the crisis phase of the scenarios than those measured before the crisis for both surgeons and anaesthetists.

Conclusion

A higher level of NTS of surgeons and anaesthetists led to quicker crisis resolution in a simulated operating theatre environment.

The simulated clinical environment: Cognitive and emotional impact among undergraduates

CONTEXT

Simulated clinical immersion (SCI) is used in undergraduate healthcare programs to expose the learner to real-life situations in authentic simulated clinical environments. For novices, the environment in which the simulation occurs can be distracting and stressful, hence potentially compromising learning.

OBJECTIVES

This study aims to determine whether SCI (with environment) imposes greater extraneous cognitive load and stress on undergraduate pharmacy students than simulated patients (SP) (without environment). It also aims to explore how features of the simulated environment influence students' perception of learning.

METHODS

In this mixed-methods study, 143 undergraduate pharmacy students experienced both SCI and SP in a crossover design. After the simulations, participants rated their cognitive load and emotions. Thirty-five students met in focus groups to explore their perception of learning in simulation.

RESULTS

Intrinsic and extraneous cognitive load and stress scores in SCI were significantly but modestly higher compared to SP. Qualitative findings reveal that the physical environment in SCI generated more stress and affected students? focus. In SP, students concentrated on clinical reasoning. SCI stimulated a focus on data collection but impeded in-depth problem solving processes.

CONCLUSION

The physical environment in simulation influences what and how students learn. SCI was reported as more cognitively demanding than SP. Our findings emphasize the need for the development of adapted instructional design guidelines in simulation for novices.

Performance of a cognitive load inventory during simulated handoffs: Evidence for validity

BACKGROUND

Advancing patient safety during handoffs remains a public health priority. The application of cognitive load theory offers promise, but is currently limited by the inability to measure cognitive load types.

OBJECTIVE

To develop and collect validity evidence for a revised self-report inventory that measures cognitive load types during a handoff.

METHODS

Based on prior published work, input from experts in cognitive load theory and handoffs, and a think-aloud exercise with residents, a revised Cognitive Load Inventory for Handoffs was developed. The Cognitive Load Inventory for Handoffs has items for intrinsic, extraneous, and germane load. Students who were second- and sixth-year students recruited from a Dutch medical school participated in four simulated handoffs (two simple and two complex cases). At the end of each handoff, study participants completed the Cognitive Load Inventory for Handoffs, Paas' Cognitive Load Scale, and one global rating item for intrinsic load, extraneous load, and germane load, respectively. Factor and correlational analyses were performed to collect evidence for validity.

RESULTS

Confirmatory factor analysis yielded a single factor that combined intrinsic and germane loads. The extraneous load items performed poorly and were removed from the model. The score from the combined intrinsic and germane load items associated, as predicted by cognitive load theory, with a commonly used measure of overall cognitive load (Pearson's r = 0.83, p < 0.001), case complexity (beta = 0.74, p < 0.001), level of experience (beta = -0.96, p < 0.001), and handoff accuracy (r = -0.34, p < 0.001).

CONCLUSION

These results offer encouragement that intrinsic load during handoffs may be measured via a self-report measure. Additional work is required to develop an adequate measure of extraneous load.

The effect of implementing cognitive load theory-based design principles in virtual reality simulation training of surgical skills: a randomized controlled trial

Background

Cognitive overload can inhibit learning, and cognitive load theory-based instructional design principles can be used to optimize learning situations. This study aims to investigate the effect of implementing cognitive load theory-based design principles in virtual reality simulation training of mastoidectomy.

Methods

Eighteen novice medical students received 1 h of self-directed virtual reality simulation training of the mastoidectomy procedure randomized for standard instructions (control) or cognitive load theory-based instructions with a worked example followed by a problem completion exercise (intervention). Participants then completed two post-training virtual procedures for assessment and comparison. Cognitive load during the post-training procedures was estimated by reaction time testing on an integrated secondary task. Final-product analysis by two blinded expert raters was used to assess the virtual mastoidectomy performances.

Results

Participants in the intervention group had a significantly increased cognitive load during the post-training procedures compared with the control group (52 vs. 41 %, p = 0.02). This was also reflected in the final-product performance: the intervention group had a significantly lower final-product score than the control group (13.0 vs. 15.4, p < 0.005).

Conclusions

Initial instruction using worked examples followed by a problem completion exercise did not reduce the cognitive load or improve the performance of the following procedures in novices. Increased cognitive load when part tasks needed to be integrated in the post-training procedures could be a possible explanation for this. Other instructional designs and methods are needed to lower the cognitive load and improve the performance in virtual reality surgical simulation training of novices.