A Review of Measuring the Cognitive Workload of Electronic Health Records

Quantifying Workload and Stress in Intensive Care Unit Nurses: Preliminary Evaluation Using Continuous Eye-Tracking

OBJECTIVE

(1) To assess mental workloads of intensive care unit (ICU) nurses in 12-hour working shifts (days and nights) using eye movement data; (2) to explore the impact of stress on the ocular metrics of nurses performing patient care in the ICU.

BACKGROUND

Prior studies have employed workload scoring systems or accelerometer data to assess ICU nurses' workload. This is the first naturalistic attempt to explore nurses' mental workload using eye movement data.

METHODS

Tobii Pro Glasses 2 eye-tracking and Empatica E4 devices were used to collect eye movement and physiological data from 15 nurses during 12-hour shifts (252 observation hours). We used mixed-effect models and an ordinal regression model with a random effect to analyze the changes in eye movement metrics during high stress episodes.

RESULTS

While the cadence and characteristics of nurse workload can vary between day shift and night shift, no significant difference in eye movement values was detected. However, eye movement metrics showed that the initial handoff period of nursing shifts has a higher mental workload compared with other times. Analysis of ocular metrics showed that stress is positively associated with an increase in number of eye fixations and gaze entropy, but negatively correlated with the duration of saccades and pupil diameter.

CONCLUSION

Eye-tracking technology can be used to assess the temporal variation of stress and associated changes with mental workload in the ICU environment. A real-time system could be developed for monitoring stress and workload for intervention development.

Cognitive Ergonomics: A Review of Interventions for Outpatient Practice

Doctoring is difficult mental work, involving many cognitively demanding processes such as diagnosing, decision-making, parallel processing, communicating, and managing the emotions of others. According to cognitive load theory (CLT), working memory is a limited cognitive resource that can support a finite amount of cognitive load. While the intrinsic cognitive load is the innate load associated with a task, the extraneous load is generated by inefficiency or suboptimal work conditions. Causes of extraneous cognitive load in healthcare include inefficiency, distractions, interruptions, multitasking, stress, poor communication, conflict, and incivility. High levels of cognitive load are associated with impaired function and an increased risk of burnout among physicians. Cognitive ergonomics is the branch of human factors and ergonomics (HFE) focused on supporting the cognitive processes of individuals within a system. In health care, where the cognitive burden on physicians is high, cognitive ergonomics can establish practices and systems that decrease extraneous cognitive load and support pertinent cognitive processes. In this review, we present cognitive ergonomics as a useful framework for conceptualizing an oft-overlooked dimension of labor and apply theory to practice by summarizing evidence-based cognitive ergonomics interventions for outpatient care settings. Our proposed interventions are structured within four general recommendations: 1. minimize distractions, interruptions, and multitasking; 2. optimize the use of the electronic health record (EHR); 3. optimize the use of health information systems (HIS); and 4. support good communication and teamwork. Best practices in cognitive ergonomics can benefit patients, minimize practice inefficiency, and support physician career longevity.

Registered nurses' reasoning process during care planning and documentation in the electronic health records: A concurrent think-aloud study

AIMS AND OBJECTIVES

To explore the clinical reasoning process of experienced registered nurses during care planning and documentation of nursing in the electronic health records of residents in long-term dementia care.

BACKGROUND

Clinical reasoning is an essential element in nursing practice. Registered nurses' clinical reasoning process during the documentation of nursing care in electronic health records has received little attention in nursing literature. Further research is needed to understand registered nurses' clinical reasoning, especially for care planning and documentation of dementia care due to its complexity and a large amount of information collected.

DESIGN

A qualitative explorative design was used with a concurrent think-aloud technique.

METHODS

The transcribed verbalisations were analysed using protocol analysis with referring phrase, assertional and script analyses. Data were collected over ten months in 2019-2020 from 12 registered nurses in three nursing homes offering special dementia care. The COREQ checklist for qualitative studies was used.

RESULTS

The nurses primarily focused on assessments and interventions during documentation. Most registered nurses used their experience and heuristics when reasoning about the residents' current health and well-being. They also used logical thinking or followed local practice rules when reasoning about planned or implemented interventions.

CONCLUSION

The registered nurses moved back and forth among all the elements in the nursing process. They used a variety of clinical reasoning attributes during care planning and nursing documentation. The most used clinical reasoning attributes were information processing, cognition and inference. The most focused information was planned and implemented interventions.

RELEVANCE TO CLINICAL PRACTICE

Knowledge of the clinical reasoning process of registered nurses during care planning and documentation should be used in developing electronic health record systems that support the workflow of registered nurses and enhance their ability to disseminate relevant information.

The Validity of Physiological Measures to Identify Differences in Intrinsic Cognitive Load

A sample of 33 experiments was extracted from the Web-of-Science database over a 5-year period (2016-2020) that used physiological measures to measure intrinsic cognitive load. Only studies that required participants to solve tasks of varying complexities using a within-subjects design were included. The sample identified a number of different physiological measures obtained by recording signals from four main body categories (heart and lungs, eyes, skin, and brain), as well as subjective measures. The overall validity of the measures was assessed by examining construct validity and sensitivity. It was found that the vast majority of physiological measures had some level of validity, but varied considerably in sensitivity to detect subtle changes in intrinsic cognitive load. Validity was also influenced by the type of task. Eye-measures were found to be the most sensitive followed by the heart and lungs, skin, and brain. However, subjective measures had the highest levels of validity. It is concluded that a combination of physiological and subjective measures is most effective in detecting changes in intrinsic cognitive load.

Evaluating Simplified Web Interfaces of Risk Models for Clinical Use: Pilot Survey Study

BACKGROUND

In this pilot study, we investigated sociotechnical factors that affect intention to use a simplified web model to support clinical decision making.

OBJECTIVE

We investigated factors that are known to affect technology adoption using the unified theory of acceptance and use of technology (UTAUT2) model. The goal was to pilot and test a tool to better support complex clinical assessments.

METHODS

Based on the results of a previously published work, we developed a web-based mobile user interface, WebModel, to allow users to work with regression equations and their predictions to evaluate the impact of various characteristics or treatments on key outcomes (eg, survival time) for chronic obstructive pulmonary disease. The WebModel provides a way to combat information overload and more easily compare treatment options. It limits the number of web forms presented to a user to between 1 and 20, rather than the dozens of detailed calculations typically required. The WebModel uses responsive design and can be used on multiple devices. To test the WebModel, we designed a questionnaire to probe the efficacy of the WebModel and assess the usability and usefulness of the system. The study was live for one month, and participants had access to it over that time. The questionnaire was administered online, and data from 674 clinical users who had access to the WebModel were captured. SPSS and R were used for statistical analysis.

RESULTS

The regression model developed from UTAUT2 constructs was a fit. Specifically, five of the seven factors were significant positive coefficients in the regression: performance expectancy (β=.2730; t=7.994; P<.001), effort expectancy (β=.1473; t=3.870; P=.001), facilitating conditions (β=.1644; t=3.849; P<.001), hedonic motivation (β=.2321; t=3.991; P<.001), and habit (β=.2943; t=12.732). Social influence was not a significant factor, while price value had a significant negative influence on intention to use the WebModel.

CONCLUSIONS

Our results indicate that multiple influences impact positive response to the system, many of which relate to the efficiency of the interface to provide clear information. Although we found that the price value was a negative factor, it is possible this was due to the removal of health workers from purchasing decisions. Given that this was a pilot test, and that the system was not used in a clinical setting, we could not examine factors related to actual workflow, patient safety, or social influence. This study shows that the concept of a simplified WebModel could be effective and efficient in reducing information overload in complex clinical decision making. We recommend further study to test this in a clinical setting and gather qualitative data from users regarding the value of the tool in practice.

Machine learning-based analysis of operator pupillary response to assess cognitive workload in clinical ultrasound imaging

Introduction

Pupillometry, the measurement of eye pupil diameter, is a well-established and objective modality correlated with cognitive workload. In this paper, we analyse the pupillary response of ultrasound imaging operators to assess their cognitive workload, captured while they undertake routine fetal ultrasound examinations. Our experiments and analysis are performed on real-world datasets obtained using remote eye-tracking under natural clinical environmental conditions.

Methods

Our analysis pipeline involves careful temporal sequence (time-series) extraction by retrospectively matching the pupil diameter data with tasks captured in the corresponding ultrasound scan video in a multi-modal data acquisition setup. This is followed by the pupil diameter pre-processing and the calculation of pupillary response sequences. Exploratory statistical analysis of the operator pupillary responses and comparisons of the distributions between ultrasonographic tasks (fetal heart versus fetal brain) and operator expertise (newly-qualified versus experienced operators) are performed. Machine learning is explored to automatically classify the temporal sequences into the corresponding ultrasonographic tasks and operator experience using temporal, spectral, and time-frequency features with classical (shallow) models, and convolutional neural networks as deep learning models.

Results

Preliminary statistical analysis of the extracted pupillary response shows a significant variation for different ultrasonographic tasks and operator expertise, suggesting different extents of cognitive workload in each case, as measured by pupillometry. The best-performing machine learning models achieve receiver operating characteristic (ROC) area under curve (AUC) values of 0.98 and 0.80, for ultrasonographic task classification and operator experience classification, respectively.

Conclusion

We conclude that we can successfully assess cognitive workload from pupil diameter changes measured while ultrasound operators perform routine scans. The machine learning allows the discrimination of the undertaken ultrasonographic tasks and scanning expertise using the pupillary response sequences as an index of the operators’ cognitive workload. A high cognitive workload can reduce operator efficiency and constrain their decision-making, hence, the ability to objectively assess cognitive workload is a first step towards understanding these effects on operator performance in biomedical applications such as medical imaging.

•

Machine learning-based pupillary response analysis is performed to assess operator cognitive workload in clinical ultrasound.

•

A systematic multi-modal data analysis pipeline is proposed using eye-tracking, pupillometry, and sonography data science.

•

Pertinent challenges of natural or real-world clinical datasets are addressed.

•

Pupillary responses around event triggers, different ultrasonographic tasks, and different operator experiences are studied.

•

Machine learning models are learnt to classify undertaken tasks or operator expertise from pupillometric time-series data.

Using Eye Tracking for Measuring Cognitive Workload During Clinical Simulations: Literature Review and Synthesis

High-fidelity clinical simulations can be used by clinicians to acquire technical (physical ability and knowledge) and non-technical (cognitive and social processes) skills. Excessive cognitive workload contributes to medical errors because of the impact on both technical and non-technical skills. Many studies measure cognitive workload with psychometric instruments that limit the assessment of cognitive workload to a single time period and may involve response bias. Using eye tracking to measure task-evoked pupillary responses allows the measurement of changes in pupil diameter related to the cognitive workload associated with a specific activity. Incorporating eye tracking with high-fidelity clinical simulations provides a reliable and continuous assessment of cognitive workload. The purpose of this literature review is to summarize the use of eye-tracking technology to measure cognitive workload of healthcare providers to generate evidence-based guidelines for measuring cognitive workload during high-fidelity clinical simulations. What this manuscript adds to the body of literature is a summary of best practices related to the different methods of measuring cognitive workload, benefits and limitations of using eye tracking, and high-fidelity clinical simulation design considerations for successful integration of eye tracking.

Clinical Decision Support and Implications for the Clinician Burnout Crisis

OBJECTIVES

This survey aimed to review aspects of clinical decision support (CDS) that contribute to burnout and identify key themes for improving the acceptability of CDS to clinicians, with the goal of decreasing said burnout.

METHODS

We performed a survey of relevant articles from 2018-2019 addressing CDS and aspects of clinician burnout from PubMed and Web of Science™. Themes were manually extracted from publications that met inclusion criteria.

RESULTS

Eighty-nine articles met inclusion criteria, including 12 review articles. Review articles were either prescriptive, describing how CDS should work, or analytic, describing how current CDS tools are deployed. The non-review articles largely demonstrated poor relevance and acceptability of current tools, and few studies showed benefits in terms of efficiency or patient outcomes from implemented CDS. Encouragingly, multiple studies highlighted steps that succeeded in improving both acceptability and relevance of CDS.

CONCLUSIONS

CDS can contribute to clinician frustration and burnout. Using the techniques of improving relevance, soliciting feedback, customization, measurement of outcomes and metrics, and iteration, the effects of CDS on burnout can be ameliorated.

Using Clinical Simulations to Train Healthcare Professionals to Use Electronic Health Records: A Literature Review

Unintended consequences are adverse events directly related to information technology and may result from inappropriate use of electronic health records by healthcare professionals. Electronic health record competency training has historically used didactic lectures with hands-on experience in a live classroom, and this method fails to teach learners proficiency because the sociotechnical factors that are present in real-world settings are excluded. Additionally, on-the-job training to gain competency can impair patient safety because it distracts clinicians from patient care activities. Clinical simulation-based electronic health record training allows learners to acquire technical and nontechnical skills in a safe environment that will not compromise patient safety. The purpose of this literature review was to summarize the current state-of-the-science on the use of clinical simulations to train healthcare professionals to use electronic health records. The benefits of using simulation-based training that incorporates an organization's contextual factors include improvement of interdisciplinary team communication, clinical performance, clinician-patient-technology communication skills, and recognition of patient safety issues. Design considerations for electronic health record training using clinical simulations involve establishing course objectives, identifying outcome measures, establishing content requirements of both the clinical simulation and electronic health record, and providing adequate debriefing.

Impact of Mobile Device-Based Clinical Decision Support Tool on Guideline Adherence and Mental Workload

OBJECTIVE

To evaluate the individual-level impact of an electronic clinical decision support (ECDS) tool, PedsGuide, on febrile infant clinical decision making and cognitive load.

METHODS

A counterbalanced, prospective, crossover simulation study was performed among attending and trainee physicians. Participants performed simulated febrile infant cases with use of PedsGuide and with standard reference text. Cognitive load was assessed using the NASA-Task Load Index (NASA-TLX), which determines mental, physical, temporal demand, effort, frustration, and performance. Usability was assessed with the System Usability Scale (SUS). Scores on cases and NASA-TLX scores were compared between condition states.

RESULTS

A total of 32 participants completed the study. Scores on febrile infant cases using PedsGuide were greater compared with standard reference text (89% vs 72%, P = .001). NASA-TLX scores were lower (ie, more optimal) with use of PedsGuide versus control (mental 6.34 vs 11.8, P < .001; physical 2.6 vs 6.1, P = .001; temporal demand 4.6 vs 8.0, P = .003; performance 4.5 vs 8.3, P < .001; effort 5.8 vs 10.7, P < .001; frustration 3.9 vs 10, P < .001). The SUS had an overall score of 88 of 100 with rating of acceptable on the acceptability scale.

CONCLUSIONS

Use of PedsGuide led to increased adherence to guidelines and decreased cognitive load in febrile infant management when compared with the use of a standard reference tool. This study employs a rarely used method of assessing ECDS tools using a multifaceted approach (medical decision-making, assessing usability, and cognitive workload,) that may be used to assess other ECDS tools in the future.