Cognitive engineering and health informatics: Applications and intersections

Multidimensional Approaches to Quality Measurement and Performance Improvement in the Ideal Crisis System

This work expands on the National Council for Mental Wellbeing whitepaper Quality Measurement in Crisis Services. The authors present 2 approaches to measure development: The first maps flow through the crisis continuum and defines metrics for each step of the process. The second uses the mnemonic ACCESS TO HELP to define system values, from the perspective of various stakeholders, with corresponding metrics. The article also includes case examples and discusses how metrics can align multiple components of a crisis system toward common goals, strategies for using metrics to drive quality improvement initiatives, and the complexities of measuring and interpreting data.

Information and Data Visualization Needs among Direct Care Nurses in the Intensive Care Unit

OBJECTIVES

Intensive care unit (ICU) direct care nurses spend 22% of their shift completing tasks within the electronic health record (EHR). Miscommunications and inefficiencies occur, particularly during patient hand-off, placing patient safety at risk. Redesigning how direct care nurses visualize and interact with patient information during hand-off is one opportunity to improve EHR use. A web-based survey was deployed to better understand the information and visualization needs at patient hand-off to inform redesign.

METHODS

A multicenter anonymous web-based survey of direct care ICU nurses was conducted (9-12/2021). Semi-structured interviews with stakeholders informed survey development. The primary outcome was identifying primary EHR data needs at patient hand-off for inclusion in future EHR visualization and interface development. Secondary outcomes included current use of the EHR at patient hand-off, EHR satisfaction, and visualization preferences. Frequencies, means, and medians were calculated for each data item then ranked in descending order to generate proportional quarters using SAS v9.4.

RESULTS

In total, 107 direct care ICU nurses completed the survey. The majority (46%, n = 49/107) use the EHR at patient hand-off to verify exchanged verbal information. Sixty-four percent (n = 68/107) indicated that current EHR visualization was insufficient. At the start of an ICU shift, primary EHR data needs included hemodynamics (mean 4.89 ± 0.37, 98%, n = 105), continuous IV medications (4.55 ± 0.73, 93%, n = 99), laboratory results (4.60 ± 0.56, 96%, n = 103), mechanical circulatory support devices (4.62 ± 0.72, 90%, n = 97), code status (4.40 ± 0.85, 59%, n = 108), and ventilation status (4.35 + 0.79, 51%, n = 108). Secondary outcomes included mean EHR satisfaction of 65 (0-100 scale, standard deviation = ± 21) and preferred future EHR user-interfaces to be organized by organ system (53%, n = 57/107) and visualized by tasks/schedule (61%, n = 65/107).

CONCLUSION

We identified information and visualization needs of direct care ICU nurses. The study findings could serve as a baseline toward redesigning an EHR interface.

Visual Analytics for Predicting Disease Outcomes Using Laboratory Test Results

Laboratory tests play an essential role in the early and accurate diagnosis of diseases. In this paper, we propose SUNRISE, a visual analytics system that allows the user to interactively explore the relationships between laboratory test results and a disease outcome. SUNRISE integrates frequent itemset mining (i.e., Eclat algorithm) with extreme gradient boosting (XGBoost) to develop more specialized and accurate prediction models. It also includes interactive visualizations to allow the user to interact with the model and track the decision process. SUNRISE helps the user probe the prediction model by generating input examples and observing how the model responds. Furthermore, it improves the user’s confidence in the generated predictions and provides them the means to validate the model’s response by illustrating the underlying working mechanism of the prediction models through visualization representations. SUNRISE offers a balanced distribution of processing load through the seamless integration of analytical methods with interactive visual representations to support the user’s cognitive tasks. We demonstrate the usefulness of SUNRISE through a usage scenario of exploring the association between laboratory test results and acute kidney injury, using large provincial healthcare databases from Ontario, Canada.

Minimally Trained Analysts Can Perform Fast, Objective Assessment of Orthopedic Technical Skill from Fluoroscopic Images

Skill assessment in orthopedics has traditionally relied on subjective impressions from a supervising surgeon. The feedback derived from these tools may be limited by bias and other practical issues. Objective analysis of intraoperative fluoroscopic images offers an inexpensive, repeatable, and precise assessment strategy without bias. Assessors generally refrain from using the scores of images obtained throughout the operation to evaluate skill for practical reasons. A new system was designed to facilitate rapid analysis of this fluoroscopy via minimally trained analysts. Four expert and four novice analysts independently measured one objective metric for skill using both a custom analysis software and a commercial alternative. Analysts were able to measure the objective metric three times faster when using the custom software, and without a practical difference in accuracy in comparison to the expert analysts using the commercial software. These results suggest that a well-designed fluoroscopy analysis system can facilitate inexpensive, reliable, and objective assessment of surgical skills.

Complexity of Documentation Needs for Children With Medical Complexity: Implications for Hospital Providers

OBJECTIVES

Care coordination is a core component of pediatric complex care programs (CCPs) supporting children with medical complexity (CMC) and their families. In this study, we aim to describe the purpose and characteristics of clinical care notes used within a pediatric CCP.

METHODS

We conducted observations of provider-family interactions during CCP clinic visits and 5 focus groups with members of the CCP. Focus groups were recorded and transcribed. Field observation notes and focus group transcripts were subjected to qualitative content analyses.

RESULTS

Four major themes help characterize clinical care notes: (1) Diversity of note types and functions: program staff author and use a number of unique note types shared across multiple stakeholders, including clinicians, families, and payers. (2) motivations for care note generation are different and explain how, why, and where they are created. (3) Program staff roles and configuration vary in relation to care note creation and use. (4) Sources of information for creating and updating notes are also diverse. Given the disparate information sources, integrating and maintaining up-to-date information for the child is challenging. To minimize information gaps, program staff devised unique but resource-intensive strategies, such as accompanying families during specialty clinic visits or visiting them inpatient.

CONCLUSIONS

CMC have complex documentation needs demonstrated by a variety of professional roles, care settings, and stakeholders involved in the generation and use of notes. Multiple opportunities exist to redesign and streamline the existing notes to support the cognitive work of clinicians providing care for CMC.

Characterizing and Visualizing Display and Task Fragmentation in the Electronic Health Record: Mixed Methods Design

BACKGROUND

The complexity of health care data and workflow presents challenges to the study of usability in electronic health records (EHRs). Display fragmentation refers to the distribution of relevant data across different screens or otherwise far apart, requiring complex navigation for the user's workflow. Task and information fragmentation also contribute to cognitive burden.

OBJECTIVE

This study aims to define and analyze some of the main sources of fragmentation in EHR user interfaces (UIs); discuss relevant theoretical, historical, and practical considerations; and use granular microanalytic methods and visualization techniques to help us understand the nature of fragmentation and opportunities for EHR optimization or redesign.

METHODS

Sunburst visualizations capture the EHR navigation structure, showing levels and sublevels of the navigation tree, allowing calculation of a new measure, the Display Fragmentation Index. Time belt visualizations present the sequences of subtasks and allow calculation of proportion per instance, a measure that quantifies task fragmentation. These measures can be used separately or in conjunction to compare EHRs as well as tasks and subtasks in workflows and identify opportunities for reductions in steps and fragmentation. We present an example use of the methods for comparison of 2 different EHR interfaces (commercial and composable) in which subjects apprehend the same patient case.

RESULTS

Screen transitions were substantially reduced for the composable interface (from 43 to 14), whereas clicks (including scrolling) remained similar.

CONCLUSIONS

These methods can aid in our understanding of UI needs under complex conditions and tasks to optimize EHR workflows and redesign.

A micro-analytic approach to understanding electronic health record navigation paths

Clinician task performance is significantly impacted by the navigational efficiency of the system interface. Here we propose and evaluate a navigational complexity framework useful for examining differences in electronic health record (EHR) interface systems and their impact on task performance. The methodological approach includes 1) expert-based methods-specifically, representational analysis (focused on interface elements), keystroke level modeling (KLM), and cognitive walkthrough; and 2) quantitative analysis of interactive behaviors based on video-captured observations. Medication administration record (MAR) tasks completed by nurses during preoperative (PreOp) patient assessment were studied across three Mayo Clinic regional campuses and three different EHR systems. By analyzing the steps executed within the interfaces involved to complete the MAR tasks, we characterized complexities in EHR navigation. These complexities were reflected in time spent on task, click counts, and screen transitions, and were found to potentially influence nurses' performance. Two of the EHR systems, employing a single screen format, required less time to complete (mean 101.5, range 106-97 s), respectively, compared to one system employing multiple screens (176 s, 73% increase). These complexities surfaced through trade-offs in cognitive processes that could potentially influence nurses' performance. Factors such as perceptual-motor activity, visual search, and memory load impacted navigational complexity. An implication of this work is that small tractable changes in interface design can substantially improve EHR navigation, overall usability, and workflow.

A Task-Analytic Framework Comparing Preoperative Electronic Health Record-Mediated Nursing Workflow in Different Settings

Preoperative care is a critical, yet complex, time-sensitive process. Optimization of workflow is challenging for many reasons, including a lack of standard workflow analysis methods. We sought to comprehensively characterize electronic health record-mediated preoperative nursing workflow. We employed a structured methodological framework to investigate and explain variations in the workflow. Video recording software captured 10 preoperative cases at Arizona and Florida regional referral centers. We compared the distribution of work for electronic health record tasks and off-screen tasks through quantitative analysis. Suboptimal patterns and reasons for variation were explored through qualitative analysis. Although both settings used the same electronic health record system, electronic health record tasks and off-screen tasks time distribution and patterns were notably different across two sites. Arizona nurses spent a longer time completing preoperative assessment. Electronic health record tasks occupied a higher proportion of time in Arizona, while off-screen tasks occupied a higher proportion in Florida. The contextual analysis helped to identify the variation associated with the documentation workload, preparation of the patient, and regional differences. These findings should seed hypotheses for future optimization efforts and research supporting standardization and harmonization of workflow across settings, post-electronic health record conversion.

Design and Evaluation of an Integrated, Patient-Focused Electronic Health Record Display for Emergency Medicine

BACKGROUND

Hospital emergency departments (EDs) are dynamic environments, involving coordination and shared decision making by staff who care for multiple patients simultaneously. While computerized information systems have been widely adopted in such clinical environments, serious issues have been raised related to their usability and effectiveness. In particular, there is a need to support clinicians to communicate and maintain awareness of a patient's health status, and progress through the ED plan of care.

OBJECTIVE

This study used work-centered usability methods to evaluate an integrated patient-focused status display designed to support ED clinicians' communication and situation awareness regarding a patient's health status and progress through their ED plan of care. The display design was informed by previous studies we conducted examining the information and cognitive support requirements of ED providers and nurses.

METHODS

ED nurse and provider participants were presented various scenarios requiring patient-prioritization and care-planning tasks to be performed using the prototype display. Participants rated the display in terms of its cognitive support, usability, and usefulness. Participants' performance on the various tasks, and their feedback on the display design and utility, was analyzed.

RESULTS

Participants provided ratings for usability and usefulness for the display sections using a work-centered usability questionnaire-mean scores for nurses and providers were 7.56 and 6.6 (1 being lowest and 9 being highest), respectively. General usability scores, based on the System Usability Scale tool, were rated as acceptable or marginally acceptable. Similarly, participants also rated the display highly in terms of support for specific cognitive objectives.

CONCLUSION

A novel patient-focused status display for emergency medicine was evaluated via a simulation-based study in terms of work-centered usability and usefulness. Participants' subjective ratings of usability, usefulness, and support for cognitive objectives were encouraging. These findings, including participants' qualitative feedback, provided insights for improving the design of the display.

The Human Interface of Biomedical Informatics

Biomedical informatics is the science of information, where information is defined as data with meaning. This definition identifies a fundamental challenge for informaticians: connecting with the healthcare team by enabling the acquisition, retrieval, and processing of information within the cognitive capabilities of the human brain. Informaticians can become aware of the constraints involved with cognitive processing and with workplace factors that impact how information is acquired and used to facilitate an improved user interface providing information to healthcare teams. Constraints affecting persons in the work environment include as follows: (1) cognitive processing of information; (2) cognitive load and memory capacity; (3) stress-affecting cognition; (4) cognitive distraction, attention, and multitasking; (5) cognitive bias and flexibility; (6) communication barriers; and (7) workplace environment. The human brain has a finite cognitive load capacity for processing new information. Short-term memory has limited throughput for processing of new informational items, while long-term memory supplies immediate simultaneous access to multiple informational items. Visual long-term memories can be extensive and detailed. Attention may be task dependent and highly variable among persons and requires maintaining control over distracting information. Multitasking reduces the effectiveness of working memory applied to each task. Transfer of information from person to person, or machine to person, is subject to cognitive bias and environmental stressors. High-stress levels increase emotional arousal to reduce memory formation and retrieval. The workplace environment can impact cognitive processes and stress, so maintaining civility augments cognitive abilities. Examples of human-computer interfaces employing principles of cognitive informatics inform design of systems to enhance the user interface.

Evidence-based design and evaluation of a whole genome sequencing clinical report for the reference microbiology laboratory

Background

Microbial genome sequencing is now being routinely used in many clinical and public health laboratories. Understanding how to report complex genomic test results to stakeholders who may have varying familiarity with genomics—including clinicians, laboratorians, epidemiologists, and researchers—is critical to the successful and sustainable implementation of this new technology; however, there are no evidence-based guidelines for designing such a report in the pathogen genomics domain. Here, we describe an iterative, human-centered approach to creating a report template for communicating tuberculosis (TB) genomic test results.

Methods

We used Design Study Methodology—a human centered approach drawn from the information visualization domain—to redesign an existing clinical report. We used expert consults and an online questionnaire to discover various stakeholders’ needs around the types of data and tasks related to TB that they encounter in their daily workflow. We also evaluated their perceptions of and familiarity with genomic data, as well as its utility at various clinical decision points. These data shaped the design of multiple prototype reports that were compared against the existing report through a second online survey, with the resulting qualitative and quantitative data informing the final, redesigned, report.

Results

We recruited 78 participants, 65 of whom were clinicians, nurses, laboratorians, researchers, and epidemiologists involved in TB diagnosis, treatment, and/or surveillance. Our first survey indicated that participants were largely enthusiastic about genomic data, with the majority agreeing on its utility for certain TB diagnosis and treatment tasks and many reporting some confidence in their ability to interpret this type of data (between 58.8% and 94.1%, depending on the specific data type). When we compared our four prototype reports against the existing design, we found that for the majority (86.7%) of design comparisons, participants preferred the alternative prototype designs over the existing version, and that both clinicians and non-clinicians expressed similar design preferences. Participants showed clearer design preferences when asked to compare individual design elements versus entire reports. Both the quantitative and qualitative data informed the design of a revised report, available online as a LaTeX template.

Conclusions

We show how a human-centered design approach integrating quantitative and qualitative feedback can be used to design an alternative report for representing complex microbial genomic data. We suggest experimental and design guidelines to inform future design studies in the bioinformatics and microbial genomics domains, and suggest that this type of mixed-methods study is important to facilitate the successful translation of pathogen genomics in the clinic, not only for clinical reports but also more complex bioinformatics data visualization software.

The impact of a diagnostic decision support system on the consultation: perceptions of GPs and patients

Background

Clinical decision support systems (DSS) aimed at supporting diagnosis are not widely used. This is mainly due to usability issues and lack of integration into clinical work and the electronic health record (EHR). In this study we examined the usability and acceptability of a diagnostic DSS prototype integrated with the EHR and in comparison with the EHR alone.

Methods

Thirty-four General Practitioners (GPs) consulted with 6 standardised patients (SPs) using only their EHR system (baseline session); on another day, they consulted with 6 different but matched for difficulty SPs, using the EHR with the integrated DSS prototype (DSS session). GPs were interviewed twice (at the end of each session), and completed the Post-Study System Usability Questionnaire at the end of the DSS session. The SPs completed the Consultation Satisfaction Questionnaire after each consultation.

Results

The majority of GPs (74%) found the DSS useful: it helped them consider more diagnoses and ask more targeted questions. They considered three user interface features to be the most useful: (1) integration with the EHR; (2) suggested diagnoses to consider at the start of the consultation and; (3) the checklist of symptoms and signs in relation to each suggested diagnosis. There were also criticisms: half of the GPs felt that the DSS changed their consultation style, by requiring them to code symptoms and signs while interacting with the patient. SPs sometimes commented that GPs were looking at their computer more than at them; this comment was made more often in the DSS session (15%) than in the baseline session (3%). Nevertheless, SP ratings on the satisfaction questionnaire did not differ between the two sessions.

Conclusions

To use the DSS effectively, GPs would need to adapt their consultation style, so that they code more information during rather than at the end of the consultation. This presents a potential barrier to adoption. Training GPs to use the system in a patient-centred way, as well as improvement of the DSS interface itself, could facilitate coding. To enhance patient acceptability, patients should be informed about the potential of the DSS to improve diagnostic accuracy.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-017-0477-6) contains supplementary material, which is available to authorized users.