Critical care information display approaches and design frameworks: A systematic review and meta-analysis

User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications

BACKGROUND

Machine learning models can help anesthesiology clinicians assess patients and make clinical and operational decisions, but well-designed human-computer interfaces are necessary for machine learning model predictions to result in clinician actions that help patients. Therefore, the goal of this study was to apply a user-centered design framework to create a user interface for displaying machine learning model predictions of postoperative complications to anesthesiology clinicians.

METHODS

Twenty-five anesthesiology clinicians (attending anesthesiologists, resident physicians, and certified registered nurse anesthetists) participated in a 3-phase study that included (phase 1) semistructured focus group interviews and a card sorting activity to characterize user workflows and needs; (phase 2) simulated patient evaluation incorporating a low-fidelity static prototype display interface followed by a semistructured interview; and (phase 3) simulated patient evaluation with concurrent think-aloud incorporating a high-fidelity prototype display interface in the electronic health record. In each phase, data analysis included open coding of session transcripts and thematic analysis.

RESULTS

During the needs assessment phase (phase 1), participants voiced that (a) identifying preventable risk related to modifiable risk factors is more important than nonpreventable risk, (b) comprehensive patient evaluation follows a systematic approach that relies heavily on the electronic health record, and (c) an easy-to-use display interface should have a simple layout that uses color and graphs to minimize time and energy spent reading it. When performing simulations using the low-fidelity prototype (phase 2), participants reported that (a) the machine learning predictions helped them to evaluate patient risk, (b) additional information about how to act on the risk estimate would be useful, and (c) correctable problems related to textual content existed. When performing simulations using the high-fidelity prototype (phase 3), usability problems predominantly related to the presentation of information and functionality. Despite the usability problems, participants rated the system highly on the System Usability Scale (mean score, 82.5; standard deviation, 10.5).

CONCLUSIONS

Incorporating user needs and preferences into the design of a machine learning dashboard results in a display interface that clinicians rate as highly usable. Because the system demonstrates usability, evaluation of the effects of implementation on both process and clinical outcomes is warranted.

Patient Information Summarization in Clinical Settings: Scoping Review

BACKGROUND

Information overflow, a common problem in the present clinical environment, can be mitigated by summarizing clinical data. Although there are several solutions for clinical summarization, there is a lack of a complete overview of the research relevant to this field.

OBJECTIVE

This study aims to identify state-of-the-art solutions for clinical summarization, to analyze their capabilities, and to identify their properties.

METHODS

A scoping review of articles published between 2005 and 2022 was conducted. With a clinical focus, PubMed and Web of Science were queried to find an initial set of reports, later extended by articles found through a chain of citations. The included reports were analyzed to answer the questions of where, what, and how medical information is summarized; whether summarization conserves temporality, uncertainty, and medical pertinence; and how the propositions are evaluated and deployed. To answer how information is summarized, methods were compared through a new framework "collect-synthesize-communicate" referring to information gathering from data, its synthesis, and communication to the end user.

RESULTS

Overall, 128 articles were included, representing various medical fields. Exclusively structured data were used as input in 46.1% (59/128) of papers, text in 41.4% (53/128) of articles, and both in 10.2% (13/128) of papers. Using the proposed framework, 42.2% (54/128) of the records contributed to information collection, 27.3% (35/128) contributed to information synthesis, and 46.1% (59/128) presented solutions for summary communication. Numerous summarization approaches have been presented, including extractive (n=13) and abstractive summarization (n=19); topic modeling (n=5); summary specification (n=11); concept and relation extraction (n=30); visual design considerations (n=59); and complete pipelines (n=7) using information extraction, synthesis, and communication. Graphical displays (n=53), short texts (n=41), static reports (n=7), and problem-oriented views (n=7) were the most common types in terms of summary communication. Although temporality and uncertainty information were usually not conserved in most studies (74/128, 57.8% and 113/128, 88.3%, respectively), some studies presented solutions to treat this information. Overall, 115 (89.8%) articles showed results of an evaluation, and methods included evaluations with human participants (median 15, IQR 24 participants): measurements in experiments with human participants (n=31), real situations (n=8), and usability studies (n=28). Methods without human involvement included intrinsic evaluation (n=24), performance on a proxy (n=10), or domain-specific tasks (n=11). Overall, 11 (8.6%) reports described a system deployed in clinical settings.

CONCLUSIONS

The scientific literature contains many propositions for summarizing patient information but reports very few comparisons of these proposals. This work proposes to compare these algorithms through how they conserve essential aspects of clinical information and through the "collect-synthesize-communicate" framework. We found that current propositions usually address these 3 steps only partially. Moreover, they conserve and use temporality, uncertainty, and pertinent medical aspects to varying extents, and solutions are often preliminary.

The Intensive Care Unit Bundle Board: A Novel Real-Time Data Visualization Tool to Improve Maintenance Care for Invasive Catheters

BACKGROUND

Critically ill patients are at greater risk of healthcare-associated infections (HAIs). The use of maintenance bundles helps to reduce this risk but also generates a rapid accumulation of complex data that is difficult to aggregate and subsequently act upon.

OBJECTIVES

We hypothesized that a digital display summarizing nursing documentation of invasive catheters (including central venous access devices, arterial catheters, and urinary catheters) would improve invasive device maintenance care and documentation. Our secondary objectives were to see if this summary would reduce the duration of problematic conditions, that is, characteristics associated with increased risk of infection.

METHODS

We developed and implemented a data visualization tool called the "Bundle Board" to display nursing observations on invasive devices. The intervention was studied in a 28-bed medical intensive care unit (MICU). The Bundle Board was piloted for 6 weeks in June 2022 and followed by a comparison phase, where one MICU had Bundle Board access and another MICU at the same center did not. We retrospectively applied tile color coding logic to prior nursing documentation from 2021 until the pilot phase to facilitate comparison pre- and post-Bundle Board release.

RESULTS

After adjusting for time, other quality improvement efforts, and nursing shift, multiple linear regression demonstrated a statistically significant improvement in the completion of catheter care and documentation during the pilot phase (p < 0.0001) and comparison phase (p = 0.002). The median duration of documented problematic conditions was significantly reduced during the pilot phase (p < 0.0001) and in the MICU with the Bundle Board (comparison phase, p = 0.027).

CONCLUSION

We successfully developed a data visualization tool that changed ICU provider behavior, resulting in increased completion and documentation of maintenance care and reduced duration of problematic conditions for invasive catheters in MICU patients.

User Perceptions of Visual Blood: An International Mixed Methods Study on Novel Blood Gas Analysis Visualization

Blood gas analysis plays a central role in modern medicine. Advances in technology have expanded the range of available parameters and increased the complexity of their interpretation. By applying user-centered design principles, it is possible to reduce the cognitive load associated with interpreting blood gas analysis. In this international, multicenter study, we explored anesthesiologists' perspectives on Visual Blood, a novel visualization technique for presenting blood gas analysis results. We conducted interviews with participants following two computer-based simulation studies, the first utilizing virtual reality (VR) (50 participants) and the second without VR (70 participants). Employing the template approach, we identified key themes in the interview responses and formulated six statements, which were rated using Likert scales from 1 (strongly disagree) to 5 (strongly agree) in an online questionnaire. The most frequently mentioned theme was the positive usability features of Visual Blood. The online survey revealed that participants found Visual Blood to be an intuitive method for interpreting blood gas analysis (median 4, interquartile range (IQR) 4-4, p < 0.001). Participants noted that minimal training was required to effectively learn how to interpret Visual Blood (median 4, IQR 4-4, p < 0.001). However, adjustments are necessary to reduce visual overload (median 4, IQR 2-4, p < 0.001). Overall, Visual Blood received a favorable response. The strengths and weaknesses derived from these data will help optimize future versions of Visual Blood to improve the presentation of blood gas analysis results.

Design and Evaluation of an Intensive Care Unit Dashboard Built in Response to the COVID-19 Pandemic: Semistructured Interview Study

BACKGROUND

Dashboards and interactive displays are becoming increasingly prevalent in most health care settings and have the potential to streamline access to information, consolidate disparate data sources and deliver new insights. Our research focuses on intensive care units (ICUs) which are heavily instrumented, critical care environments that generate vast amounts of data and frequently require individualized support for each patient. Consequently, clinicians experience a high cognitive load, which can translate to suboptimal performance. The global COVID-19 pandemic exacerbated this problem by generating a large number of additional hospitalizations, which necessitated a new tool that would help manage ICUs' census. In a previous study, we interviewed clinicians at the University Hospitals Bristol and Weston National Health Service Foundation Trust to capture the requirements for bespoke dashboards that would alleviate this problem.

OBJECTIVE

This study aims to design, implement, and evaluate an ICU dashboard to allow for monitoring of the high volume of patients in need of critical care, particularly tailored to high-demand situations, such as those seen during the COVID-19 pandemic.

METHODS

Building upon the previously gathered requirements, we developed a dashboard, integrated it within the ICU of a National Health Service trust, and allowed all staff to access our tool. For evaluation purposes, participants were recruited and interviewed following a 25-day period during which they were able to use the dashboard clinically. The semistructured interviews followed a topic guide aimed at capturing the usability of the dashboard, supplemented with additional questions asked post hoc to probe themes established during the interview. Interview transcripts were analyzed using a thematic analysis framework that combined inductive and deductive approaches and integrated the Technology Acceptance Model.

RESULTS

A total of 10 participants with 4 different roles in the ICU (6 consultants, 2 junior doctors, 1 nurse, and 1 advanced clinical practitioner) participated in the interviews. Our analysis generated 4 key topics that prevailed across the data: our dashboard met the usability requirements of the participants and was found useful and intuitive; participants perceived that it impacted their delivery of patient care by improving the access to the information and better equipping them to do their job; the tool was used in a variety of ways and for different reasons and tasks; and there were barriers to integration of our dashboard into practice, including familiarity with existing systems, which stifled the adoption of our tool.

CONCLUSIONS

Our findings show that the perceived utility of the dashboard had a positive impact on the clinicians' workflows in the ICU. Improving access to information translated into more efficient patient care and transformed some of the existing processes. The introduction of our tool was met with positive reception, but its integration during the COVID-19 pandemic limited its adoption into practice.

Managing multiple perspectives in the collaborative design process of a team health information technology

We need to design technologies that support the work of health care teams; designing such solutions should integrate different clinical roles. However, we know little about the actual collaboration that occurs in the design process for a team-based care solution. This study examines how multiple perspectives were managed in the design of a team health IT solution aimed at supporting clinician information needs during pediatric trauma care transitions. We focused our analysis on four co-design sessions that involved multiple clinicians caring for pediatric trauma patients. We analyzed design session transcripts using content analysis and process coding guided by Détienne's (2006) co-design framework. We expanded upon Détienne (2006) three collaborative activities to identify specific themes and processes of collaboration between care team members engaged in the design process. The themes and processes describe how team members collaborated in a team health IT design process that resulted in a highly usable technology.

Evolution - removing paper and digitising the hospital

Purpose

A transition from paper to Electronic Health Records has numerous benefits, including better communication and information exchange and decreased errors by medical staff. However, if managed poorly, it can result in frustration, causing errors in patient care and reduced patient-clinician interaction. Furthermore, a drop in staff morale and clinician burnout due to familiarising themselves with the technology has been mentioned in previous studies. Therefore, the aim of this project is to monitor the change in morale of staff of the Oral and Maxillofacial Department in a hospital which underwent the change in October 2020. Objectives: To observe staff morale during transition from paper to Electronic Health Records; to encourage feedback.

Methods

After carrying out a Patient & Public Involvement consultation and receiving local research and development approval, a questionnaire was distributed to all members of the maxillofacial outpatients department on a regular basis.

Results

On average, around 25 members responded to the questionnaire during each collection. There was a noticeable divergence in responses week on week according to job role and age, but minimal difference is noted from gender point of view after the first week. The study emphasised the position that not all members were happy with the new system but only a small minority would want to return to paper notes.

Conclusion

Staff members adapt to change at different rates, which are multifactorial in nature. A change of this scale should be monitored closely to allow for a smoother transition and ensure staff burnout is minimised.

Factors contributing to vaccine hesitancy and reduced vaccine confidence in rural underserved populations

Vaccination remains one of the most effective ways to limit the spread of infectious diseases, and reduce mortality and morbidity in rural areas. Waning public confidence in vaccines, especially the COVID-19 vaccine, remains a cause for concern. A number of individuals in the US and worldwide remain complacent, choosing not to be vaccinated and/or delay COVID-19 vaccination, resulting in suboptimal herd immunity. The primary goal of this study is to identify modifiable factors contributing to COVID-19 vaccine hesitancy among vaccine-eligible individuals with access to vaccines in two under-resourced rural states, Alaska and Idaho. This qualitative study used semi-structured interviews with providers and focus groups with community participants in Alaska and Idaho. A moderator's guide was used to facilitate interviews and focus groups conducted and recorded using Zoom and transcribed verbatim. Thematic, qualitative analysis was conducted using QDA Miner. Themes and subthemes that emerged were labeled, categorized, and compared to previously described determinants of general vaccine hesitancy: established contextual, individual and/or social influences, vaccine and vaccination-specific concerns. Themes (n = 9) and sub-themes (n = 51) identified during the qualitative analysis highlighted a factor's contributing to COVID-19 vaccine hesitancy and poor vaccine uptake. Relevant influenceable factors were grouped into three main categories: confidence, complacency, and convenience. Vaccines are effective public health interventions to promote health and prevent diseases in rural areas. Practical solutions to engage healthcare providers, researchers, vaccine advocates, vaccine manufacturers, and other partners in local communities are needed to increase public trust in immunization systems to achieve community immunity.

Efficiency, efficacy and subjective user satisfaction of alternative laboratory report formats. An investigation on behalf of the Working Group for Postanalytical Phase (WG-POST), of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM)

OBJECTIVES

Although laboratory result presentation may lead to information overload and subsequent missed or delayed diagnosis, little has been done in the past to improve this post-analytical issue. We aimed to investigate the efficiency, efficacy and user satisfaction of alternative report formats.

METHODS

We redesigned cumulative (sparkline format) and single reports (improved tabular and z-log format) and tested these on 46 physicians, nurses and medical students in comparison to the classical tabular formats, by asking standardized questions on general items on the reports as well as on suspected diagnosis and follow-up treatment or diagnostics.

RESULTS

Efficacy remained at a very high level both in the new formats as well as in the classical formats. We found no significant difference in any of the groups. Efficiency improved in all groups when using the sparkline cumulative format and marginally when showing the improved tabular format. When asking medical questions, efficiency and efficacy remained similar between report formats and groups. All alternative reports were subjectively more attractive to the majority of participants.

CONCLUSIONS

Showing cumulative reports as a graphical display led to faster detection of general information on the report with the same level of correctness. Considering the familiarity bias of the classical single report formats, the borderline-significant improvement of the alternative tabular format and the non-inferiority of the z-log format, suggests that single reports might benefit from some improvements derived from basic information design.

Utilizing eye tracking to assess electronic health record use by pharmacists in the intensive care unit

DISCLAIMER

In an effort to expedite the publication of articles related to the COVID-19 pandemic, AJHP is posting these manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

PURPOSE

A study was conducted using high-fidelity electronic health record (EHR)-based simulations with incorporated eye tracking to understand the workflow of critical care pharmacists within the EHR, with specific attention to the data elements most frequently viewed.

METHODS

Eight critical care pharmacists were given 25 minutes to review 3 simulated intensive care unit (ICU) charts deployed in the simulation instance of the EHR. Using monitor-based eye trackers, time spent reviewing screens, clinical information accessed, and screens used to access specific information were reviewed and quantified to look for trends.

RESULTS

Overall, pharmacists viewed 25.5 total and 15.1 unique EHR screens per case. The majority of time was spent looking at screens focused on medications, followed by screens displaying notes, laboratory values, and vital signs. With regard to medication data, the vast majority of screen visitations were to view information on opioids/sedatives and antibiotics. With regard to laboratory values, the majority of views were focused on basic chemistry and hematology data. While there was significant variance between pharmacists, individual navigation patterns remained constant across cases.

CONCLUSION

The study results suggest that in addition to medication information, laboratory data and clinical notes are key focuses of ICU pharmacist review of patient records and that navigation to multiple screens is required in order to view these data with the EHR. New pharmacy-specific EHR interfaces should consolidate these elements within a primary interface.

Clinical Informatics and Quality Improvement in the Pediatric Intensive Care Unit

Clinical informatics can support quality improvement and patient safety in the pediatric intensive care unit (PICU) in several ways including data extraction, analysis, and decision support enabled by electronic health records (EHRs), and databases and registries. Clinical decision support (CDS), embedded in EHRs, now an integral part of the workflow in the PICU, includes several tools and is increasingly leveraging artificial intelligence (AI). Understanding the opportunities and challenges can improve the engagement of clinicians with the design, validation, and implementation of CDS, improve satisfaction with CDS, and improve patient safety, care quality, and value.

Requirements for bespoke ICU Dashboard in response to the COVID-19 Pandemic

Background

Intensive care units (ICUs) around the world are in high demand due to patients with COVID-19 requiring hospitalization. As researchers at the University of Bristol, we were approached to develop a bespoke data visualization dashboard to assist two local ICUs during the pandemic that will centralize disparate data sources in the ICU to help reduce the cognitive load on busy ICU staff in the ever-evolving pandemic.

Objective

The aim of this study was to conduct interviews with ICU staff in University Hospitals Bristol and Weston National Health Service Foundation Trust to elicit requirements for a bespoke dashboard to monitor the high volume of patients, particularly during the COVID-19 pandemic.

Methods

We conducted six semistructured interviews with clinical staff to obtain an overview of their requirements for the dashboard and to ensure its ultimate suitability for end users. Interview questions aimed to understand the job roles undertaken in the ICU, potential uses of the dashboard, specific issues associated with managing COVID-19 patients, key data of interest, and any concerns about the introduction of a dashboard into the ICU.

Results

From our interviews, we found the following design requirements: (1) a flexible dashboard, where the functionality can be updated quickly and effectively to respond to emerging information about the management of this new disease; (2) a mobile dashboard, which allows staff to move around on wards with a dashboard, thus potentially replacing paper forms to enable detailed and consistent data entry; (3) a customizable and intuitive dashboard, where individual users would be able to customize the appearance of the dashboard to suit their role; (4) real-time data and trend analysis via informative data visualizations that help busy ICU staff to understand a patient’s clinical trajectory; and (5) the ability to manage tasks and staff, tracking both staff and patient movements, handovers, and task monitoring to ensure the highest quality of care.

Conclusions

The findings of this study confirm that digital solutions for ICU use would potentially reduce the cognitive load of ICU staff and reduce clinical errors at a time of notably high demand of intensive health care.

Integration of an ICU Visualization Dashboard (i-Dashboard) as a Platform to Facilitate Multidisciplinary Rounds: A Cluster Randomized Controlled Trial (Preprint)

Background

 Multidisciplinary rounds (MDRs) are scheduled, patient-focused communication mechanisms among multidisciplinary providers in the intensive care unit (ICU).

Objective

i-Dashboard is a custom-developed visualization dashboard that supports (1) key information retrieval and reorganization, (2) time-series data, and (3) display on large touch screens during MDRs. This study aimed to evaluate the performance, including the efficiency of prerounding data gathering, communication accuracy, and information exchange, and clinical satisfaction of integrating i-Dashboard as a platform to facilitate MDRs.

Methods

A cluster-randomized controlled trial was performed in 2 surgical ICUs at a university hospital. Study participants included all multidisciplinary care team members. The performance and clinical satisfaction of i-Dashboard during MDRs were compared with those of the established electronic medical record (EMR) through direct observation and questionnaire surveys.

Results

Between April 26 and July 18, 2021, a total of 78 and 91 MDRs were performed with the established EMR and i-Dashboard, respectively. For prerounding data gathering, the median time was 10.4 (IQR 9.1-11.8) and 4.6 (IQR 3.5-5.8) minutes using the established EMR and i-Dashboard (P<.001), respectively. During MDRs, data misrepresentations were significantly less frequent with i-Dashboard (median 0, IQR 0-0) than with the established EMR (4, IQR 3-5; P<.001). Further, effective recommendations were significantly more frequent with i-Dashboard than with the established EMR (P<.001). The questionnaire results revealed that participants favored using i-Dashboard in association with the enhancement of care plan development and team participation during MDRs.

Conclusions

 i-Dashboard increases efficiency in data gathering. Displaying i-Dashboard on large touch screens in MDRs may enhance communication accuracy, information exchange, and clinical satisfaction. The design concepts of i-Dashboard may help develop visualization dashboards that are more applicable for ICU MDRs.

Trial Registration

ClinicalTrials.gov NCT04845698; https://clinicaltrials.gov/ct2/show/NCT04845698

Iterative heuristic design of temporal graphic displays with clinical domain experts

Conventional electronic health record information displays are not optimized for efficient information processing. Graphical displays that integrate patient information can improve information processing, especially in data-rich environments such as critical care. We propose an adaptable and reusable approach to patient information display with modular graphical components (widgets). We had two study objectives. First, reduce numerous widget prototype alternatives to preferred designs. Second, derive widget design feature recommendations. Using iterative human-centered design methods, we interviewed experts to hone design features of widgets displaying frequently measured data elements, e.g., heart rate, for acute care patient monitoring and real-time clinical decision-making. Participant responses to design queries were coded to calculate feature-set agreement, average prototype score, and prototype agreement. Two iterative interview cycles covering 64 design queries and 86 prototypes were needed to reach consensus on six feature sets. Interviewers agreed that line graphs with a smoothed or averaged trendline, 24-h timeframe, and gradient coloring for urgency were useful and informative features. Moreover, users agreed that widgets should include key functions: (1) adjustable reference ranges, (2) expandable timeframes, and (3) access to details on demand. Participants stated graphical widgets would be used to identify correlating patterns and compare abnormal measures across related data elements at a specific time. Combining theoretical principles and validated design methods was an effective and reproducible approach to designing widgets for healthcare displays. The findings suggest our widget design features and recommendations match critical care clinician expectations for graphical information display of continuous and frequently updated patient data.

Complexity of Documentation Needs for Children With Medical Complexity: Implications for Hospital Providers

OBJECTIVES

Care coordination is a core component of pediatric complex care programs (CCPs) supporting children with medical complexity (CMC) and their families. In this study, we aim to describe the purpose and characteristics of clinical care notes used within a pediatric CCP.

METHODS

We conducted observations of provider-family interactions during CCP clinic visits and 5 focus groups with members of the CCP. Focus groups were recorded and transcribed. Field observation notes and focus group transcripts were subjected to qualitative content analyses.

RESULTS

Four major themes help characterize clinical care notes: (1) Diversity of note types and functions: program staff author and use a number of unique note types shared across multiple stakeholders, including clinicians, families, and payers. (2) motivations for care note generation are different and explain how, why, and where they are created. (3) Program staff roles and configuration vary in relation to care note creation and use. (4) Sources of information for creating and updating notes are also diverse. Given the disparate information sources, integrating and maintaining up-to-date information for the child is challenging. To minimize information gaps, program staff devised unique but resource-intensive strategies, such as accompanying families during specialty clinic visits or visiting them inpatient.

CONCLUSIONS

CMC have complex documentation needs demonstrated by a variety of professional roles, care settings, and stakeholders involved in the generation and use of notes. Multiple opportunities exist to redesign and streamline the existing notes to support the cognitive work of clinicians providing care for CMC.

Physician experiences of screen-level features in a prominent electronic health record: Design recommendations from a qualitative study

The goal of this qualitative study was to assess physicians' perceptions around features of key screens within a prominent commercial EHR, and to solicit end-user recommendations for improved retrieval of high-priority clinical information. We conducted a qualitative, descriptive study of 25 physicians in a medical ICU setting. at a tertiary academic medical center. An in-depth, semi-structured interview guide was developed to elicit physician perceptions on information retrieval as well as favorable and unfavorable features of specific EHR screens. Transcripts were independently coded in a qualitative software management tool by at least two trained coders using a common code book. We successfully obtained vendor permission to map physicians perception's on full Epic© screenshots. Among the 25 physician participants (13 female; 5 attending physicians, 9 fellows, 11 residents), the majority of participants reported experiencing challenges finding clinical information in the EHR. We present the most favorable and unfavorable screen-level features for four central EHR screens: Flowsheet, Notes/Chart Review, Results Review, and Vital Signs. We also compiled participants' recommendations for a comprehensive EHR dashboard screen to better support clinical workflow and information retrieval in the medical ICU through User-Centered Design. ICU physicians demonstrated a mix of positive and negative attitudes toward specific screen-level features in a major vendor-based EHR system. Physician perceptions of information overload emerged as a theme across multiple EHR screens. Our findings underscore the importance of qualitative research and end-user feedback in EHR software design and interface optimization at both the vendor and institutional level.

Provider Preferences for Patient-Generated Health Data Displays in Pediatric Asthma: A Participatory Design Approach

Objective  There is a lack of evidence on how to best integrate patient-generated health data (PGHD) into electronic health record (EHR) systems in a way that supports provider needs, preferences, and workflows. The purpose of this study was to investigate provider preferences for the graphical display of pediatric asthma PGHD to support decisions and information needs in the outpatient setting.

Methods  In December 2019, we conducted a formative evaluation of information display prototypes using an iterative, participatory design process. Using multiple types of PGHD, we created two case-based vignettes for pediatric asthma and designed accompanying displays to support treatment decisions. Semi-structured interviews and questionnaires with six participants were used to evaluate the display usability and determine provider preferences.

Results  We identified provider preferences for display features, such as the use of color to indicate different levels of abnormality, the use of patterns to trend PGHD over time, and the display of environmental data. Preferences for display content included the amount of information and the relationship between data elements.

Conclusion  Overall, provider preferences for PGHD include a desire for greater detail, additional sources, and visual integration with relevant EHR data. In the design of PGHD displays, it appears that the visual synthesis of multiple PGHD elements facilitates the interpretation of the PGHD. Clinicians likely need more information to make treatment decisions when PGHD displays are introduced into practice. Future work should include the development of interactive interface displays with full integration of PGHD into EHR systems.

Impact of integrated graphical display on expert and novice diagnostic performance in critical care

OBJECTIVE

To determine the impact of a graphical information display on diagnosing circulatory shock.

MATERIALS AND METHODS

This was an experimental study comparing integrated and conventional information displays. Participants were intensivists or critical care fellows (experts) and first-year medical residents (novices).

RESULTS

The integrated display was associated with higher performance (87% vs 82%; P < .001), less time (2.9 vs 3.5 min; P = .008), and more accurate etiology (67% vs 54%; P = .048) compared to the conventional display. When stratified by experience, novice physicians using the integrated display had higher performance (86% vs 69%; P < .001), less time (2.9 vs 3.7 min; P = .03), and more accurate etiology (65% vs 42%; P = .02); expert physicians using the integrated display had nonsignificantly improved performance (87% vs 82%; P = .09), time (2.9 vs 3.3; P = .28), and etiology (69% vs 67%; P = .81).

DISCUSSION

The integrated display appeared to support efficient information processing, which resulted in more rapid and accurate circulatory shock diagnosis. Evidence more strongly supported a difference for novices, suggesting that graphical displays may help reduce expert-novice performance gaps.

A Perioperative Care Display for Understanding High Acuity Patients

BACKGROUND

The data visualization literature asserts that the details of the optimal data display must be tailored to the specific task, the background of the user, and the characteristics of the data. The general organizing principle of a concept-oriented display is known to be useful for many tasks and data types.

OBJECTIVES

In this project, we used general principles of data visualization and a co-design process to produce a clinical display tailored to a specific cognitive task, chosen from the anesthesia domain, but with clear generalizability to other clinical tasks. To support the work of the anesthesia-in-charge (AIC) our task was, for a given day, to depict the acuity level and complexity of each patient in the collection of those that will be operated on the following day. The AIC uses this information to optimally allocate anesthesia staff and providers across operating rooms.

METHODS

We used a co-design process to collaborate with participants who work in the AIC role. We conducted two in-depth interviews with AICs and engaged them in subsequent input on iterative design solutions.

RESULTS

Through a co-design process, we found (1) the need to carefully match the level of detail in the display to the level required by the clinical task, (2) the impedance caused by irrelevant information on the screen such as icons relevant only to other tasks, and (3) the desire for a specific but optional trajectory of increasingly detailed textual summaries.

CONCLUSION

This study reports a real-world clinical informatics development project that engaged users as co-designers. Our process led to the user-preferred design of a single binary flag to identify the subset of patients needing further investigation, and then a trajectory of increasingly detailed, text-based abstractions for each patient that can be displayed when more information is needed.

Modeling physician variability to prioritize relevant medical record information

Objective

Patient information can be retrieved more efficiently in electronic medical record (EMR) systems by using machine learning models that predict which information a physician will seek in a clinical context. However, information-seeking behavior varies across EMR users. To explicitly account for this variability, we derived hierarchical models and compared their performance to nonhierarchical models in identifying relevant patient information in intensive care unit (ICU) cases.

Materials and methods

Critical care physicians reviewed ICU patient cases and selected data items relevant for presenting at morning rounds. Using patient EMR data as predictors, we derived hierarchical logistic regression (HLR) and standard logistic regression (LR) models to predict their relevance.

Results

In 73 pairs of HLR and LR models, the HLR models achieved an area under the receiver operating characteristic curve of 0.81, 95% confidence interval (CI) [0.80-0.82], which was statistically significantly higher than that of LR models (0.75, 95% CI [0.74-0.76]). Further, the HLR models achieved statistically significantly lower expected calibration error (0.07, 95% CI [0.06-0.08]) than LR models (0.16, 95% CI [0.14-0.17]).

Discussion

The physician reviewers demonstrated variability in selecting relevant data. Our results show that HLR models perform significantly better than LR models with respect to both discrimination and calibration. This is likely due to explicitly modeling physician-related variability.

Conclusion

Hierarchical models can yield better performance when there is physician-related variability as in the case of identifying relevant information in the EMR.

User-Centered Clinical Display Design Issues for Inpatient Providers

BACKGROUND

Suboptimal information display in electronic health records (EHRs) is a notorious pain point for users. Designing an effective display is difficult, due in part to the complex and varied nature of clinical practice.

OBJECTIVE

This article aims to understand the goals, constraints, frustrations, and mental models of inpatient medical providers when accessing EHR data, to better inform the display of clinical information.

METHODS

A multidisciplinary ethnographic study of inpatient medical providers.

RESULTS

Our participants' primary goal was usually to assemble a clinical picture around a given question, under the constraints of time pressure and incomplete information. To do so, they tend to use a mental model of multiple layers of abstraction when thinking of patients and disease; they prefer immediate pattern recognition strategies for answering clinical questions, with breadth-first or depth-first search strategies used subsequently if needed; and they are sensitive to data relevance, completeness, and reliability when reading a record.

CONCLUSION

These results conflict with the ubiquitous display design practice of separating data by type (test results, medications, notes, etc.), a mismatch that is known to encumber efficient mental processing by increasing both navigation burden and memory demands on users. A popular and obvious solution is to select or filter the data to display exactly what is presumed to be relevant to the clinical question, but this solution is both brittle and mistrusted by users. A less brittle approach that is more aligned with our users' mental model could use abstraction to summarize details instead of filtering to hide data. An abstraction-based approach could allow clinicians to more easily assemble a clinical picture, to use immediate pattern recognition strategies, and to adjust the level of displayed detail to their particular needs. It could also help the user notice unanticipated patterns and to fluidly shift attention as understanding evolves.

Situation Awareness-Oriented Patient Monitoring with Visual Patient Technology: A Qualitative Review of the Primary Research

Visual Patient technology is a situation awareness-oriented visualization technology that translates numerical and waveform patient monitoring data into a new user-centered visual language. Vital sign values are converted into colors, shapes, and rhythmic movements-a language humans can easily perceive and interpret-on a patient avatar model in real time. In this review, we summarize the current state of the research on the Visual Patient, including the technology, its history, and its scientific context. We also provide a summary of our primary research and a brief overview of research work on similar user-centered visualizations in medicine. In several computer-based studies under various experimental conditions, Visual Patient transferred more information per unit time, increased perceived diagnostic certainty, and lowered perceived workload. Eye tracking showed the technology worked because of the way it synthesizes and transforms vital sign information into new and logical forms corresponding to the real phenomena. The technology could be particularly useful for improving situation awareness in settings with high cognitive demand or when users must make quick decisions. This comprehensive review of Visual Patient research is the foundation for an evaluation of the technology in clinical applications, starting with a high-fidelity simulation study in early 2020.

Association of an Electronic Health Record Add-on App for Neonatal Bilirubin Management With Physician Efficiency and Care Quality

IMPORTANCE

The usefulness of electronic health record (EHR) systems could be significantly enhanced by innovative, third-party EHR add-on apps.

OBJECTIVE

To evaluate whether an EHR add-on app for neonatal bilirubin management can save clinicians time and improve patient care.

DESIGN, SETTING, AND PARTICIPANTS

This quality improvement study was conducted at the University of Utah Health Well Baby nursery and outpatient clinics and consisted of 4 substudies: (1) time savings were estimated in an experimental task-timing study comparing the time required for physicians to manage newborns' bilirubin levels with and without the add-on app, (2) app use was estimated from app logs, (3) health care use measures and guideline compliance were compared retrospectively before and after the intervention, and (4) clinician-perceived usability was measured through System Usability Scale surveys. The study took place between April 1, 2016, and September 3, 2019. Data analyses were conducted from October 30, 2018, to September 23, 2019.

INTERVENTIONS

At baseline, clinicians used a manual approach to ensure compliance with an evidence-based clinical guideline for neonatal bilirubin management. To facilitate guideline compliance, an EHR add-on app that automatically retrieves, organizes, and visualizes relevant patient data was developed. The app provides patient-specific assessments and recommendations, including the risk of rebound hyperbilirubinemia following phototherapy based on a predictive model. The add-on app was integrated with the University of Utah Health EHR on April 12, 2017.

MAIN OUTCOMES AND MEASURES

Clinician time savings, app use, health care use measures, guideline-compliant phototherapy ordering, and perceived usability as measured by the System Usability Scale survey. The survey is composed of 10 statements with responses ranging from 1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100, with ratings described as worst imaginable (mean System Usability Scale score, 12.5), awful (20.3), poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best imaginable (90.9).

RESULTS

In 2018, the application was used 20 516 times by clinicians for 91.84% of eligible newborns. Use of the app saved 66 seconds for bilirubin management tasks compared with a commonly used tool (95% CI, 53-79 seconds; P < .001). Following the intervention, health care use rates remained stable, while orders for clinically appropriate phototherapy during hospitalization increased for newborns with bilirubin levels above the guideline-recommended threshold (odds ratio, 1.84; 95% CI, 1.16-2.90; P = .009). Surveys indicated excellent usability (System Usability Scale score, 83.90; 95% CI, 81.49-86.31).

CONCLUSIONS AND RELEVANCE

Well-designed EHR add-on apps may save clinicians time and improve patient care. If time-saving apps, such as the bilirubin app, were implemented widely across institutions and care domains, the potential association with improved patient care and clinician efficiency could be significant. The University of Utah Health bilirubin app is being prepared for release into EHR app stores as free-to-use software.

Using machine learning to selectively highlight patient information

BACKGROUND

Electronic medical record (EMR) systems need functionality that decreases cognitive overload by drawing the clinician's attention to the right data, at the right time. We developed a Learning EMR (LEMR) system that learns statistical models of clinician information-seeking behavior and applies those models to direct the display of data in future patients. We evaluated the performance of the system in identifying relevant patient data in intensive care unit (ICU) patient cases.

METHODS

To capture information-seeking behavior, we enlisted critical care medicine physicians who reviewed a set of patient cases and selected data items relevant to the task of presenting at morning rounds. Using patient EMR data as predictors, we built machine learning models to predict their relevancy. We prospectively evaluated the predictions of a set of high performing models.

RESULTS

On an independent evaluation data set, 25 models achieved precision of 0.52, 95% CI [0.49, 0.54] and recall of 0.77, 95% CI [0.75, 0.80] in identifying relevant patient data items. For data items missed by the system, the reviewers rated the effect of not seeing those data from no impact to minor impact on patient care in about 82% of the cases.

CONCLUSION

Data-driven approaches for adaptively displaying data in EMR systems, like the LEMR system, show promise in using information-seeking behavior of clinicians to identify and highlight relevant patient data.