The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. Crit Care Med. 2011;39:1626-1634. [PMID: 21478739 DOI: 10.1097/ccm.0b013e31821858a0]

Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences

OBJECTIVES

To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities.

MATERIALS AND METHODS

EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings' and APPs' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.

RESULTS

Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.

DISCUSSION

We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.

CONCLUSION

An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.

Improving Situation Awareness to Advance Patient Outcomes: A Systematic Literature Review

Improving nurses' situation awareness skills would likely improve patient status recognition and prevent adverse events. Technologies such as electronic health record dashboards can be a promising approach to support nurses' situation awareness. However, the effect of these dashboards on this skill is unknown. This systematic literature review explores the evidence around interventions to improve nurses' situation awareness at the point of care. Current research on this subject is limited. Studies that examined the use of electronic health record dashboards as an intervention had weak evidence to support their effectiveness. Other interventions, including communication interventions and structured nursing assessments, may also improve situation awareness, but more research is needed to confirm this. It is important to carefully consider the design and content of situation awareness interventions, as well as the specific outcomes being measured, when designing situation awareness interventions. Overall, there is a need for higher-quality research in this area to determine the most effective interventions for improving nurse situation awareness. Future studies should focus on developing dashboards that follow a theoretical situation awareness model information and represent all situation awareness levels.

Electronic Health Record Usability When Caring for Children With Medical Complexity

OBJECTIVES

To evaluate usability of and clinician satisfaction with the electronic health record (EHR) in the context of caring for children with medical complexity (CMC) at a large academic pediatric hospital and to identify key areas for targeted improvements.

METHODS

Cross-sectional study of pediatric faculty and advanced practice providers across several pediatric specialties using an online Research Electronic Data Capture survey. EHR usability was measured with 6 validated questions from the National Usability-Focused Health Information System Scale, and satisfaction with common EHR functionalities was measured with 6 original Likert-scale questions and 3 free-text questions. Results were analyzed with bivariate testing.

RESULTS

More than half of providers completed the survey (n = 81, response rate 56%). When asked 6 positively-phrased, validated usability questions, a majority of respondents either agreed or were neutral. Respondents were most dissatisfied with information quality of the summary view and most satisfied with physician communication. Older respondents were less satisfied with EHR usability (P < .01). Focusing on functionalities needed for CMC, the majority of respondents were dissatisfied with the chart review process. More respondents were dissatisfied with order entry (P = .002) and documentation (P = .017) when caring for CMC compared with caring for other patients. The most cited challenges were locating recent patient data, performing an accurate medication reconciliation, and lack of specialized documentation templates.

CONCLUSIONS

Clinicians are less satisfied with common EHR functionalities when caring for CMC compared with caring for all other patients. Targeted interventions to improve usability and common EHR functionalities for CMC are necessary to improve the user experience.

Development and usability testing of a patient digital twin for critical care education: a mixed methods study

Background

Digital twins are computerized patient replicas that allow clinical interventions testing in silico to minimize preventable patient harm. Our group has developed a novel application software utilizing a digital twin patient model based on electronic health record (EHR) variables to simulate clinical trajectories during the initial 6 h of critical illness. This study aimed to assess the usability, workload, and acceptance of the digital twin application as an educational tool in critical care.

Methods

A mixed methods study was conducted during seven user testing sessions of the digital twin application with thirty-five first-year internal medicine residents. Qualitative data were collected using a think-aloud and semi-structured interview format, while quantitative measurements included the System Usability Scale (SUS), NASA Task Load Index (NASA-TLX), and a short survey.

Results

Median SUS scores and NASA-TLX were 70 (IQR 62.5-82.5) and 29.2 (IQR 22.5-34.2), consistent with good software usability and low to moderate workload, respectively. Residents expressed interest in using the digital twin application for ICU rotations and identified five themes for software improvement: clinical fidelity, interface organization, learning experience, serious gaming, and implementation strategies.

Conclusion

A digital twin application based on EHR clinical variables showed good usability and high acceptance for critical care education.

A Novel Continuous Real-Time Vital Signs Viewer for Intensive Care Units: Design and Evaluation Study

BACKGROUND

Clinicians working in intensive care units (ICUs) are immersed in a cacophony of alarms and a relentless onslaught of data. Within this frenetic environment, clinicians make high-stakes decisions using many data sources and are often oversaturated with information of varying quality. Traditional bedside monitors only depict static vital signs data, and these data are not easily viewable remotely. Clinicians must rely on separate nursing charts-handwritten or electric-to review physiological patterns, including signs of potential clinical deterioration. An automated physiological data viewer has been developed to provide at-a-glance summaries and to assist with prioritizing care for multiple patients who are critically ill.

OBJECTIVE

This study aims to evaluate a novel vital signs viewer system in a level 1 trauma center by subjectively assessing the viewer's utility in a high-volume ICU setting.

METHODS

ICU attendings were surveyed during morning rounds. Physicians were asked to conduct rounds normally, using data reported from nurse charts and briefs from fellows to inform their clinical decisions. After the physician finished their assessment and plan for the patient, they were asked to complete a questionnaire. Following completion of the questionnaire, the viewer was presented to ICU physicians on a tablet personal computer that displayed the patient's physiologic data (ie, shock index, blood pressure, heart rate, temperature, respiratory rate, and pulse oximetry), summarized for up to 72 hours. After examining the viewer, ICU physicians completed a postview questionnaire. In both questionnaires, the physicians were asked questions regarding the patient's stability, status, and need for a higher or lower level of care. A hierarchical clustering analysis was used to group participating ICU physicians and assess their general reception of the viewer.

RESULTS

A total of 908 anonymous surveys were collected from 28 ICU physicians from February 2015 to June 2017. Regarding physicians' perception of whether the viewer enhanced the ability to assess multiple patients in the ICU, 5% (45/908) strongly agreed, 56.6% (514/908) agreed, 35.3% (321/908) were neutral, 2.9% (26/908) disagreed, and 0.2% (2/908) strongly disagreed.

CONCLUSIONS

Morning rounds in a trauma center ICU are conducted in a busy environment with many data sources. This study demonstrates that organized physiologic data and visual assessment can improve situation awareness, assist clinicians with recognizing changes in patient status, and prioritize care.

Patient Information Summarization in Clinical Settings: Scoping Review

BACKGROUND

Information overflow, a common problem in the present clinical environment, can be mitigated by summarizing clinical data. Although there are several solutions for clinical summarization, there is a lack of a complete overview of the research relevant to this field.

OBJECTIVE

This study aims to identify state-of-the-art solutions for clinical summarization, to analyze their capabilities, and to identify their properties.

METHODS

A scoping review of articles published between 2005 and 2022 was conducted. With a clinical focus, PubMed and Web of Science were queried to find an initial set of reports, later extended by articles found through a chain of citations. The included reports were analyzed to answer the questions of where, what, and how medical information is summarized; whether summarization conserves temporality, uncertainty, and medical pertinence; and how the propositions are evaluated and deployed. To answer how information is summarized, methods were compared through a new framework "collect-synthesize-communicate" referring to information gathering from data, its synthesis, and communication to the end user.

RESULTS

Overall, 128 articles were included, representing various medical fields. Exclusively structured data were used as input in 46.1% (59/128) of papers, text in 41.4% (53/128) of articles, and both in 10.2% (13/128) of papers. Using the proposed framework, 42.2% (54/128) of the records contributed to information collection, 27.3% (35/128) contributed to information synthesis, and 46.1% (59/128) presented solutions for summary communication. Numerous summarization approaches have been presented, including extractive (n=13) and abstractive summarization (n=19); topic modeling (n=5); summary specification (n=11); concept and relation extraction (n=30); visual design considerations (n=59); and complete pipelines (n=7) using information extraction, synthesis, and communication. Graphical displays (n=53), short texts (n=41), static reports (n=7), and problem-oriented views (n=7) were the most common types in terms of summary communication. Although temporality and uncertainty information were usually not conserved in most studies (74/128, 57.8% and 113/128, 88.3%, respectively), some studies presented solutions to treat this information. Overall, 115 (89.8%) articles showed results of an evaluation, and methods included evaluations with human participants (median 15, IQR 24 participants): measurements in experiments with human participants (n=31), real situations (n=8), and usability studies (n=28). Methods without human involvement included intrinsic evaluation (n=24), performance on a proxy (n=10), or domain-specific tasks (n=11). Overall, 11 (8.6%) reports described a system deployed in clinical settings.

CONCLUSIONS

The scientific literature contains many propositions for summarizing patient information but reports very few comparisons of these proposals. This work proposes to compare these algorithms through how they conserve essential aspects of clinical information and through the "collect-synthesize-communicate" framework. We found that current propositions usually address these 3 steps only partially. Moreover, they conserve and use temporality, uncertainty, and pertinent medical aspects to varying extents, and solutions are often preliminary.

Standardized Comparison of Voice-Based Information and Documentation Systems to Established Systems in Intensive Care: Crossover Study

BACKGROUND

The medical teams in intensive care units (ICUs) spend increasing amounts of time at computer systems for data processing, input, and interpretation purposes. As each patient creates about 1000 data points per hour, the available information is abundant, making the interpretation difficult and time-consuming. This data flood leads to a decrease in time for evidence-based, patient-centered care. Information systems, such as patient data management systems (PDMSs), are increasingly used at ICUs. However, they often create new challenges arising from the increasing documentation burden.

OBJECTIVE

New concepts, such as artificial intelligence (AI)-based assistant systems, are hence introduced to the workflow to cope with these challenges. However, there is a lack of standardized, published metrics in order to compare the various data input and management systems in the ICU setting. The objective of this study is to compare established documentation and retrieval processes with newer methods, such as PDMSs and voice information and documentation systems (VIDSs).

METHODS

In this crossover study, we compare traditional, paper-based documentation systems with PDMSs and newer AI-based VIDSs in terms of performance (required time), accuracy, mental workload, and user experience in an intensive care setting. Performance is assessed on a set of 6 standardized, typical ICU tasks, ranging from documentation to medical interpretation.

RESULTS

A total of 60 ICU-experienced medical professionals participated in the study. The VIDS showed a statistically significant advantage compared to the other 2 systems. The tasks were completed significantly faster with the VIDS than with the PDMS (1-tailed t59=12.48; Cohen d=1.61; P<.001) or paper documentation (t59=20.41; Cohen d=2.63; P<.001). Significantly fewer errors were made with VIDS than with the PDMS (t59=3.45; Cohen d=0.45; P=.03) and paper-based documentation (t59=11.2; Cohen d=1.45; P<.001). The analysis of the mental workload of VIDS and PDMS showed no statistically significant difference (P=.06). However, the analysis of subjective user perception showed a statistically significant perceived benefit of the VIDS compared to the PDMS (P<.001) and paper documentation (P<.001).

CONCLUSIONS

The results of this study show that the VIDS reduced error rate, documentation time, and mental workload regarding the set of 6 standardized typical ICU tasks. In conclusion, this indicates that AI-based systems such as the VIDS tested in this study have the potential to reduce this workload and improve evidence-based and safe patient care.

Dealing with information overload: a comprehensive review

Information overload is a problem that is being exacerbated by the ongoing digitalization of the world of work and the growing use of information and communication technologies. Therefore, the aim of this systematic literature review is to provide an insight into existing measures for prevention and intervention related to information overload. The methodological approach of the systematic review is based on the PRISMA standards. A keyword search in three interdisciplinary scientific databases and other more practice-oriented databases resulted in the identification of 87 studies, field reports, and conceptual papers that were included in the review. The results show that a considerable number of papers have been published on interventions on the behavioral prevention level. At the level of structural prevention, there are also many proposals on how to design work to reduce information overload. A further distinction can be made between work design approaches at the level of information and communication technology and at the level of teamwork and organizational regulations. Although the identified studies cover a wide range of possible interventions and design approaches to address information overload, the strength of the evidence from these studies is mixed.

Evaluation of Digital Health Strategy to Support Clinician-Led Critically Ill Patient Population Management: A Randomized Crossover Study

To investigate whether a novel acute care multipatient viewer (AMP), created with an understanding of clinician information and process requirements, could reduce time to clinical decision-making among clinicians caring for populations of acutely ill patients compared with a widely used commercial electronic medical record (EMR).

DESIGN

Single center randomized crossover study.

SETTING

Quaternary care academic hospital.

SUBJECTS

Attending and in-training critical care physicians, and advanced practice providers.

INTERVENTIONS

AMP.

MEASUREMENTS AND MAIN RESULTS

We compared ICU clinician performance in structured clinical task completion using two electronic environments-the standard commercial EMR (Epic) versus the novel AMP in addition to Epic. Twenty subjects (10 pairs of clinicians) participated in the study. During the study session, each participant completed the tasks on two ICUs (7-10 beds each) and eight individual patients. The adjusted time for assessment of the entire ICU and the adjusted total time to task completion were significantly lower using AMP versus standard commercial EMR (-6.11; 95% CI, -7.91 to -4.30 min and -5.38; 95% CI, -7.56 to -3.20 min, respectively; p < 0.001). The adjusted time for assessment of individual patients was similar using both the EMR and AMP (0.73; 95% CI, -0.09 to 1.54 min; p = 0.078). AMP was associated with a significantly lower adjusted task load (National Aeronautics and Space Administration-Task Load Index) among clinicians performing the task versus the standard EMR (22.6; 95% CI, -32.7 to -12.4 points; p < 0.001). There was no statistically significant difference in adjusted total errors when comparing the two environments (0.68; 95% CI, 0.36-1.30; p = 0.078).

CONCLUSIONS

When compared with the standard EMR, AMP significantly reduced time to assessment of an entire ICU, total time to clinical task completion, and clinician task load. Additional research is needed to assess the clinicians' performance while using AMP in the live ICU setting.

Using electronic health record audit log data for research: insights from early efforts

Electronic health record audit logs capture a time-sequenced record of clinician activities while using the system. Audit log data therefore facilitate unobtrusive measurement at scale of clinical work activities and workflow as well as derivative, behavioral proxies (eg, teamwork). Given its considerable research potential, studies leveraging these data have burgeoned. As the field has matured, the challenges of using the data to answer significant research questions have come into focus. In this Perspective, we draw on our research experiences and insights from the broader audit log literature to advance audit log research. Specifically, we make 2 complementary recommendations that would facilitate substantial progress toward audit log-based measures that are: (1) transparent and validated, (2) standardized to allow for multisite studies, (3) sensitive to meaningful variability, (4) broader in scope to capture key aspects of clinical work including teamwork and coordination, and (5) linked to patient and clinical outcomes.

Patient Safety Issues From Information Overload in Electronic Medical Records

BACKGROUND AND OBJECTIVE

Electronic health records (EHRs) have become ubiquitous in medicine and continue to grow in informational content. Little has been documented regarding patient safety from the resultant information overload. The objective of this literature review is to better understand how information overload in EHR affects patient safety.

METHODS

A literature search was performed using the Transparent Reporting of Systematic Reviews and Meta-Analyses standards for literature review. PubMed and Web of Science were searched and articles selected that were relevant to EHR information overload based on keywords.

RESULTS

The literature search yielded 28 articles meeting the criteria for the study. Information overload was found to increase physician cognitive load and error rates in clinical simulations. Overabundance of clinically irrelevant information, poor data display, and excessive alerting were consistently identified as issues that may lead to information overload.

CONCLUSIONS

Information overload in EHRs may result in higher error rates and negatively impact patient safety. Further studies are necessary to define the role of EHR in adverse patient safety events and to determine methods to mitigate these errors. Changes focused on the usability of EHR should be considered with the end user (physician) in mind. Federal agencies have a role to play in encouraging faster adoption of improved EHR interfaces.

The Reduction in Medical Errors on Implementing an Intensive Care Information System in a Setting Where a Hospital Electronic Medical Record System is Already in Use: Retrospective Analysis

Background

Although the various advantages of clinical information systems in intensive care units (ICUs), such as intensive care information systems (ICISs), have been reported, their role in preventing medical errors remains unclear.

Objective

This study aimed to investigate the changes in the incidence and type of errors in the ICU before and after ICIS implementation in a setting where a hospital electronic medical record system is already in use.

Methods

An ICIS was introduced to the general ICU of a university hospital. After a step-by-step implementation lasting 3 months, the ICIS was used for all patients starting from April 2019. We performed a retrospective analysis of the errors in the ICU during the 6-month period before and after ICIS implementation by using data from an incident reporting system, and the number, incidence rate, type, and patient outcome level of errors were determined.

Results

From April 2018 to September 2018, 755 patients were admitted to the ICU, and 719 patients were admitted from April 2019 to September 2019. The number of errors was 153 in the 2018 study period and 71 in the 2019 study period. The error incidence rates in 2018 and 2019 were 54.1 (95% CI 45.9-63.4) and 27.3 (95% CI 21.3-34.4) events per 1000 patient-days, respectively (P<.001). During both periods, there were no significant changes in the composition of the types of errors (P=.16), and the most common type of error was medication error.

Conclusions

ICIS implementation was temporally associated with a 50% reduction in the number and incidence rate of errors in the ICU. Although the most common type of error was medication error in both study periods, ICIS implementation significantly reduced the number and incidence rate of medication errors.

Trial Registration

University Hospital Medical Information Network Clinical Trials Registry UMIN000041471; https://center6.umin.ac.jp/cgi-open-bin/ctr_e/ctr_view.cgi?recptno=R000047345

Clinical Informatics and Quality Improvement in the Pediatric Intensive Care Unit

Clinical informatics can support quality improvement and patient safety in the pediatric intensive care unit (PICU) in several ways including data extraction, analysis, and decision support enabled by electronic health records (EHRs), and databases and registries. Clinical decision support (CDS), embedded in EHRs, now an integral part of the workflow in the PICU, includes several tools and is increasingly leveraging artificial intelligence (AI). Understanding the opportunities and challenges can improve the engagement of clinicians with the design, validation, and implementation of CDS, improve satisfaction with CDS, and improve patient safety, care quality, and value.

[Artificial Intelligence: Challenges and Applications in Intensive Care Medicine]

The high workload in intensive care medicine arises from the exponential growth of medical knowledge, the flood of data generated by the permanent and intensive monitoring of intensive care patients, and the documentation burden. Artificial intelligence (AI) is predicted to have a great impact on ICU work in the near future as it will be applicable in many areas of critical care medicine. These applications include documentation through speech recognition, predictions for decision support, algorithms for parameter optimisation and the development of personalised intensive care medicine. AI-based decision support systems can augment human therapy decisions. Primarily through machine learning, a sub-discipline of AI, self-adaptive algorithms can learn to recognise patterns and make predictions. For actual use in clinical settings, the explainability of such systems is a prerequisite. Intensive care staff spends a large amount of their working hours on documentation, which has increased up to 50% of work time with the introduction of PDMS. Speech recognition has the potential to reduce this documentation burden. It is not yet precise enough to be usable in the clinic. The application of AI in medicine, with the help of large data sets, promises to identify diagnoses more quickly, develop individualised, precise treatments, support therapeutic decisions, use resources with maximum effectiveness and thus optimise the patient experience in the near future.

Big data and predictive analytics in neurocritical care

PURPOSE OF REVIEW

To describe predictive data and workflow in the intensive care unit when managing neurologically ill patients.

RECENT FINDINGS

In the era of Big Data in medicine, intensive critical care units are data-rich environments. Neurocritical care adds another layer of data with advanced multimodal monitoring to prevent secondary brain injury from ischemia, tissue hypoxia, and a cascade of ongoing metabolic events. A step closer toward personalized medicine is the application of multimodal monitoring of cerebral hemodynamics, bran oxygenation, brain metabolism, and electrophysiologic indices, all of which have complex and dynamic interactions. These data are acquired and visualized using different tools and monitors facing multiple challenges toward the goal of the optimal decision support system. In this review, we highlight some of the predictive data used to diagnose, treat, and prognosticate the neurologically ill patients. We describe information management in neurocritical care units including data acquisition, wrangling, analysis, and visualization.

The Next-Generation Electronic Health Record in the ICU: A Focus on User-Technology Interface to Optimize Patient Safety and Quality

As a knowledge-based field of medicine, critical care medicine has benefited from the use of the electronic health records (EHRs) in daily practice, as intensive care unit (ICU) patients generate thousands of pieces of clinical data each day.1 ICU teams must review, interpret, and take action on these data points when managing multiple patients in a time-constrained environment. The increasing number of available data facts to be processed by ICU clinicians for decision-making surpasses human cognitive capacity. ICU physicians described the current display and representation of patient data in the EHR as suboptimum. Performance dashboards are an information delivery system that display the most important information about performance objectives to ICU directors, allowing them to monitor and manage their ICU performance more effectively. The development of visualization dashboards that monitor ICU performance will still need to adhere to usability principles such as Jakob Nielsen's heuristics. The goal of improving EHR interfaces will directly enhance provider well-being, patient outcomes, and quality of care.

Dual Process Theory and Cognitive Load: How Intensivists Make Diagnoses

Improving clinical reasoning in order to reduce frequency of diagnostic errors is an important area of study. The authors discuss dual process theory as a model of clinical reasoning and explore the role that cognitive load plays in clinical reasoning in the intensive care unit environment.

Pilot Testing of Simulation in the Evaluation of a Novel, Rapidly Deployable Electronic Health Record for use in Disaster Intensive Care

OBJECTIVES

The SARS-CoV-2 pandemic has highlighted the need for rapid creation and management of ICU field hospitals with effective remote monitoring which is dependent on the rapid deployment and integration of an Electronic Health Record (EHR). We describe the use of simulation to evaluate a rapidly scalable hub-and-spoke model for EHR deployment and monitoring using asynchronous training.

METHODS

We adapted existing commercial EHR products to serve as the point of entry from a simulated hospital and a separate system for tele-ICU support and monitoring of the interfaced data. To train our users we created a modular video-based curriculum to facilitate asynchronous training. Effectiveness of the curriculum was assessed through completion of common ICU documentation tasks in a high-fidelity simulation. Additional endpoints include assessment of EHR navigation, user satisfaction (Net Promoter), system usability (System Usability Scale-SUS), and cognitive load (NASA-TLX).

RESULTS

A total of 5 participants achieved a 100% task completion on all domains except ventilator data (91%). Systems demonstrated high degrees of satisfaction (Net Promoter = 65.2), acceptable usability (SUS = 66.5), and acceptable cognitive load (NASA-TLX = 41.5); with higher levels of cognitive load correlating with the number of screens employed.

CONCLUSIONS

Clinical usability of a comprehensive and rapidly deployable EHR was acceptable in an intensive care simulation which was preceded by < 1 hour of video education about the EHR. This model should be considered in plans for integrated clinical response with remote and accessory facilities.

Complexity of Documentation Needs for Children With Medical Complexity: Implications for Hospital Providers

OBJECTIVES

Care coordination is a core component of pediatric complex care programs (CCPs) supporting children with medical complexity (CMC) and their families. In this study, we aim to describe the purpose and characteristics of clinical care notes used within a pediatric CCP.

METHODS

We conducted observations of provider-family interactions during CCP clinic visits and 5 focus groups with members of the CCP. Focus groups were recorded and transcribed. Field observation notes and focus group transcripts were subjected to qualitative content analyses.

RESULTS

Four major themes help characterize clinical care notes: (1) Diversity of note types and functions: program staff author and use a number of unique note types shared across multiple stakeholders, including clinicians, families, and payers. (2) motivations for care note generation are different and explain how, why, and where they are created. (3) Program staff roles and configuration vary in relation to care note creation and use. (4) Sources of information for creating and updating notes are also diverse. Given the disparate information sources, integrating and maintaining up-to-date information for the child is challenging. To minimize information gaps, program staff devised unique but resource-intensive strategies, such as accompanying families during specialty clinic visits or visiting them inpatient.

CONCLUSIONS

CMC have complex documentation needs demonstrated by a variety of professional roles, care settings, and stakeholders involved in the generation and use of notes. Multiple opportunities exist to redesign and streamline the existing notes to support the cognitive work of clinicians providing care for CMC.

Never Enough Time: Mixed Methods Study Identifies Drivers of Temporal Demand That Contribute to Burnout Among Physicians Who Care for Pediatric Hematology-Oncology Patients

PURPOSE

Burnout is a syndrome of emotional exhaustion, depersonalization, and reduced personal accomplishment because of chronic occupational stress. Approximately one third of pediatric hematology-oncology physicians experience burnout. The goal of this mixed methods study was to determine the prevalence and drivers of burnout among physicians caring for pediatric hematology-oncology patients at our institution.

MATERIALS AND METHODS

This mixed methods, cross-sectional study was conducted at a large academic cancer center. Validated survey instruments were used to measure burnout, job demands, experience with patient safety events, and workplace culture. Quantitative data informed development of a semistructured interview guide, and physicians were randomly selected to participate in individual interviews. Interviews were transcribed and analyzed via content analysis based on a priori codes.

RESULTS

The survey was distributed to 132 physicians, and 53 complete responses were received (response rate 40%). Of the 53 respondents, 15 (28%) met criteria for burnout. Experiencing burnout was associated with increased temporal demand. Twenty-six interviews were conducted. Qualitative themes revealed that frequent meetings, insufficient support staff, and workflow interruptions were key drivers of temporal demand and that temporal demand contributed to burnout through emotional exhaustion and reduced personal accomplishment.

CONCLUSION

Nearly one-third of participating physicians met criteria for burnout, and burnout was associated with increased temporal demand. Qualitative interviews identified specific drivers of temporal demand and burnout, which can be targeted for intervention. This methodology can be easily adapted for broad use and may represent an effective strategy for identifying and mitigating institution-specific drivers of burnout.

Electronic health record usability and workload changes over time for provider and nursing staff following transition to new EHR

The ubiquity of EHRs in healthcare means that small EHR inefficiencies can have a major impact on clinician workload. We conducted a sequential explanatory mixed methods study to: 1) identify EHR-associated workload and usability effects for clinicians following an EHR change over time, 2) determine workload and usability differences for providers (MD and Advance Practice Nurses) versus nurses (RNs and MAs), 3) determine if usability predicts workload, 4) identify potential sources of EHR design flaws. Workload (NASA-Task Load Index) and usability (System Usability Scale) measures were administered pre, 6-8 month and 30-32 months post-implementation. We found significant increase in perceived workload post-implementation that persisted for 2.5 years (p < .001). The workload increase was associated with usability ratings, which in turn may relate to EHR interface design violations identified by a heuristic evaluation. Our findings suggest further innovation and attention to interface design flaws are needed to improve EHR usability and reduce clinician workload.

Impact of a problem-oriented view on clinical data retrieval

OBJECTIVE

The electronic health record (EHR) data deluge makes data retrieval more difficult, escalating cognitive load and exacerbating clinician burnout. New auto-summarization techniques are needed. The study goal was to determine if problem-oriented view (POV) auto-summaries improve data retrieval workflows. We hypothesized that POV users would perform tasks faster, make fewer errors, be more satisfied with EHR use, and experience less cognitive load as compared with users of the standard view (SV).

METHODS

Simple data retrieval tasks were performed in an EHR simulation environment. A randomized block design was used. In the control group (SV), subjects retrieved lab results and medications by navigating to corresponding sections of the electronic record. In the intervention group (POV), subjects clicked on the name of the problem and immediately saw lab results and medications relevant to that problem.

RESULTS

With POV, mean completion time was faster (173 seconds for POV vs 205 seconds for SV; P < .0001), the error rate was lower (3.4% for POV vs 7.7% for SV; P = .0010), user satisfaction was greater (System Usability Scale score 58.5 for POV vs 41.3 for SV; P < .0001), and cognitive task load was less (NASA Task Load Index score 0.72 for POV vs 0.99 for SV; P < .0001).

DISCUSSION

The study demonstrates that using a problem-based auto-summary has a positive impact on 4 aspects of EHR data retrieval, including cognitive load.

CONCLUSION

EHRs have brought on a data deluge, with increased cognitive load and physician burnout. To mitigate these increases, further development and implementation of auto-summarization functionality and the requisite knowledge base are needed.

Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects

Electronic health records (EHR) use is often considered a significant contributor to clinician burnout. Informatics researchers often measure clinical workload using EHR-derived audit logs and use it for quantifying the contribution of EHR use to clinician burnout. However, translating clinician workload measured using EHR-based audit logs into a meaningful burnout metric requires an alignment with the conceptual and theoretical principles of burnout. In this perspective, we describe a systems-oriented conceptual framework to achieve such an alignment and describe the pragmatic realization of this conceptual framework using 3 key dimensions: standardizing the measurement of EHR-based clinical work activities, implementing complementary measurements, and using appropriate instruments to assess burnout and its downstream outcomes. We discuss how careful considerations of such dimensions can help in augmenting EHR-based audit logs to measure factors that contribute to burnout and for meaningfully assessing downstream patient safety outcomes.

Improving Clinician Decisions and Communication in Critical Care Using Novel Information Technology

INTRODUCTION

The electronic medical record (EMR) is presumed to support clinician decisions by documenting and retrieving patient information. Research shows that the EMR variably affects patient care and clinical decision making. The way information is presented likely has a significant impact on this variability. Well-designed representations of salient information can make a task easier by integrating information in useful patterns that clinicians use to make improved clinical judgments and decisions. Using Cognitive Systems Engineering methods, our research team developed a novel health information technology (NHIT) that interfaces with the EMR to display salient clinical information and enabled communication with a dedicated text-messaging feature. The software allows clinicians to customize displays according to their role and information needs. Here we present results of usability and validation assessments of the NHIT.

MATERIALS AND METHODS

Our subjects were physicians, nurses, respiratory therapists, and physician trainees. Two arms of this study were conducted, a usability assessment and then a validation assessment. The usability assessment was a computer-based simulation using deceased patient data. After a brief five-minute orientation, the usability assessment measured individual clinician performance of typical tasks in two clinical scenarios using the NHIT. The clinical scenarios included patient admission to the unit and patient readiness for surgery. We evaluated clinician perspective about the NHIT after completing tasks using 7-point Likert scale surveys. In the usability assessment, the primary outcome was participant perceptions about the system's ease of use compared to the legacy system.A subsequent cross-over, validation assessment compared performance of two clinical teams during simulated care scenarios: one using only the legacy IT system and one using the NHIT in addition to the legacy IT system. We oriented both teams to the NHIT during a 1-hour session on the night before the first scenario. Scenarios were conducted using high-fidelity simulation in a real burn intensive care unit room. We used observations, task completion times, semi-structured interviews, and surveys to compare user decisions and perceptions about their performance. The primary outcome for the validation assessment was time to reach accurate (correct) decision points.

RESULTS

During the usability assessment, clinicians were able to complete all tasks requested. Clinicians reported the NHIT was easier to use and the novel information display allowed for easier data interpretation compared to subject recollection of the legacy EMR.In the validation assessment, a more junior team of clinicians using the NHIT arrived at accurate diagnoses and decision points at similar times as a more experienced team. Both teams noted improved communication between team members when using the NHIT and overall rated the NHIT as easier to use than the legacy EMR, especially with respect to finding information.

CONCLUSIONS

The primary findings of these assessments are that clinicians found the NHIT easy to use despite minimal training and experience and that it did not degrade clinician efficiency or decision-making accuracy. These findings are in contrast to common user experiences when introduced to new EMRs in clinical practice.

The importance of theory and method: A brief reflection on an innovative program of research examining how situational factors influence physicians' clinical reasoning

Clinical reasoning, a complex process that involves gathering and synthesizing information to make diagnostic and treatment decisions, is a topic researchers frequently study to mitigate errors. Scientific reasoning has several similarities with clinical reasoning, including the need to generate hypotheses; observe, gather, and interpret evidence; engage in the process of elimination; draw conclusions; and refine and test new hypotheses. However, researchers have only recently begun to take into consideration the role that situational factors (also known as contextual factors), such as language barriers or the lack of diagnostic test results, can play in diagnostic error. Additionally, questions remain about the best ways to teach these complex processes.

Behavior change techniques to promote healthcare professionals' eHealth competency: A systematic review of interventions

INTRODUCTION

The use of eHealth is rapidly -->increasing; however, many healthcare professionals have insufficient eHealth competency. Consequently, interventions addressing eHealth competency might be useful in fostering the effective use of eHealth.

OBJECTIVE

Our systematic review aimed to identify and evaluate the behavior change techniques applied in interventions to promote healthcare professionals' eHealth competency.

METHODS

We conducted a systematic literature review following the Joanna Briggs Institute's Manual for Evidence Synthesis. Published quantitative studies were identified through screening PubMed, Embase, and CINAHL. Two reviewers independently performed full-text and quality assessment. Eligible interventions were targeted to any healthcare professional and aimed at promoting eHealth capability or motivation. We synthesized the interventions narratively using the Behavior Change Technique Taxonomy v1 and the COM-B model.

RESULTS

This review included 32 studies reporting 34 heterogeneous interventions that incorporated 29 different behavior change techniques. The interventions were most likely to improve the capability to use eHealth and less likely to enhance motivation toward using eHealth. The promising techniques to promote both capability and motivation were action planning and participatory approach. Information about colleagues' approval, emotional social support, monitoring emotions, restructuring or adding objects to the environment, and credible source are techniques worth further investigation.

CONCLUSIONS

We found that interventions tended to focus on promoting capability, although motivation would be as crucial for competent eHealth performance. Our findings indicated that empathy, encouragement, and user-centered changes in the work environment could improve eHealth competency as a whole. Evidence-based techniques should be favored in the development of interventions, and further intervention research should focus on nurses and multifaceted competency required for using different eHealth systems and devices.

Modeling physician variability to prioritize relevant medical record information

OBJECTIVE

Patient information can be retrieved more efficiently in electronic medical record (EMR) systems by using machine learning models that predict which information a physician will seek in a clinical context. However, information-seeking behavior varies across EMR users. To explicitly account for this variability, we derived hierarchical models and compared their performance to nonhierarchical models in identifying relevant patient information in intensive care unit (ICU) cases.

MATERIALS AND METHODS

Critical care physicians reviewed ICU patient cases and selected data items relevant for presenting at morning rounds. Using patient EMR data as predictors, we derived hierarchical logistic regression (HLR) and standard logistic regression (LR) models to predict their relevance.

RESULTS

In 73 pairs of HLR and LR models, the HLR models achieved an area under the receiver operating characteristic curve of 0.81, 95% confidence interval (CI) [0.80-0.82], which was statistically significantly higher than that of LR models (0.75, 95% CI [0.74-0.76]). Further, the HLR models achieved statistically significantly lower expected calibration error (0.07, 95% CI [0.06-0.08]) than LR models (0.16, 95% CI [0.14-0.17]).

DISCUSSION

The physician reviewers demonstrated variability in selecting relevant data. Our results show that HLR models perform significantly better than LR models with respect to both discrimination and calibration. This is likely due to explicitly modeling physician-related variability.

CONCLUSION

Hierarchical models can yield better performance when there is physician-related variability as in the case of identifying relevant information in the EMR.

User-Centered Clinical Display Design Issues for Inpatient Providers

BACKGROUND

Suboptimal information display in electronic health records (EHRs) is a notorious pain point for users. Designing an effective display is difficult, due in part to the complex and varied nature of clinical practice.

OBJECTIVE

This article aims to understand the goals, constraints, frustrations, and mental models of inpatient medical providers when accessing EHR data, to better inform the display of clinical information.

METHODS

A multidisciplinary ethnographic study of inpatient medical providers.

RESULTS

Our participants' primary goal was usually to assemble a clinical picture around a given question, under the constraints of time pressure and incomplete information. To do so, they tend to use a mental model of multiple layers of abstraction when thinking of patients and disease; they prefer immediate pattern recognition strategies for answering clinical questions, with breadth-first or depth-first search strategies used subsequently if needed; and they are sensitive to data relevance, completeness, and reliability when reading a record.

CONCLUSION

These results conflict with the ubiquitous display design practice of separating data by type (test results, medications, notes, etc.), a mismatch that is known to encumber efficient mental processing by increasing both navigation burden and memory demands on users. A popular and obvious solution is to select or filter the data to display exactly what is presumed to be relevant to the clinical question, but this solution is both brittle and mistrusted by users. A less brittle approach that is more aligned with our users' mental model could use abstraction to summarize details instead of filtering to hide data. An abstraction-based approach could allow clinicians to more easily assemble a clinical picture, to use immediate pattern recognition strategies, and to adjust the level of displayed detail to their particular needs. It could also help the user notice unanticipated patterns and to fluidly shift attention as understanding evolves.

Going digital: a checklist in preparing for hospital-wide electronic medical record implementation and digital transformation

Objective In an era of rapid digitisation of Australian hospitals, practical guidance is needed in how to successfully implement electronic medical records (EMRs) as both a technical innovation and a major transformative change in clinical care. The aim of the present study was to develop a checklist that clearly and comprehensively defines the steps that best prepare hospitals for EMR implementation and digital transformation. Methods The checklist was developed using a formal methodological framework comprised of: literature reviews of relevant issues; an interactive workshop involving a multidisciplinary group of digital leads from Queensland hospitals; a draft document based on literature and workshop proceedings; and a review and feedback from senior clinical leads. Results The final checklist comprised 19 questions, 13 related to EMR implementation and six to digital transformation. Questions related to the former included organisational considerations (leadership, governance, change leaders, implementation plan), technical considerations (vendor choice, information technology and project management teams, system and hardware alignment with clinician workflows, interoperability with legacy systems) and training (user training, post-go-live contingency plans, roll-out sequence, staff support at point of care). Questions related to digital transformation included cultural considerations (clinically focused vision statement and communication strategy, readiness for change surveys), management of digital disruption syndromes and plans for further improvement in patient care (post-go-live optimisation of digital system, quality and benefit evaluation, ongoing digital innovation). Conclusion This evidence-based, field-tested checklist provides guidance to hospitals planning EMR implementation and separates readiness for EMR from readiness for digital transformation. What is known about the topic? Many hospitals throughout Australia have implemented, or are planning to implement, hospital wide electronic medical records (EMRs) with varying degrees of functionality. Few hospitals have implemented a complete end-to-end digital system with the ability to bring about major transformation in clinical care. Although the many challenges in implementing EMRs have been well documented, they have not been incorporated into an evidence-based, field-tested checklist that can practically assist hospitals in preparing for EMR implementation as both a technical innovation and a vehicle for major digital transformation of care. What does this paper add? This paper outlines a 19-question checklist that was developed using a formal methodological framework comprising literature review of relevant issues, proceedings from an interactive workshop involving a multidisciplinary group of digital leads from hospitals throughout Queensland, including three hospitals undertaking EMR implementation and one hospital with complete end-to-end EMR, and review of a draft checklist by senior clinical leads within a statewide digital healthcare improvement network. The checklist distinguishes between issues pertaining to EMR as a technical innovation and EMR as a vehicle for digital transformation of patient care. What are the implications for practitioners? Successful implementation of a hospital-wide EMR requires senior managers, clinical leads, information technology teams and project management teams to fully address key operational and strategic issues. Using an issues checklist may help prevent any one issue being inadvertently overlooked or underemphasised in the planning and implementation stages, and ensure the EMR is fully adopted and optimally used by clinician users in an ongoing digital transformation of care.

Graphical Presentations of Clinical Data in a Learning Electronic Medical Record

BACKGROUND

Complex electronic medical records (EMRs) presenting large amounts of data create risks of cognitive overload. We are designing a Learning EMR (LEMR) system that utilizes models of intensive care unit (ICU) physicians' data access patterns to identify and then highlight the most relevant data for each patient.

OBJECTIVES

We used insights from literature and feedback from potential users to inform the design of an EMR display capable of highlighting relevant information.

METHODS

We used a review of relevant literature to guide the design of preliminary paper prototypes of the LEMR user interface. We observed five ICU physicians using their current EMR systems in preparation for morning rounds. Participants were interviewed and asked to explain their interactions and challenges with the EMR systems. Findings informed the revision of our prototypes. Finally, we conducted a focus group with five ICU physicians to elicit feedback on our designs and to generate ideas for our final prototypes using participatory design methods.

RESULTS

Participating physicians expressed support for the LEMR system. Identified design requirements included the display of data essential for every patient together with diagnosis-specific data and new or significantly changed information. Respondents expressed preferences for fishbones to organize labs, mouseovers to access additional details, and unobtrusive alerts minimizing color-coding. To address the concern about possible physician overreliance on highlighting, participants suggested that non-highlighted data should remain accessible. Study findings led to revised prototypes, which will inform the development of a functional user interface.

CONCLUSION

In the feedback we received, physicians supported pursuing the concept of a LEMR system. By introducing novel ways to support physicians' cognitive abilities, such a system has the potential to enhance physician EMR use and lead to better patient outcomes. Future plans include laboratory studies of both the utility of the proposed designs on decision-making, and the possible impact of any automation bias.

Derivation and validation of a computable phenotype for acute decompensated heart failure in hospitalized patients

Background

With higher adoption of electronic health records at health-care centers, electronic search algorithms (computable phenotype) for identifying acute decompensated heart failure (ADHF) among hospitalized patients can be an invaluable tool to enhance data abstraction accuracy and efficacy in order to improve clinical research accrual and patient centered outcomes. We aimed to derive and validate a computable phenotype for ADHF in hospitalized patients.

Methods

We screened 256, 443 eligible (age > 18 years and with prior research authorization) individuals who were admitted to Mayo Clinic Hospital in Rochester, MN, from January 1, 2006, through December 31, 2014. Using a randomly selected derivation cohort of 938 patients, several iterations of a free-text electronic search were developed and refined. The computable phenotype was subsequently validated in an independent cohort 100 patients. The sensitivity and specificity of the computable phenotype were compared to the gold standard (expert review of charts) and International Classification of Diseases-9 (ICD-9) codes for Acute Heart Failure.

Results

In the derivation cohort, the computable phenotype achieved a sensitivity of 97.5%, and specificity of 100%, whereas ICD-9 codes for Acute Heart Failure achieved a sensitivity of 47.5% and specificity of 96.7%. When all Heart Failure codes (ICD-9) were used, sensitivity and specificity were 97.5 and 86.6%, respectively. In the validation cohort, the sensitivity and specificity of the computable phenotype were 100 and 98.5%. The sensitivity and specificity for the ICD-9 codes (Acute Heart Failure) were 42 and 98.5%. Upon use of all Heart Failure codes (ICD-9), sensitivity and specificity were 96.8 and 91.3%.

Conclusions

Our results suggest that using computable phenotype to ascertain ADHF from the clinical notes contained within the electronic medical record are feasible and reliable. Our computable phenotype outperformed ICD-9 codes for the detection of ADHF.

Assessing the cognitive and work load of an inpatient safety dashboard in the context of opioid management

For health information technology to realize its potential to improve flow, care, and patient safety, applications should be intuitive to use and burden neutral for frontline clinicians. We assessed the impact of a patient safety dashboard on clinician cognitive and work load within a simulated information-seeking task for safe inpatient opioid medication management. Compared to use of an electronic health record for the same task, the dashboard was associated with significantly reduced time on task, mouse clicks, and mouse movement (each p < 0.001), with no significant increases in cognitive load nor task inaccuracy. Cognitive burden was higher for users with less experience, possibly partly attributable to usability issues identified during this study. Findings underscore the importance of assessing the usability, cognitive, and work load analysis during the design and implementation of health information technology applications.

Evaluative Research of Technologies for Prehospital Communication and Coordination: a Systematic Review

Various technologies have been designed and developed to support communication and coordination between the field and hospital during a medical emergency. The usability issues and human factors entailed in these new technologies are important to their application and effectiveness, suggesting the need to examine this information in a systematic review. The systematic review aims to synthesize the user-centered evaluative research of prehospital communication technologies. We conducted a systematic literature search in four databases (Medline, Cochrane, Embase, and Web of Science) for articles published between the years 2000 and 2019. We included articles that evaluated the technologies developed for supporting prehospital communication and collaboration, and were published in English. A total of 918 articles were retrieved and screened, with 17 articles included for in-depth analysis. Two authors conducted independent screens and reviews of the articles using a list of inclusion/exclusion criteria and defined factors. The types of technology of reviewed articles included ambulance-based telemedicine, wearable, handheld, and Internet of Things (IoT) devices. Even though these technologies have demonstrated high levels of user acceptance, the reviewed studies noted a variety of challenges faced by emergency care providers, which were grouped into three categories-technical, usability, and organizational challenges. Our review also highlighted the paucity of evaluative research of prehospital communication technologies and the lack of user engagement throughout system design process. Based on the results, we discuss the importance of adopting user-centered design approaches and accounting for three social-technical factors in designing technologies for time-critical medical settings, including cognitive and physical stressor, workflow, and context. This systematic review presents an overview of key evaluative research of prehospital communication technologies. The paucity of evaluative research in prehospital communication technology and challenges faced in adopting advanced technological solutions in emergency care highlight the need to adopt user-centered design and take into account socio-technical issues at the point of system design.

Association of Health Record Visualizations With Physicians' Cognitive Load When Prioritizing Hospitalized Patients

IMPORTANCE

Current electronic health records (EHRs) contribute to increased physician cognitive workload when completing clinical tasks.

OBJECTIVE

To assess the association of different design features of an EHR-based information visualization tool with the cognitive load of physicians during the clinical prioritization process.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study included a convenience sample of 29 attending physicians at Seattle Children's Hospital, a large tertiary academic pediatric hospital. Data collection took place from August 2017 through October 2017, and analysis occurred from August to October 2018.

EXPOSURE

Physician participants used 3 prototypes with novel visualizations of simulated EHR data that highlighted 1 of 3 key patient characteristics, as follows: (1) acuity, (2) clinical problem list, and (3) clinical change.

MAIN OUTCOMES AND MEASURES

Cognitive workload was measured using the NASA Task Load Index (TLX) scale (range, 1-100, with lower scores indicating lower cognitive workload). Cognitive workload was assessed for the 2 following clinical prioritization tasks: (1) finding information for a specific patient and (2) comparing results among patients for each prototype. Participants ranked 5 hypothetical patients from having the highest to the lowest priority in each design.

RESULTS

A total of 29 physician participants (15 [52%] men; 14 [48%] women; mean [range] age, 43 [35-58] years; mean [range] time in practice, 11 [3-30] years) completed the study. For task 1, the prototype highlighting clinical change was associated with lower median (interquartile range) NASA TLX scores compared with the prototype highlighting acuity (30.3 [15.2-41.6] vs 48.5 [18.7-59.3]; P = .02). For task 2, the prototype highlighting clinical change was associated with lower median (interquartile range) NASA TLX scores compared with the prototype highlighting the clinical problem list (29.1 [16.3-50.8] vs 43.5 [26.6-55.9]; P = .02). The prototype highlighting clinical change had the lowest TLX score in 17 of 29 rankings (59%) for task 1 (χ24 = 24.4; P < .001) and 18 of 29 rankings (62%) for task 2 (χ24 = 17.2; P = .002).

CONCLUSIONS AND RELEVANCE

In this study, well-designed EHR-based information visualizations that highlighted and featured clinically meaningful information patterns significantly reduced physician cognitive workload when prioritizing patient needs.

Data Omission by Physician Trainees on ICU Rounds

OBJECTIVES

Incomplete patient data, either due to difficulty gathering and synthesizing or inappropriate data filtering, can lead clinicians to misdiagnosis and medical error. How completely ICU interprofessional rounding teams appraise the patient data set that informs clinical decision-making is unknown. This study measures how frequently physician trainees omit data from prerounding notes ("artifacts") and verbal presentations during daily rounds.

DESIGN

Observational study.

SETTING

Tertiary academic medical ICU with an established electronic health record and where physician trainees are the primary presenters during daily rounds.

SUBJECTS

Presenters (medical student or resident physician), interprofessional rounding team.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We quantified the amount and types of patient data omitted from photocopies of physician trainees' artifacts and audio recordings of oral ICU rounds presentations when compared with source electronic health record data. An audit of 157 patient presentations including 6,055 data elements across nine domains revealed 100% of presentations contained omissions. Overall, 22.9% of data were missing from artifacts and 42.4% from presentations. The interprofessional team supplemented only 4.1% of additional available data. Frequency of trainee data omission varied by data type and sociotechnical factors. The strongest predictor of trainee verbal omissions was a preceding failure to include the data on the artifact. Passive data gathering via electronic health record macros resulted in extremely complete artifacts but paradoxically predicted greater likelihood of verbal omission when compared with manual notation. Interns verbally omitted the most data, whereas medical students omitted the least.

CONCLUSIONS

In an academic rounding model reliant on trainees to preview and select data for presentation during ICU rounds, verbal appraisal of patient data was highly incomplete. Additional trainee oversight and education, improved electronic health record tools, and novel academic rounding paradigms are needed to address this potential source of medical error.

Patient information organization in the intensive care setting: expert knowledge elicitation with card sorting methods

Introduction

Many electronic health records fail to support information uptake because they impose low-level information organization tasks on users. Clinical concept-oriented views have shown information processing improvements, but the specifics of this organization for critical care are unclear.

Objective

To determine high-level cognitive processes and patient information organization schema in critical care.

Methods

We conducted an open card sort of 29 patient data elements and a modified Delphi card sort of 65 patient data elements. Study participants were 39 clinicians with varied critical care training and experience. We analyzed the open sort with a hierarchical cluster analysis (HCA) and factor analysis (FA). The Delphi sort was split into three initiating groups that resulted in three unique solutions. We compared results between open sort analyses (HCA and FA), between card sorting exercises (open and Delphi), and across the Delphi solutions.

Results

Between the HCA and FA, we observed common constructs including cardiovascular and hemodynamics, infectious disease, medications, neurology, patient overview, respiratory, and vital signs. The more comprehensive Delphi sort solutions also included gastrointestinal, renal, and imaging constructs.

Conclusions

We identified primarily system-based groupings (e.g., cardiovascular, respiratory). Source-based (e.g., medications, laboratory) groups became apparent when participants were asked to sort a longer list of concepts. These results suggest a hybrid approach to information organization, which may combine systems, source, or problem-based groupings, best supports clinicians' mental models. These results can contribute to the design of information displays to better support clinicians' access and interpretation of information for critical care decisions.

Critical care information display approaches and design frameworks: A systematic review and meta-analysis

OBJECTIVE

To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians.

METHODS

We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author.

RESULTS

Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible.

CONCLUSION

Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.

Measuring electronic communication networks in virtual care teams using electronic health records access-log data

OBJECTIVE

To develop methods for measuring electronic communication networks in virtual care teams using electronic health records (EHR) access-log data.

METHODS

For a convenient sample of 100 surgical colorectal cancer patients, we used time-stamped EHR access-log data extracted from an academic medical center's EHR system to construct communication networks among healthcare professionals (HCPs) in each patient's virtual care team. We measured communication linkages between HCPs using the inverse of the average time between access events in which the source HCPs sent information to and the destination HCPs retrieved information from the EHR system. Social network analysis was used to examine and visualize communication network structures, identify principal care teams, and detect meaningful structural differences across networks. We conducted a non-parametric multivariate analysis of variance (MANOVA) to test the association between care teams' communication network structures and patients' cancer stage and site.

RESULTS

The 100 communication networks showed substantial variations in size and structures. Principal care teams, the subset of HCPs who formed the core of the communication networks, had higher proportions of nurses, physicians, and pharmacists and a lower proportion of laboratory medical technologists than the overall networks. The distributions of conditional uniform graph quantiles suggested that our network-construction technique captured meaningful underlying structures that were different from random unstructured networks. MANOVA results found that the networks' topologies were associated with patients' cancer stage and site.

CONCLUSIONS

This study demonstrates that it is feasible to use EHR access-log data to measure and examine communication networks in virtual care teams. The proposed methods captured salient communication patterns in care teams that were associated with patients' clinical differences.

Evaluating the demographic and clinical minimum data sets of Iranian National Electronic Health Record

BACKGROUND

Designing a standard data set is necessary to overcome the dispersion of data among different health information systems. The objective of this study was to evaluate the current demographic and clinical minimum data sets (MDSs) of Iranian National Electronic Health Record (known as SEPAS) and to identify most necessary data elements.

METHODS

Data were collected using a list of current demographic and clinical data of SEPAS and a self-administered questionnaire. All faculty members of six health related fields and the hospital authorities, and IT and HIM administrators of 6 hospitals in Kerman University of Medical Sciences were invited to participate in this study. The content validity of the questionnaire was confirmed by six medical informatics and HIM experts and the reliability was determined by Cronbach's alpha (α =0.95). SPSS v18 was used to generate descriptive statistics.

RESULTS

Survey results indicated that 15 data elements should become mandatory elements of MDS for communicating data to SEPAS. These elements include patient's name, surname, father's name, nationality, cell number, job, residential address, residence place, passport number (for non-Iranian patients), diagnosis date, death time, death place and the unit of the hospital where the patient died. Moreover, participants suggested 33 additional demographic and clinical data elements to be communicated mandatorily to SEPAS.

CONCLUSION

The results of this study showed that the minimum data sets of Iranian national electronic health record needs to be revised. Using the proposed MDSs by this study can improve the quality and efficiency of information and reduce redundancy by adding necessary data and preventing communication of unnecessary data. The method employed in this study can be used for investigating, refining and completing the MDSs of other health information systems.

Association of Data Integration Technologies With Intensive Care Clinician Performance: A Systematic Review and Meta-analysis

IMPORTANCE

Sources of data in the intensive care setting are increasing exponentially, but the benefits of displaying multiparametric, high-frequency data are unknown. Decision making may not benefit from this technology if clinicians remain cognitively overburdened by poorly designed data integration and visualization technologies (DIVTs).

OBJECTIVE

To systematically review and summarize the published evidence on the association of user-centered DIVTs with intensive care clinician performance.

DATA SOURCES

MEDLINE, Embase, Cochrane Central Register of Controlled Trials, PsycINFO, and Web of Science were searched in May 2014 and January 2018.

STUDY SELECTION

Studies had 3 requirements: (1) the study tested a viable DIVT, (2) participants involved were intensive care clinicians, and (3) the study reported quantitative results associated with decision making in an intensive care setting.

DATA EXTRACTION AND SYNTHESIS

Of 252 records screened, 20 studies, published from 2004 to 2016, were included. The human factors framework to assess health technologies was applied to measure study completeness, and the Quality Assessment Instrument was used to assess the quality of the studies. PRISMA guidelines were adapted to conduct the systematic review and meta-analysis.

MAIN OUTCOMES AND MEASURES

Study completeness and quality; clinician performance; physical, mental, and temporal demand; effort; frustration; time to decision; and decision accuracy.

RESULTS

Of the 20 included studies, 16 were experimental studies with 410 intensive care clinician participants and 4 were survey-based studies with 1511 respondents. Scores for study completeness ranged from 27 to 43, with a maximum score of 47, and scores for study quality ranged from 46 to 79, with a maximum score of 90. Of 20 studies, DIVTs were evaluated in clinical settings in 2 studies (10%); time to decision was measured in 14 studies (70%); and decision accuracy was measured in 11 studies (55%). Measures of cognitive workload pooled in the meta-analysis suggested that any DIVT was an improvement over paper-based data in terms of self-reported performance, mental and temporal demand, and effort. With a maximum score of 22, median (IQR) mental demand scores for electronic display were 10 (7-13), tabular display scores were 8 (6.0-11.5), and novel visualization scores were 8 (6-12), compared with 17 (14-19) for paper. The median (IQR) temporal demand scores were also lower for all electronic visualizations compared with paper, with scores of 8 (6-11) for electronic display, 7 (6-11) for tabular and bar displays, 7 (5-11) for novel visualizations, and 16 (14.3-19.0) for paper. The median (IQR) performance scores improved for all electronic visualizations compared with paper (lower score indicates better self-reported performance), with scores of 6 (3-11) for electronic displays, 6 (4-11) for tabular and bar displays, 6 (4-11) for novel visualizations, and 14 (11-16) for paper. Frustration and physical demand domains of cognitive workload did not change, and differences between electronic displays were not significant.

CONCLUSIONS AND RELEVANCE

This review suggests that DIVTs are associated with increased integration and consistency of data. Much work remains to identify which visualizations effectively reduce cognitive workload to enhance decision making based on intensive care data. Standardizing human factors testing by developing a repository of open access benchmarked test protocols, using a set of outcome measures, scenarios, and data sets, may accelerate the design and selection of the most appropriate DIVT.

An attention based deep learning model of clinical events in the intensive care unit

This study trained long short-term memory (LSTM) recurrent neural networks (RNNs) incorporating an attention mechanism to predict daily sepsis, myocardial infarction (MI), and vancomycin antibiotic administration over two week patient ICU courses in the MIMIC-III dataset. These models achieved next-day predictive AUC of 0.876 for sepsis, 0.823 for MI, and 0.833 for vancomycin administration. Attention maps built from these models highlighted those times when input variables most influenced predictions and could provide a degree of interpretability to clinicians. These models appeared to attend to variables that were proxies for clinician decision-making, demonstrating a challenge of using flexible deep learning approaches trained with EHR data to build clinical decision support. While continued development and refinement is needed, we believe that such models could one day prove useful in reducing information overload for ICU physicians by providing needed clinical decision support for a variety of clinically important tasks.

Eye-tracking reveals how observation chart design features affect the detection of patient deterioration: An experimental study

Particular design features intended to improve usability - including graphically displayed observations and integrated colour-based scoring-systems - have been shown to increase the speed and accuracy with which users of hospital observation charts detect abnormal patient observations. We used eye-tracking to evaluate two potential cognitive mechanisms underlying these effects. Novice chart-users completed a series of experimental trials in which they viewed patient data presented on one of three observation chart designs (varied within-subjects), and indicated which observation was abnormal (or that none were). A chart that incorporated both graphically displayed observations and an integrated colour-based scoring-system yielded faster, more accurate responses and fewer, shorter fixations than a graphical chart without a colour-based scoring-system. The latter, in turn, yielded the same advantages over a tabular chart (which incorporated neither design feature). These results suggest that both colour-based scoring-systems and graphically displayed observations improve search efficiency and reduce the cognitive resources required to process vital sign data.

The use of analytic hierarchy process for measuring the complexity of medical diagnosis

Diagnostic complexity is an important contextual factor affecting a variety of medical outcomes. Existing measurements of diagnosis complexity either rely on crude proxies or use fine-grained measures that employ indicators from proprietary data that are not readily available. Hence, the study of this important construct in fields such as medical informatics has been hampered by the difficulty of measuring diagnostic complexity. This article presents a novel approach for conceptualizing and operationalizing diagnostic task complexity as a multi-dimensional construct, which employs the readily available International Classification of Diseases codes from medical encounters in hospitals and uses Analytic Hierarchical Process methodology. We demonstrate the reliability of the proposed approach and show that despite using a relatively simple procedure, it is able to predict readmission rates just as well as (or even better) than some of the sophisticated measures that have been used in recent studies (namely, the LaCE score index).

Quality Improvement and Safety in Pediatric Emergency Medicine

Pediatric emergency medicine quality work continues to focus on the National Academies of Sciences, Engineering, and Medicine's 6 domains of quality, with a need for specific emphasis on equity and patient centeredness. Adopting the principles of high-reliability organizations, pediatric emergency departments should become increasing transparent with benchmarking and collaboration across institutions in order to develop an infrastructure for quality and safety to improve the care of pediatric patients in the emergency department.

Too much or too little? Investigating the usability of high and low data displays of the same electronic medical record

The high data density on electronic medical record screens is touted as a major usability issue. However, it may not be a problem if the data is relevant and well-organized. Our objective was to test this assumption using a comprehensive set of measures that assess the three pillars of usability: efficiency (both physical and cognitive), effectiveness, and satisfaction. Physicians were asked to go through a series of tasks using two versions of the same electronic medical record: one where all the display items were separated into tabs (the original display), and one where important display items were grouped logically in one tab (the redesigned display). Results supported the hypothesis that combining relevant data in organized fashion into a smaller location would improve usability. The findings highlight the role of good display organization to mitigate the effects of high data density, as well as the importance of assessing cognitive load as part of usability studies.

Comparative usability evaluation of consultation order templates in a simulated primary care environment

Communication breakdowns in the referral process negatively impact clinical workflow and patient safety. There is a lack of evidence demonstrating the impact of published design recommendations addressing contributing issues with consultation order templates. This study translated the recommendations into a computer-based prototype and conducted a comparative usability evaluation. With a scenario-based simulation, 30 clinicians (referrers) participated in a within-group, counterbalanced experiment comparing the prototype with their present electronic order entry system. The prototype significantly increased satisfaction (Cohen's d = 1.80, 95% CI [1.19, 2.41], p < .001), and required significantly less mental effort (d = 0.67 [0.14, 1.20], p < .001). Regarding efficiency, the prototype required significantly fewer mouse clicks (mean difference = 29 clicks, p < .001). Although overall task time did not differ significantly (d = -0.05 [-0.56, 0.47]), the prototype significantly quickened identification of the appropriate specialty clinic (mean difference = 12 s, d = 0.98 [0.43, 1.52], p < .001). The experimental evidence demonstrated that clinician-centered interfaces significantly improved system usability during ordering of consultations.