Assessment of Electronic Health Record Search Patterns and Practices by Practitioners in a Large Integrated Health Care System

Data Representation Structure to Support Clinical Decision-Making in the Pediatric Intensive Care Unit: Interview Study and Preliminary Decision Support Interface Design

BACKGROUND

Clinical decision-making is a complex cognitive process that relies on the interpretation of a large variety of data from different sources and involves the use of knowledge bases and scientific recommendations. The representation of clinical data plays a key role in the speed and efficiency of its interpretation. In addition, the increasing use of clinical decision support systems (CDSSs) provides assistance to clinicians in their practice, allowing them to improve patient outcomes. In the pediatric intensive care unit (PICU), clinicians must process high volumes of data and deal with ever-growing workloads. As they use multiple systems daily to assess patients' status and to adjust the health care plan, including electronic health records (EHR), clinical systems (eg, laboratory, imaging and pharmacy), and connected devices (eg, bedside monitors, mechanical ventilators, intravenous pumps, and syringes), clinicians rely mostly on their judgment and ability to trace relevant data for decision-making. In these circumstances, the lack of optimal data structure and adapted visual representation hinder clinician's cognitive processes and clinical decision-making skills.

OBJECTIVE

In this study, we designed a prototype to optimize the representation of clinical data collected from existing sources (eg, EHR, clinical systems, and devices) via a structure that supports the integration of a home-developed CDSS in the PICU. This study was based on analyzing end user needs and their clinical workflow.

METHODS

First, we observed clinical activities in a PICU to secure a better understanding of the workflow in terms of staff tasks and their use of EHR on a typical work shift. Second, we conducted interviews with 11 clinicians from different staff categories (eg, intensivists, fellows, nurses, and nurse practitioners) to compile their needs for decision support. Third, we structured the data to design a prototype that illustrates the proposed representation. We used a brain injury care scenario to validate the relevance of integrated data and the utility of main functionalities in a clinical context. Fourth, we held design meetings with 5 clinicians to present, revise, and adapt the prototype to meet their needs.

RESULTS

We created a structure with 3 levels of abstraction-unit level, patient level, and system level-to optimize clinical data representation and display for efficient patient assessment and to provide a flexible platform to host the internally developed CDSS. Subsequently, we designed a preliminary prototype based on this structure.

CONCLUSIONS

The data representation structure allows prioritizing patients via criticality indicators, assessing their conditions using a personalized dashboard, and monitoring their courses based on the evolution of clinical values. Further research is required to define and model the concepts of criticality, problem recognition, and evolution. Furthermore, feasibility tests will be conducted to ensure user satisfaction.

Training the next generation of delivery science researchers: 10-year experience of a post-doctoral research fellowship program within an integrated care system

Introduction

Learning health systems require a workforce of researchers trained in the methods of identifying and overcoming barriers to effective, evidence-based care. Most existing postdoctoral training programs, such as NIH-funded postdoctoral T32 awards, support basic and epidemiological science with very limited focus on rigorous delivery science methods for improving care. In this report, we present the 10-year experience of developing and implementing a Delivery Science postdoctoral fellowship embedded within an integrated health care delivery system.

Methods

In 2012, the Kaiser Permanente Northern California Division of Research designed and implemented a 2-year postdoctoral Delivery Science Fellowship research training program to foster research expertise in identifying and addressing barriers to evidence-based care within health care delivery systems.

Results

Since 2014, 20 fellows have completed the program. Ten fellows had PhD-level scientific training, and 10 fellows had clinical doctorates (eg, MD, RN/PhD, PharmD). Fellowship alumni have graduated to faculty research positions at academic institutions (9), and research or clinical organizations (4). Seven alumni now hold positions in Kaiser Permanente's clinical operations or medical group (7).

Conclusions

This delivery science fellowship program has succeeded in training graduates to address delivery science problems from both research and operational perspectives. In the next 10 years, additional goals of the program will be to expand its reach (eg, by developing joint research training models in collaboration with clinical fellowships) and strengthen mechanisms to support transition from fellowship to the workforce, especially for researchers from underrepresented groups.

Video Recording Patients for Direct Care Purposes: Systematic Review and Narrative Synthesis of International Empirical Studies and UK Professional Guidance

BACKGROUND

Video recordings of patients may offer advantages to supplement patient assessment and clinical decision-making. However, little is known about the practice of video recording patients for direct care purposes.

OBJECTIVE

We aimed to synthesize empirical studies published internationally to explore the extent to which video recording patients is acceptable and effective in supporting direct care and, for the United Kingdom, to summarize the relevant guidance of professional and regulatory bodies.

METHODS

Five electronic databases (MEDLINE, Embase, APA PsycINFO, CENTRAL, and HMIC) were searched from 2012 to 2022. Eligible studies evaluated an intervention involving video recording of adult patients (≥18 years) to support diagnosis, care, or treatment. All study designs and countries of publication were included. Websites of UK professional and regulatory bodies were searched to identify relevant guidance. The acceptability of video recording patients was evaluated using study recruitment and retention rates and a framework synthesis of patients' and clinical staff's perspectives based on the Theoretical Framework of Acceptability by Sekhon. Clinically relevant measures of impact were extracted and tabulated according to the study design. The framework approach was used to synthesize the reported ethico-legal considerations, and recommendations of professional and regulatory bodies were extracted and tabulated.

RESULTS

Of the 14,221 abstracts screened, 27 studies met the inclusion criteria. Overall, 13 guidance documents were retrieved, of which 7 were retained for review. The views of patients and clinical staff (16 studies) were predominantly positive, although concerns were expressed about privacy, technical considerations, and integrating video recording into clinical workflows; some patients were anxious about their physical appearance. The mean recruitment rate was 68.2% (SD 22.5%; range 34.2%-100%; 12 studies), and the mean retention rate was 73.3% (SD 28.6%; range 16.7%-100%; 17 studies). Regarding effectiveness (10 studies), patients and clinical staff considered video recordings to be valuable in supporting assessment, care, and treatment; in promoting patient engagement; and in enhancing communication and recall of information. Observational studies (n=5) favored video recording, but randomized controlled trials (n=5) did not demonstrate that video recording was superior to the controls. UK guidelines are consistent in their recommendations around consent, privacy, and storage of recordings but lack detailed guidance on how to operationalize these recommendations in clinical practice.

CONCLUSIONS

Video recording patients for direct care purposes appears to be acceptable, despite concerns about privacy, technical considerations, and how to incorporate recording into clinical workflows. Methodological quality prevents firm conclusions from being drawn; therefore, pragmatic trials (particularly in older adult care and the movement disorders field) should evaluate the impact of video recording on diagnosis, treatment monitoring, patient-clinician communication, and patient safety. Professional and regulatory documents should signpost to practical guidance on the implementation of video recording in routine practice.

TRIAL REGISTRATION

PROSPERO CRD42022331825: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=331825.

Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures

OBJECTIVE

The aim of this article is to compare the aims, measures, methods, limitations, and scope of studies that employ vendor-derived and investigator-derived measures of electronic health record (EHR) use, and to assess measure consistency across studies.

MATERIALS AND METHODS

We searched PubMed for articles published between July 2019 and December 2021 that employed measures of EHR use derived from EHR event logs. We coded the aims, measures, methods, limitations, and scope of each article and compared articles employing vendor-derived and investigator-derived measures.

RESULTS

One hundred and two articles met inclusion criteria; 40 employed vendor-derived measures, 61 employed investigator-derived measures, and 1 employed both. Studies employing vendor-derived measures were more likely than those employing investigator-derived measures to observe EHR use only in ambulatory settings (83% vs 48%, P = .002) and only by physicians or advanced practice providers (100% vs 54% of studies, P < .001). Studies employing vendor-derived measures were also more likely to measure durations of EHR use (P < .001 for 6 different activities), but definitions of measures such as time outside scheduled hours varied widely. Eight articles reported measure validation. The reported limitations of vendor-derived measures included measure transparency and availability for certain clinical settings and roles.

DISCUSSION

Vendor-derived measures are increasingly used to study EHR use, but only by certain clinical roles. Although poorly validated and variously defined, both vendor- and investigator-derived measures of EHR time are widely reported.

CONCLUSION

The number of studies using event logs to observe EHR use continues to grow, but with inconsistent measure definitions and significant differences between studies that employ vendor-derived and investigator-derived measures.

Team is brain: leveraging EHR audit log data for new insights into acute care processes

OBJECTIVE

To determine whether novel measures of contextual factors from multi-site electronic health record (EHR) audit log data can explain variation in clinical process outcomes.

MATERIALS AND METHODS

We selected one widely-used process outcome: emergency department (ED)-based team time to deliver tissue plasminogen activator (tPA) to patients with acute ischemic stroke (AIS). We evaluated Epic audit log data (that tracks EHR user-interactions) for 3052 AIS patients aged 18+ who received tPA after presenting to an ED at three Northern California health systems (Stanford Health Care, UCSF Health, and Kaiser Permanente Northern California). Our primary outcome was door-to-needle time (DNT) and we assessed bivariate and multivariate relationships with six audit log-derived measures of treatment team busyness and prior team experience.

RESULTS

Prior team experience was consistently associated with shorter DNT; teams with greater prior experience specifically on AIS cases had shorter DNT (minutes) across all sites: (Site 1: -94.73, 95% CI: -129.53 to 59.92; Site 2: -80.93, 95% CI: -130.43 to 31.43; Site 3: -42.95, 95% CI: -62.73 to 23.17). Teams with greater prior experience across all types of cases also had shorter DNT at two sites: (Site 1: -6.96, 95% CI: -14.56 to 0.65; Site 2: -19.16, 95% CI: -36.15 to 2.16; Site 3: -11.07, 95% CI: -17.39 to 4.74). Team busyness was not consistently associated with DNT across study sites.

CONCLUSIONS

EHR audit log data offers a novel, scalable approach to measure key contextual factors relevant to clinical process outcomes across multiple sites. Audit log-based measures of team experience were associated with better process outcomes for AIS care, suggesting opportunities to study underlying mechanisms and improve care through deliberate training, team-building, and scheduling to maximize team experience.

Value Implications of Sourcing Electronic Health Records: The Role of Physician Practice Integration

Should hospitals source electronic health records (EHR) systems from a single vendor or multiple vendors to deliver high-value care? We study hospitals’ EHR sourcing strategies based on their degree of integration with physician practices and its impact on the value of healthcare delivered. We propose a novel framework to define healthcare value as the extent to which a hospital effectively expends clinical resources to deliver services that improve patient outcomes. Drawing on modular systems and transaction cost economics theories, we propose a moderated-mediation model that explores the pathways through which EHR sourcing strategies can create value in healthcare. We test our research hypotheses on a large, longitudinal sample of U.S. hospitals and observe that hospitals with EHR configurations closer to single sourcing strategies exhibit greater health information sharing compared with hospitals with multisourced EHR systems. Furthermore, we find that hospital-physician practice integration moderates the impact of single sourcing on health information sharing, which in turn, improves value. Specifically, tighter integration between hospitals and physician practices can create greater value if it is aligned with hospitals’ EHR sourcing strategies. As the healthcare industry moves toward value-based payment reform, our findings provide a useful roadmap to practitioners and policy makers to improve the performance of hospitals and healthcare providers.

History: Rajiv Kohli, Senior Editor; Sunil Wattal, Associate Editor.

Funding: I.R. Bardhan thanks the Foster Parker Centennial Professorship and the Dean’s Research Excellence Grant at the McCombs School of Business at UT Austin for generous financial support. C. Bao thanks the Spears Fellowship at Oklahoma State University for financial support.

Supplemental Material: The online appendix is available at https://doi.org/10.1287/isre.2022.1183 .

Evaluating the Impact on Clinical Task Efficiency of a Natural Language Processing Algorithm for Searching Medical Documents: Prospective Crossover Study

Background

Information retrieval (IR) from the free text within electronic health records (EHRs) is time consuming and complex. We hypothesize that natural language processing (NLP)–enhanced search functionality for EHRs can make clinical workflows more efficient and reduce cognitive load for clinicians.

Objective

This study aimed to evaluate the efficacy of 3 levels of search functionality (no search, string search, and NLP-enhanced search) in supporting IR for clinical users from the free text of EHR documents in a simulated clinical environment.

Methods

A clinical environment was simulated by uploading 3 sets of patient notes into an EHR research software application and presenting these alongside 3 corresponding IR tasks. Tasks contained a mixture of multiple-choice and free-text questions. A prospective crossover study design was used, for which 3 groups of evaluators were recruited, which comprised doctors (n=19) and medical students (n=16). Evaluators performed the 3 tasks using each of the search functionalities in an order in accordance with their randomly assigned group. The speed and accuracy of task completion were measured and analyzed, and user perceptions of NLP-enhanced search were reviewed in a feedback survey.

Results

NLP-enhanced search facilitated more accurate task completion than both string search (5.14%; P=.02) and no search (5.13%; P=.08). NLP-enhanced search and string search facilitated similar task speeds, both showing an increase in speed compared to the no search function, by 11.5% (P=.008) and 16.0% (P=.007) respectively. Overall, 93% of evaluators agreed that NLP-enhanced search would make clinical workflows more efficient than string search, with qualitative feedback reporting that NLP-enhanced search reduced cognitive load.

Conclusions

To the best of our knowledge, this study is the largest evaluation to date of different search functionalities for supporting target clinical users in realistic clinical workflows, with a 3-way prospective crossover study design. NLP-enhanced search improved both accuracy and speed of clinical EHR IR tasks compared to browsing clinical notes without search. NLP-enhanced search improved accuracy and reduced the number of searches required for clinical EHR IR tasks compared to direct search term matching.

Opportunities to use electronic health record audit logs to improve cancer care

The rapid adoption of electronic health records (EHRs) has created extensive repositories of digitized data that can be used to inform improvements in care delivery, processes, and patient outcomes. While the clinical data captured in EHRs are widely used for such efforts, EHRs also capture audit log data that reflect how users interact with the EHR to deliver care. Automatically collected audit log data provide a unique opportunity for new insights into EHR user behavior and decision‐making processes. Here, we provide an overview of audit log data and examples that could be used to improve oncology care and outcomes in four domains: diagnostic reasoning and consumption, care team collaboration and communication, patient outcomes and experience, and provider burnout/fatigue. This data source could identify gaps in performance and care, physician uptake of EHR features that enhance decision‐making, and integration of data trends for oncology. Ensuring researchers and oncologists are familiar with the data's potential and developing the data engineering capacity to utilize this rich data source, will expand the breadth of research to improve cancer care.

Evaluating Electronic Health Record Limitations and Time Expenditure in a German Medical Center

OBJECTIVES

This study set out to obtain a general profile of physician time expenditure and electronic health record (EHR) limitations in a large university medical center in Germany. We also aim to illustrate the merit of a tool allowing for easier capture and prioritization of specific clinical needs at the point of care for which the current study will inform development in subsequent work.

METHODS

Nineteen physicians across six different departments participated in this study. Direct clinical observations were conducted with 13 out of 19 physicians for a total of 2,205 minutes, and semistructured interviews were conducted with all participants. During observations, time was measured for larger activity categories (searching information, reading information, documenting information, patient interaction, calling, and others). Semistructured interviews focused on perceived limitations, frustrations, and desired improvements regarding the EHR environment.

RESULTS

Of the observed time, 37.1% was spent interacting with the health records (9.0% searching, 7.7% reading, and 20.5% writing), 28.0% was spent interacting with patients corrected for EHR use (26.9% of time in a patient's presence), 6.8% was spent calling, and 28.1% was spent on other activities. Major themes of discontent were a spread of patient information, high and often repeated documentation burden, poor integration of (new) information into workflow, limits in information exchange, and the impact of such problems on patient interaction. Physicians stated limited means to address such issues at the point of care.

CONCLUSION

In the study hospital, over one-third of physicians' time was spent interacting with the EHR, environment, with many aspects of used systems far from optimal and no convenient way for physicians to address issues as they occur at the point of care. A tool facilitating easier identification and registration of issues, as they occur, may aid in generating a more complete overview of limitations in the EHR environment.

A simple electronic medical record system designed for research

With the extensive deployment of electronic medical record (EMR) systems, EMR usability remains a significant source of frustration to clinicians. There is a significant research need for software that emulates EMR systems and enables investigators to conduct laboratory-based human–computer interaction studies. We developed an open-source software package that implements the display functions of an EMR system. The user interface emphasizes the temporal display of vital signs, medication administrations, and laboratory test results. It is well suited to support research about clinician information-seeking behaviors and adaptive user interfaces in terms of measures that include task accuracy, time to completion, and cognitive load. The Simple EMR System is freely available to the research community and is on GitHub.

Use, Impact, Weaknesses, and Advanced Features of Search Functions for Clinical Use in Electronic Health Records: A Scoping Review

Objective  Although vast amounts of patient information are captured in electronic health records (EHRs), effective clinical use of this information is challenging due to inadequate and inefficient access to it at the point of care. The purpose of this study was to conduct a scoping review of the literature on the use of EHR search functions within a single patient's record in clinical settings to characterize the current state of research on the topic and identify areas for future study.

Methods  We conducted a literature search of four databases to identify articles on within-EHR search functions or the use of EHR search function in the context of clinical tasks. After reviewing titles and abstracts and performing a full-text review of selected articles, we included 17 articles in the analysis. We qualitatively identified themes in those articles and synthesized the literature for each theme.

Results  Based on the 17 articles analyzed, we delineated four themes: (1) how clinicians use search functions, (2) impact of search functions on clinical workflow, (3) weaknesses of current search functions, and (4) advanced search features. Our review found that search functions generally facilitate patient information retrieval by clinicians and are positively received by users. However, existing search functions have weaknesses, such as yielding false negatives and false positives, which can decrease trust in the results, and requiring a high cognitive load to perform an inclusive search of a patient's record.

Conclusion  Despite the widespread adoption of EHRs, only a limited number of articles describe the use of EHR search functions in a clinical setting, despite evidence that they benefit clinician workflow and productivity. Some of the weaknesses of current search functions may be addressed by enhancing EHR search functions with collaborative filtering.

Sepsis Subclasses: A Framework for Development and Interpretation

Sepsis is defined as a dysregulated host response to infection that leads to life-threatening acute organ dysfunction. It afflicts approximately 50 million people worldwide annually and is often deadly, even when evidence-based guidelines are applied promptly. Many randomized trials tested therapies for sepsis over the past 2 decades, but most have not proven beneficial. This may be because sepsis is a heterogeneous syndrome, characterized by a vast set of clinical and biologic features. Combinations of these features, however, may identify previously unrecognized groups, or "subclasses" with different risks of outcome and response to a given treatment. As efforts to identify sepsis subclasses become more common, many unanswered questions and challenges arise. These include: 1) the semantic underpinning of sepsis subclasses, 2) the conceptual goal of subclasses, 3) considerations about study design, data sources, and statistical methods, 4) the role of emerging data types, and 5) how to determine whether subclasses represent "truth." We discuss these challenges and present a framework for the broader study of sepsis subclasses. This framework is intended to aid in the understanding and interpretation of sepsis subclasses, provide a mechanism for explaining subclasses generated by different methodologic approaches, and guide clinicians in how to consider subclasses in bedside care.

Hybrid collaborative filtering methods for recommending search terms to clinicians

With increasing and extensive use of electronic health records (EHR), clinicians are often challenged in retrieving relevant patient information efficiently and effectively to arrive at a diagnosis. While using the search function built into an EHR can be more useful than browsing in a voluminous patient record, it is cumbersome and repetitive to search for the same or similar information on similar patients. To address this challenge, there is a critical need to build effective recommender systems that can recommend search terms to clinicians accurately. In this study, we developed a hybrid collaborative filtering model to recommend search terms for a specific patient to a clinician. The model draws on information from patients' clinical encounters and the searches that were performed during them. To generate recommendations, the model uses search terms which are (1) frequently co-occurring with the ICD codes recorded for the patient and (2) highly relevant to the most recent search terms. In one variation of the model (Hybrid Collaborative Filtering Method for Healthcare, or HCFMH), we use only the most recent ICD codes assigned to the patient, and in the other (Co-occurrence Pattern based HCFMH, or cpHCFMH), all ICD codes. We have conducted comprehensive experiments to evaluate the proposed model. These experiments demonstrate that our model outperforms state-of-the-art baseline methods for top-N search term recommendation on different data sets.

Modeling physician variability to prioritize relevant medical record information

OBJECTIVE

Patient information can be retrieved more efficiently in electronic medical record (EMR) systems by using machine learning models that predict which information a physician will seek in a clinical context. However, information-seeking behavior varies across EMR users. To explicitly account for this variability, we derived hierarchical models and compared their performance to nonhierarchical models in identifying relevant patient information in intensive care unit (ICU) cases.

MATERIALS AND METHODS

Critical care physicians reviewed ICU patient cases and selected data items relevant for presenting at morning rounds. Using patient EMR data as predictors, we derived hierarchical logistic regression (HLR) and standard logistic regression (LR) models to predict their relevance.

RESULTS

In 73 pairs of HLR and LR models, the HLR models achieved an area under the receiver operating characteristic curve of 0.81, 95% confidence interval (CI) [0.80-0.82], which was statistically significantly higher than that of LR models (0.75, 95% CI [0.74-0.76]). Further, the HLR models achieved statistically significantly lower expected calibration error (0.07, 95% CI [0.06-0.08]) than LR models (0.16, 95% CI [0.14-0.17]).

DISCUSSION

The physician reviewers demonstrated variability in selecting relevant data. Our results show that HLR models perform significantly better than LR models with respect to both discrimination and calibration. This is likely due to explicitly modeling physician-related variability.

CONCLUSION

Hierarchical models can yield better performance when there is physician-related variability as in the case of identifying relevant information in the EMR.