Information overload and missed test results in electronic health record-based settings

Impact of Early Direct Patient Notification on Follow-Up Completion for Nonurgent Actionable Incidental Radiologic Findings

PURPOSE

The aim of this study was to evaluate whether early direct patient notification in addition to an existing multistage recommendation-tracking system (Backstop) increases follow-up completion rates for actionable incidental findings (AIFs). Patient attitudes toward early notification were also assessed.

METHODS

This prospective, randomized controlled trial recruited patients with AIFs requiring follow-up being enrolled into the Backstop system. Patients were randomized into four groups: those receiving additional early direct notification in a mailed letter (group 1, similar to Pennsylvania Act 112), by phone (group 2), or in an electronic portal message (group 3) and a control group (group 4) without additional notifications added to the existing Backstop system. Differences in follow-up completion rates among these groups were determined using χ2 tests. Patients were surveyed on binary yes/no and Likert-type scale questions, and descriptive statistics are reported.

RESULTS

Data from 2,548 randomized patients were analyzed for the study, including 593 patients notified by letter, 637 notified by phone, 701 notified by portal, and 617 control patients. Group 3 demonstrated the lowest rate of follow-up completion within 1 month of the follow-up due date at 36.4%, compared with 58.7% for group 1, 60.4% for group 2, and 53.2% for group 4 (P < .0001 for all). Group 2 was the only group to have a significantly higher completion rate than group 4 (P = .014). Patients responded positively regarding early notification and preferred electronic portal communication.

CONCLUSIONS

Early direct notification had a mixed impact on follow-up completion rates on the basis of communication modality but was positively received by patients and may have health care benefits when implemented within a recommendation-tracking system.

Optimizing Best Practice Advisory alerts in electronic medical records with a multi-pronged strategy at a tertiary care hospital in Singapore

Objective

Clinical decision support (CDS) alerts can aid in improving patient care. One CDS functionality is the Best Practice Advisory (BPA) alert notification system, wherein BPA alerts are automated alerts embedded in the hospital's electronic medical records (EMR). However, excessive alerts can change clinician behavior; redundant and repetitive alerts can contribute to alert fatigue. Alerts can be optimized through a multipronged strategy. Our study aims to describe these strategies adopted and evaluate the resultant BPA alert optimization outcomes.

Materials and Methods

This retrospective single-center study was done at Jurong Health Campus. Aggregated, anonymized data on patient demographics and alert statistics were collected from January 1, 2018 to December 31, 2021. "Preintervention" period was January 1-December 31, 2018, and "postintervention" period was January 1-December 31, 2021. The intervention period was the intervening period. Categorical variables were reported as frequencies and proportions and compared using the chi-square test. Continuous data were reported as median (interquartile range, IQR) and compared using the Wilcoxon rank-sum test. Statistical significance was defined at P < .05.

Results

There was a significant reduction of 59.6% in the total number of interruptive BPA alerts, despite an increase in the number of unique BPAs from 54 to 360 from pre- to postintervention. There was a 74% reduction in the number of alerts from the 7 BPAs that were optimized from the pre- to postintervention period. There was a significant increase in percentage of overall interruptive BPA alerts with action taken (8 [IQR 7.7-8.4] to 54.7 [IQR 52.5-58.9], P-value < .05) and optimized BPAs with action taken (32.6 [IQR 32.3-32.9] to 72.6 [IQR 64.3-73.4], P-value < .05). We estimate that the reduction in alerts saved 3600 h of providers' time per year.

Conclusions

A significant reduction in interruptive alert volume, and a significant increase in action taken rates despite manifold increase in the number of unique BPAs could be achieved through concentrated efforts focusing on governance, data review, and visualization using a system-embedded tool, combined with the CDS Five Rights framework, to optimize alerts. Improved alert compliance was likely multifactorial-due to decreased repeated alert firing for the same patient; better awareness due to stakeholders' involvement; and less fatigue since unnecessary alerts were removed. Future studies should prospectively focus on patients' clinical chart reviews to assess downstream effects of various actions taken, identify any possibility of harm, and collect end-user feedback regarding the utility of alerts.

User Perceptions of Visual Blood: An International Mixed Methods Study on Novel Blood Gas Analysis Visualization

Blood gas analysis plays a central role in modern medicine. Advances in technology have expanded the range of available parameters and increased the complexity of their interpretation. By applying user-centered design principles, it is possible to reduce the cognitive load associated with interpreting blood gas analysis. In this international, multicenter study, we explored anesthesiologists' perspectives on Visual Blood, a novel visualization technique for presenting blood gas analysis results. We conducted interviews with participants following two computer-based simulation studies, the first utilizing virtual reality (VR) (50 participants) and the second without VR (70 participants). Employing the template approach, we identified key themes in the interview responses and formulated six statements, which were rated using Likert scales from 1 (strongly disagree) to 5 (strongly agree) in an online questionnaire. The most frequently mentioned theme was the positive usability features of Visual Blood. The online survey revealed that participants found Visual Blood to be an intuitive method for interpreting blood gas analysis (median 4, interquartile range (IQR) 4-4, p < 0.001). Participants noted that minimal training was required to effectively learn how to interpret Visual Blood (median 4, IQR 4-4, p < 0.001). However, adjustments are necessary to reduce visual overload (median 4, IQR 2-4, p < 0.001). Overall, Visual Blood received a favorable response. The strengths and weaknesses derived from these data will help optimize future versions of Visual Blood to improve the presentation of blood gas analysis results.

Design and Evaluation of an Intensive Care Unit Dashboard Built in Response to the COVID-19 Pandemic: Semistructured Interview Study

BACKGROUND

Dashboards and interactive displays are becoming increasingly prevalent in most health care settings and have the potential to streamline access to information, consolidate disparate data sources and deliver new insights. Our research focuses on intensive care units (ICUs) which are heavily instrumented, critical care environments that generate vast amounts of data and frequently require individualized support for each patient. Consequently, clinicians experience a high cognitive load, which can translate to suboptimal performance. The global COVID-19 pandemic exacerbated this problem by generating a large number of additional hospitalizations, which necessitated a new tool that would help manage ICUs' census. In a previous study, we interviewed clinicians at the University Hospitals Bristol and Weston National Health Service Foundation Trust to capture the requirements for bespoke dashboards that would alleviate this problem.

OBJECTIVE

This study aims to design, implement, and evaluate an ICU dashboard to allow for monitoring of the high volume of patients in need of critical care, particularly tailored to high-demand situations, such as those seen during the COVID-19 pandemic.

METHODS

Building upon the previously gathered requirements, we developed a dashboard, integrated it within the ICU of a National Health Service trust, and allowed all staff to access our tool. For evaluation purposes, participants were recruited and interviewed following a 25-day period during which they were able to use the dashboard clinically. The semistructured interviews followed a topic guide aimed at capturing the usability of the dashboard, supplemented with additional questions asked post hoc to probe themes established during the interview. Interview transcripts were analyzed using a thematic analysis framework that combined inductive and deductive approaches and integrated the Technology Acceptance Model.

RESULTS

A total of 10 participants with 4 different roles in the ICU (6 consultants, 2 junior doctors, 1 nurse, and 1 advanced clinical practitioner) participated in the interviews. Our analysis generated 4 key topics that prevailed across the data: our dashboard met the usability requirements of the participants and was found useful and intuitive; participants perceived that it impacted their delivery of patient care by improving the access to the information and better equipping them to do their job; the tool was used in a variety of ways and for different reasons and tasks; and there were barriers to integration of our dashboard into practice, including familiarity with existing systems, which stifled the adoption of our tool.

CONCLUSIONS

Our findings show that the perceived utility of the dashboard had a positive impact on the clinicians' workflows in the ICU. Improving access to information translated into more efficient patient care and transformed some of the existing processes. The introduction of our tool was met with positive reception, but its integration during the COVID-19 pandemic limited its adoption into practice.

Revenue and Cost Analysis of a System Utilizing Natural Language Processing and a Nurse Coordinator for Radiology Follow-up Recommendations

Radiology reports often contain recommendations for follow-up imaging, Provider adherence to these radiology recommendations can be incomplete, which may result in patient harm, lost revenue, or litigation. This study sought to perform a revenue assessment of a hybrid natural language processing (NLP) and human follow-up system. Reports generated from January 2020 to April 2021 that were indexed as overdue from follow-up recommendations by mPower Follow-Up Recommendation Algorithm (Nuance Communications Inc., Burlington, MA), were assessed for follow up and revenue. Follow-up exams completed because of the hybrid system were tabulated and given revenue amounts based on Medicare national reimbursement rates. These rates were then summated. A total of n =3011 patients were flagged via the mPower algorithm as having not received a timely follow-up indicated for procedure. Of these, n = 427 required the quality nurse to contact their healthcare provider to place orders. The follow-up imaging of these patients accounted for $62,937.66 of revenue. This revenue was calculated as higher than personnel cost (based on national average quality and safety nurse salary and time allotted on follow-ups). Our results indicate that a hybrid human-artificial intelligence follow-up system can be profitable, while potentially adding to patient safety. Our revenue figure likely significantly underestimates the true revenue obtained at our institution. This was due to the use of Medicare national reimbursement rates to calculate revenue, for the purposes of generalizability.

Refining Clinical Phenotypes to Improve Clinical Decision Support and Reduce Alert Fatigue: A Feasibility Study

BACKGROUND

Chronic kidney disease (CKD) is common and associated with adverse clinical outcomes. Most care for early CKD is provided in primary care, including hypertension (HTN) management. Computerized clinical decision support (CDS) can improve the quality of care for CKD but can also cause alert fatigue for primary care physicians (PCPs). Computable phenotypes (CPs) are algorithms to identify disease populations using, for example, specific laboratory data criteria.

OBJECTIVES

Our objective was to determine the feasibility of implementation of CDS alerts by developing CPs and estimating potential alert burden.

METHODS

We utilized clinical guidelines to develop a set of five CPs for patients with stage 3 to 4 CKD, uncontrolled HTN, and indications for initiation or titration of guideline-recommended antihypertensive agents. We then conducted an iterative data analytic process consisting of database queries, data validation, and subject matter expert discussion, to make iterative changes to the CPs. We estimated the potential alert burden to make final decisions about the scope of the CDS alerts. Specifically, the number of times that each alert could fire was limited to once per patient.

RESULTS

In our primary care network, there were 239,339 encounters for 105,992 primary care patients between April 1, 2018 and April 1, 2019. Of these patients, 9,081 (8.6%) had stage 3 and 4 CKD. Almost half of the CKD patients, 4,191 patients, also had uncontrolled HTN. The majority of CKD patients were female, elderly, white, and English-speaking. We estimated that 5,369 alerts would fire if alerts were triggered multiple times per patient, with a mean number of alerts shown to each PCP ranging from 0.07-to 0.17 alerts per week.

CONCLUSION

Development of CPs and estimation of alert burden allows researchers to iteratively fine-tune CDS prior to implementation. This method of assessment can help organizations balance the tradeoff between standardization of care and alert fatigue.

"Are we there yet?" Ten persistent hazards and inefficiencies with the use of medication administration technology from the perspective of practicing nurses

OBJECTIVES

(1) Characterize persistent hazards and inefficiencies in inpatient medication administration; (2) Explore cognitive attributes of medication administration tasks; and (3) Discuss strategies to reduce medication administration technology-related hazards.

MATERIALS AND METHODS

Interviews were conducted with 32 nurses practicing at 2 urban, eastern and western US health systems. Qualitative analysis using inductive and deductive coding included consensus discussion, iterative review, and coding structure revision. We abstracted hazards and inefficiencies through the lens of risks to patient safety and the cognitive perception-action cycle (PAC).

RESULTS

Persistent safety hazards and inefficiencies related to MAT organized around the PAC cycle included: (1) Compatibility constraints create information silos; (2) Missing action cues; (3) Intermittent communication flow between safety monitoring systems and nurses; (4) Occlusion of important alerts by other, less helpful alerts; (5) Dispersed information: Information required for tasks is not collocated; (6) Inconsistent data organization: Mismatch of the display and the user's mental model; (7) Hidden medication administration technologies (MAT) limitations: Inaccurate beliefs about MAT functionality contribute to overreliance on the technology; (8) Software rigidity causes workarounds; (9) Cumbersome dependencies between technology and the physical environment; and (10) Technology breakdowns require adaptive actions.

DISCUSSION

Errors might persist in medication administration despite successful Bar Code Medication Administration and Electronic Medication Administration Record deployment for reducing errors. Opportunities to improve MAT require a deeper understanding of high-level reasoning in medication administration, including control over the information space, collaboration tools, and decision support.

CONCLUSION

Future medication administration technology should consider a deeper understanding of nursing knowledge work for medication administration.

Changes in Primary Care Quality Associated With Implementation of the Veterans Health Administration Preventive Health Inventory

Importance

The COVID-19 pandemic caused significant disruptions in primary care delivery. The Veterans Health Administration (VHA) launched the Preventive Health Inventory (PHI) program-a multicomponent care management intervention, including a clinical dashboard and templated electronic health record note-to support primary care in delivering chronic disease care and preventive care that had been delayed by the pandemic.

Objectives

To describe patient, clinician, and clinic correlates of PHI use in primary care clinics and to examine associations between PHI adoption and clinical quality measures.

Design, Setting, and Participants

This quality improvement study used VHA administrative data from February 1, 2021, through February 28, 2022, from a national cohort of 216 VHA primary care clinics that have implemented the PHI. Participants comprised 829 527 veterans enrolled in primary care in clinics with the highest and lowest decile of PHI use as of February 2021.

Exposure

Templated electronic health record note documenting use of the PHI.

Main Outcomes and Measures

Diabetes and blood pressure clinical quality measures were the primary outcomes. Interrupted time series models were applied to estimate changes in diabetes and hypertension quality measures associated with PHI implementation. Low vs high PHI use was stratified at the facility level to measure whether systematic differences in uptake were associated with quality.

Results

A total of 216 primary clinics caring for 829 527 unique veterans (mean [SD] age, 64.1 [16.9] years; 755 158 of 829 527 [91%] were men) formed the study cohort. Use of the PHI varied considerably across clinics. The clinics in the highest decile of PHI use completed a mean (SD) of 32 997.4 (14 019.3) notes in the electronic health record per 100 000 veterans compared with 56.5 (35.3) notes per 100 000 veterans at the clinics in the lowest decile of use (P < .001). Compared with the clinics with the lowest use of the PHI, clinics with the highest use had a larger mean (SD) clinic size (12 072 [7895] patients vs 5713 [5825] patients; P < .001), were more likely to be urban (91% vs 57%; P < .001), and served more non-Hispanic Black veterans (16% vs 5%; P < .001) and Hispanic veterans (14% vs 4%; P < .001). Staffing did not differ meaningfully between high- and low-use clinics (mean [SD] ratio of full-time equivalent staff to clinician, 3.4 [1.2] vs 3.4 [0.8], respectively; P < .001). After PHI implementation, compared with the clinics with the lowest use, those with the highest use had fewer veterans with a hemoglobin A1c greater than 9% or missing (mean [SD], 6577 [3216] per 100 000 veterans at low-use clinics; 9928 [4236] per 100 000 veterans at high-use clinics), more veterans with an annual hemoglobin A1c measurement (mean [SD], 13 181 [5625] per 100 000 veterans at high-use clinics; 8307 [3539] per 100 000 veterans at low-use clinics), and more veterans with adequate blood pressure control (mean [SD], 20 582 [12 201] per 100 000 veterans at high-use clinics; 12 276 [6850] per 100 000 veterans at low-use clinics).

Conclusions and Relevance

This quality improvement study of the implementation of the VHA PHI suggests that higher use of a multicomponent care management intervention was associated with improved quality-of-care metrics. The study also found significant variation in PHI uptake, with higher uptake associated with clinics with more racial and ethnic diversity and larger, urban clinic sites.

The safety risk of information overload and bureaucracy in oncology clinical trial conduct

Performance of clinical trials has led to major therapeutic developments and substantial improvements in the field of medical oncology. To ensure patient's safety, regulatory aspects for proper clinical trial conduct have been increased over the past two decades but seem to cause information overload and ineffective bureaucracy, possibly even impacting patient safety. To put this in perspective, after the implementation of Directive 2001/20/EC in the European Union, a 90 per cent increase in trial launching time, a 25 per cent decrease in patient participation and a 98 per cent rise in administrative trial costs were reported. The time to initiate a clinical trial has increased from a few months to several years in the past three decades. Moreover, there is a serious risk that information overload with relatively unimportant data endangers the decision-making processes and distracts from essential patient safety information. It is now a critical moment in time to improve efficient clinical trial conduct for our future patients diagnosed with cancer. We are convinced that a reduction of the administrative regulations, information overload, and simplification of the procedures for trial conductance may improve patient safety. In this Current Perspective, we give insight in the current regulatory aspects of clinical research, evaluate the practical consequences of these regulations, and propose specific improvements for optimal clinical trial conduct.

Disruption vs. evolution in laboratory medicine. Current challenges and possible strategies, making laboratories and the laboratory specialist profession fit for the future

Since beginning of medical diagnostics, laboratory specialists have done an amazing job, continuously improving quality, spectrum and speed of laboratory tests, currently contributing to the majority of medical decision making. These improvements are mostly of an incremental evolutionary fashion, meaning improvements of current processes. Sometimes these evolutionary innovations are of a radical fashion, such as the invention of automated analyzers replacing manual testing or the implementation of mass spectrometry, leading to one big performance leap instead of several small ones. In few cases innovations may be of disruptive nature. In laboratory medicine this would be applicable to digitalization of medicine or the decoding of the human genetic material. Currently, laboratory medicine is again facing disruptive innovations or technologies, which need to be adapted to as soon as possible. One of the major disruptive technologies is the increasing availability and medical use of artificial intelligence. It is necessary to rethink the position of the laboratory specialist within healthcare settings and the added value he or she can provide to patient care. The future of the laboratory specialist profession is bright, as it the only medical profession comprising such vast experience in patient diagnostics. However, laboratory specialists need to develop strategies to provide this expertise, by adopting to the quickly evolving technologies and demands. This opinion paper summarizes some of the disruptive technologies as well as strategies to secure and/or improve the quality of diagnostic patient care and the laboratory specialist profession.

Monitoring the Implementation of Tobacco Cessation Support Tools: Using Novel Electronic Health Record Activity Metrics

BACKGROUND

Clinical decision support (CDS) tools in electronic health records (EHRs) are often used as core strategies to support quality improvement programs in the clinical setting. Monitoring the impact (intended and unintended) of these tools is crucial for program evaluation and adaptation. Existing approaches for monitoring typically rely on health care providers' self-reports or direct observation of clinical workflows, which require substantial data collection efforts and are prone to reporting bias.

OBJECTIVE

This study aims to develop a novel monitoring method leveraging EHR activity data and demonstrate its use in monitoring the CDS tools implemented by a tobacco cessation program sponsored by the National Cancer Institute's Cancer Center Cessation Initiative (C3I).

METHODS

We developed EHR-based metrics to monitor the implementation of two CDS tools: (1) a screening alert reminding clinic staff to complete the smoking assessment and (2) a support alert prompting health care providers to discuss support and treatment options, including referral to a cessation clinic. Using EHR activity data, we measured the completion (encounter-level alert completion rate) and burden (the number of times an alert was fired before completion and time spent handling the alert) of the CDS tools. We report metrics tracked for 12 months post implementation, comparing 7 cancer clinics (2 clinics implemented the screening alert and 5 implemented both alerts) within a C3I center, and identify areas to improve alert design and adoption.

RESULTS

The screening alert fired in 5121 encounters during the 12 months post implementation. The encounter-level alert completion rate (clinic staff acknowledged completion of screening in EHR: 0.55; clinic staff completed EHR documentation of screening results: 0.32) remained stable over time but varied considerably across clinics. The support alert fired in 1074 encounters during the 12 months. Providers acted upon (ie, not postponed) the support alert in 87.3% (n=938) of encounters, identified a patient ready to quit in 12% (n=129) of encounters, and ordered a referral to the cessation clinic in 2% (n=22) of encounters. With respect to alert burden, on average, both alerts fired over 2 times (screening alert: 2.7; support alert: 2.1) before completion; time spent postponing the screening alert was similar to completing (52 vs 53 seconds) the alert, and time spent postponing the support alert was more than completing (67 vs 50 seconds) the alert per encounter. These findings inform four areas where the alert design and use can be improved: (1) improving alert adoption and completion through local adaptation, (2) improving support alert efficacy by additional strategies including training in provider-patient communication, (3) improving the accuracy of tracking for alert completion, and (4) balancing alert efficacy with the burden.

CONCLUSIONS

EHR activity metrics were able to monitor the success and burden of tobacco cessation alerts, allowing for a more nuanced understanding of potential trade-offs associated with alert implementation. These metrics can be used to guide implementation adaptation and are scalable across diverse settings.

Providers' and Patients' Perspectives on Diagnostic Errors in the Acute Care Setting

BACKGROUND

Diagnostic errors (DEs) have been studied extensively in ambulatory care, but less work has been done in the acute care setting. In this study, the authors examined health care providers' and patients' perspectives about the classification of DEs, the main causes and scope of DEs in acute care, the main gaps in current systems, and the need for innovative solutions.

METHODS

A qualitative mixed methods study was conducted, including semistructured interviews with health care providers and focus groups with patient advisors. Using grounded theory approach, thematic categories were derived from the interviews and focus groups.

RESULTS

The research team conducted interviews with 17 providers and two focus groups with seven patient advisors. Both providers and patient advisors struggled to define and describe DEs in acute care settings. Although participants agreed that DEs pose a significant risk to patient safety, their perception of the frequency of DEs was mixed. Most participants identified communication failures, lack of comfort with diagnostic uncertainty, incorrect clinical evaluation, and cognitive load as key causes of DEs. Most respondents believed that non-information technology (IT) tools and processes (for example, communication improvement strategies) could significantly reduce DEs.

CONCLUSION

The study findings represent an important supplement to our understanding of DEs in acute care settings and the advancement of a culture of patient safety in the context of patient-centered care and patient engagement. Health care organizations should consider the key factors identified in this study when trying to create a culture that engages clinicians and patients in reducing DEs.

Optimizing Endoscopy Procedure Documentation Improves Guideline-Adherent Care in Upper Gastrointestinal Bleeding

BACKGROUND AND AIMS

Upper GI bleeding (UGIB) is a common indication for inpatient esophagogastroduodenoscopy (EGD). Guideline adherence improves post-EGD care, including appropriate medication dosing/duration and follow-up procedures that reduce UGIB-related morbidity. We aimed to optimize and standardize post-EGD documentation to improve process and clinical outcomes in UGIB-related care.

METHODS

We performed a prospective quality improvement study of inpatient UGIB endoscopies at an academic tertiary referral center during 6/2019-7/2021. Guidelines were used to develop etiology/severity-specific electronic health record note templates. Participants (39 faculty/15 trainees) completed 10-min training in template content/use. We collected pre/post-intervention process data on "Minimal Standard Report" (MSR) documentation including patient disposition, diet, and medications. We also recorded documentation of re-bleed precautions and follow-up procedures. Study outcomes included guideline-based medication prescriptions, ordering of follow-up EGD, and post-discharge re-bleeding. Pre/post-intervention analysis was performed using chi-square tests.

RESULTS

From a pre-intervention baseline of 199 patients to 459 patients post-intervention, compliance improved with inpatient PPI (53.4-77.9%, p < 0.001) and discharge PPI (31.3-61.0%, p < 0.001) prescriptions. There was improvement in MSR completion (28.6-42.5%, p < 0.001). Compliance improved with octreotide prescriptions (75.0-93.6%, p = 0.002) and follow-up EGD order (61.3-87.1%, p < 0.001). There was no change in post-discharge re-bleeding. 82.6% of cases used templates.

CONCLUSIONS

Our project leveraged endoscopy software to standardize documentation, resulting in improved clinical care behavior and efficiency. Our intervention required low burden of maintenance, and sustainability with high utilization over 9 months. Similar endoscopy templates can be applied to other health systems and procedures to improve care.

Multidisciplinary Approach to Improve Human Immunodeficiency Virus and Syphilis Testing Rates in Emergency Departments

BACKGROUND

Best practice guidelines recommend that patients at risk for sexually transmitted infections (STIs), such as gonorrhea (GC) and chlamydia, should also be tested for human immunodeficiency virus (HIV) and syphilis. This prospective quality assurance study aimed to increase HIV and syphilis testing rates in emergency departments (EDs) across the Cleveland Clinic Health System from January 1, 2020 through January 1, 2022.

METHODS

A multidisciplinary team of emergency medicine, infectious diseases, pharmacy, and microbiology personnel convened to identify barriers to HIV and syphilis testing during ED encounters at which GC/chlamydia were tested. The following interventions were implemented in response: rapid HIV testing with new a workflow for results follow-up, a standardized STI-screening order panel, and feedback to clinicians about ordering patterns.

RESULTS

There were 57 797 ED visits with GC/chlamydia testing completed during the study period. Human immunodeficiency virus testing was ordered at 5% of these encounters before the interventions were implemented and increased to 8%, 23%, and 36% after each successive intervention. Syphilis testing increased from 9% before the interventions to 12%, 28%, and 39% after each successive intervention. In multivariable analyses adjusted for age, gender, and location, the odds ratio for HIV and syphilis testing after all interventions was 11.72 (95% confidence interval [CI], 10.82-12.71; P ≤.001) and 6.79 (95% CI, 6.34-7.27; P ≤.001), respectively.

CONCLUSIONS

The multidisciplinary intervention resulted in improved testing rates for HIV and syphilis.

Clinical Decision Support for Glycemic Management Reduces Hospital Length of Stay

OBJECTIVE

Dysglycemia influences hospital outcomes and resource utilization. Clinical decision support (CDS) holds promise for optimizing care by overcoming management barriers. This study assessed the impact on hospital length of stay (LOS) of an alert-based CDS tool in the electronic medical record that detected dysglycemia or inappropriate insulin use, coined as gaps in care (GIC).

RESEARCH DESIGN AND METHODS

Using a 12-month interrupted time series among hospitalized persons aged ≥18 years, our CDS tool identified GIC and, when active, provided recommendations. We compared LOS during 6-month-long active and inactive periods using linear models for repeated measures, multiple comparison adjustment, and mediation analysis.

RESULTS

Among 4,788 admissions with GIC, average LOS was shorter during the tool's active periods. LOS reductions occurred for all admissions with GIC (-5.7 h, P = 0.057), diabetes and hyperglycemia (-6.4 h, P = 0.054), stress hyperglycemia (-31.0 h, P = 0.054), patients admitted to medical services (-8.4 h, P = 0.039), and recurrent hypoglycemia (-29.1 h, P = 0.074). Subgroup analysis showed significantly shorter LOS in recurrent hypoglycemia with three events (-82.3 h, P = 0.006) and nonsignificant in two (-5.2 h, P = 0.655) and four or more (-14.8 h, P = 0.746). Among 22,395 admissions with GIC (4,788, 21%) and without GIC (17,607, 79%), LOS reduction during the active period was 1.8 h (P = 0.053). When recommendations were provided, the active tool indirectly and significantly contributed to shortening LOS through its influence on GIC events during admissions with at least one GIC (P = 0.027), diabetes and hyperglycemia (P = 0.028), and medical services (P = 0.019).

CONCLUSIONS

Use of the alert-based CDS tool to address inpatient management of dysglycemia contributed to reducing LOS, which may reduce costs and improve patient well-being.

A Customized Electronic Health Record-Based Tool Highlights and Addresses Gaps in Patient Safety

The authors present a tool to improve gaps in patient safety using the electronic health record. The tool integrates gap identification, passive alerts, and actions into a single interface embedded within clinicians' workflow. The tool was developed to address venous thromboembolism prophylaxis, prevention of hypo- and hyperglycemia, code status documentation, bowel movement frequency, and skilled nursing facility transitions. Alerts and actions during silent and live periods were retrospectively analyzed. The most prevalent safety gaps were lack of venous thromboembolism prophylaxis (40.4% of alerts), constipation (19.3%), and lack of code status (18.4%). Disparities in safety gaps were present by patient race, sex, and socioeconomic status. Usability testing showed positive feedback without significant alert burden. Thus, a safety gap tool was successfully built to study and address patient safety issues. The tool's strengths are its integration within the electronic health record, ease of use, customizability, and scalability.

Patient Safety Issues From Information Overload in Electronic Medical Records

BACKGROUND AND OBJECTIVE

Electronic health records (EHRs) have become ubiquitous in medicine and continue to grow in informational content. Little has been documented regarding patient safety from the resultant information overload. The objective of this literature review is to better understand how information overload in EHR affects patient safety.

METHODS

A literature search was performed using the Transparent Reporting of Systematic Reviews and Meta-Analyses standards for literature review. PubMed and Web of Science were searched and articles selected that were relevant to EHR information overload based on keywords.

RESULTS

The literature search yielded 28 articles meeting the criteria for the study. Information overload was found to increase physician cognitive load and error rates in clinical simulations. Overabundance of clinically irrelevant information, poor data display, and excessive alerting were consistently identified as issues that may lead to information overload.

CONCLUSIONS

Information overload in EHRs may result in higher error rates and negatively impact patient safety. Further studies are necessary to define the role of EHR in adverse patient safety events and to determine methods to mitigate these errors. Changes focused on the usability of EHR should be considered with the end user (physician) in mind. Federal agencies have a role to play in encouraging faster adoption of improved EHR interfaces.

Efficiency, efficacy and subjective user satisfaction of alternative laboratory report formats. An investigation on behalf of the Working Group for Postanalytical Phase (WG-POST), of the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM)

OBJECTIVES

Although laboratory result presentation may lead to information overload and subsequent missed or delayed diagnosis, little has been done in the past to improve this post-analytical issue. We aimed to investigate the efficiency, efficacy and user satisfaction of alternative report formats.

METHODS

We redesigned cumulative (sparkline format) and single reports (improved tabular and z-log format) and tested these on 46 physicians, nurses and medical students in comparison to the classical tabular formats, by asking standardized questions on general items on the reports as well as on suspected diagnosis and follow-up treatment or diagnostics.

RESULTS

Efficacy remained at a very high level both in the new formats as well as in the classical formats. We found no significant difference in any of the groups. Efficiency improved in all groups when using the sparkline cumulative format and marginally when showing the improved tabular format. When asking medical questions, efficiency and efficacy remained similar between report formats and groups. All alternative reports were subjectively more attractive to the majority of participants.

CONCLUSIONS

Showing cumulative reports as a graphical display led to faster detection of general information on the report with the same level of correctness. Considering the familiarity bias of the classical single report formats, the borderline-significant improvement of the alternative tabular format and the non-inferiority of the z-log format, suggests that single reports might benefit from some improvements derived from basic information design.

Exploratory analysis of novel electronic health record variables for quantification of healthcare delivery strain, prediction of mortality, and prediction of imminent discharge

OBJECTIVE

To explore the relationship between novel, time-varying predictors for healthcare delivery strain (eg, counts of patient orders per hour) and imminent discharge and in-hospital mortality.

MATERIALS AND METHODS

We conducted a retrospective cohort study using data from adults hospitalized at 21 Kaiser Permanente Northern California hospitals between November 1, 2015 and October 31, 2020 and the nurses caring for them. Patient data extracted included demographics, diagnoses, severity measures, occupancy metrics, and process of care metrics (eg, counts of intravenous drip orders per hour). We linked these data to individual registered nurse records and created multiple dynamic, time-varying predictors (eg, mean acute severity of illness for all patients cared for by a nurse during a given hour). All analyses were stratified by patients' initial hospital unit (ward, stepdown unit, or intensive care unit). We used discrete-time hazard regression to assess the association between each novel time-varying predictor and the outcomes of discharge and mortality, separately.

RESULTS

Our dataset consisted of 84 162 161 hourly records from 954 477 hospitalizations. Many novel time-varying predictors had strong associations with the 2 study outcomes. However, most of the predictors did not merely track patients' severity of illness; instead, many of them only had weak correlations with severity, often with complex relationships over time.

DISCUSSION

Increasing availability of process of care data from automated electronic health records will permit better quantification of healthcare delivery strain. This could result in enhanced prediction of adverse outcomes and service delays.

CONCLUSION

New conceptual models will be needed to use these new data elements.

A Retrospective Case Series of Concomitant Carbapenem and Valproic Acid Use: Are Best Practice Advisories Working?

Background: A Best Practice Advisory (BPA) warns clinicians of an interaction between carbapenems and valproic acid (VPA) that can cause significant declines in VPA levels leading to serious consequences for patients treated for seizure disorder and unknown implications for alternative indications. Objective: The goal of this study was to assess BPA efficacy in avoiding concomitant VPA/carbapenems, and to characterize use of these agents, clinical implications, and potential alternative therapeutic options. Methods: Retrospective chart review was performed on all patients over the course of 1 year who were concomitantly prescribed a carbapenem and VPA at Hartford Hospital, Hartford, CT. Data collected included: level of care, duration of concomitant therapy, indications, VPA levels during or surrounding overlap, documentation of the interaction, and therapeutic implications. Results: Carbapenems and VPA were administered to 591 and 625 patients, respectively; the BPA fired 126 times in 24 patients, and 15 patients were initiated on these agents concomitantly. Eight (53%) patients received VPA for seizures. The remaining seven (47%) received VPA for alternative indications. Eight of nine VPA levels were sub-therapeutic during carbapenem therapy and polypharmacy was administered in all patients receiving VPA for non-convulsive indications. Conclusion: Co-prescribing of these drugs was rare; however, the BPA was ineffective in 63% of instances. Reductions in VPA efficacy for any indication should be expected with concomitant carbapenem administration. Antibiotics other than carbapenems should be considered when coverage of multidrug resistant Gram-negative pathogens is required in patients whose VPA treatment cannot be interrupted or switched to a therapeutic alternative.

Information Complexity and Behavior Intention to Prescribe Antibiotics Based on the Antimicrobial Susceptibility Testing Report: The Mediating Roles of Information Overload and Attitude

Background: The antimicrobial susceptibility testing (AST) report has guiding significance for physicians to prescribe antibiotics. This study aims to examine the effect of the AST report information complexity on physician’s intention to prescribe antibiotics based on the AST report, as well as the mediating role of information overload and attitude.

Methods: A cross-sectional study conducted on 411 physicians in a general hospital in China in July 2021. Data were collected by a self-reported questionnaire. A serial multiple mediation model was tested to explore the sequential causality between the information complexity of the AST report, information overload, attitude, and behavior intention to prescribe antibiotics based on the AST report by using the SPSS macro PROCESS program.

Results: Information complexity, information overload, attitude and behavior intention were significantly correlated (p < 0.01). Information complexity can not only have a direct positive impact on the intention to prescribe antibiotics based on the AST report (effect = 0.173; SE = 0.044; Boot95%CI: LL = 0.089, UL = 0.260), but also have an indirect impact on behavior intention through the independent mediating role of information overload (effect = 0.025; SE = 0.011; Boot 95%CI: LL = 0.008, UL = 0.050) and the independent mediating role of attitude (effect = 0.130; SE = 0.025; Boot 95%CI: LL = 0.086, UL = 0.180), while the chain of information overload and attitude played a masking effect between information complexity and behavior intention (effect = −0.013; SE = 0.004; Boot 95%CI: LL = −0.023, UL = −0.005).

Conclusion: The increase in information complexity can encourage physicians to prescribe antibiotics based on the AST report, information overload and attitude can promote this effect. It is necessary to provide physicians with sufficient information to prescribe antibiotics without increasing the burden on them. At the same time, publicity and standardized training should be conducted for physicians to interpret the AST report better and faster.

Disease evolution and risk-based disease trajectories in congestive heart failure patients

Congestive Heart Failure (CHF) is among the most prevalent chronic diseases worldwide, and is commonly associated with comorbidities and complex health conditions. Consequently, CHF patients are typically hospitalized frequently, and are at a high risk of premature death. Early detection of an envisaged patient disease trajectory is crucial for precision medicine. However, despite the abundance of patient-level data, cardiologists currently struggle to identify disease trajectories and track the evolution patterns of the disease over time, especially in small groups of patients with specific disease subtypes. The present study proposed a five-step method that allows clustering CHF patients, detecting cluster similarity, and identifying disease trajectories, and promises to overcome the existing difficulties. This work is based on a rich dataset of patients' records spanning ten years of hospital visits. The dataset contains all the health information documented in the hospital during each visit, including diagnoses, lab results, clinical data, and demographics. It utilizes an innovative Cluster Evolution Analysis (CEA) method to analyze the complex CHF population where each subject is potentially associated with numerous variables. We have defined sub-groups for mortality risk levels, which we used to characterize patients' disease evolution by refined data clustering in three points in time over ten years, and generating patients' migration patterns across periods. The results elicited 18, 23, and 25 clusters respective to the first, second, and third visits, uncovering clinically interesting small sub-groups of patients. In the following post-processing stage, we identified meaningful patterns. The analysis yielded fine-grained patient clusters divided into several finite risk levels, including several small-sized groups of high-risk patients. Significantly, the analysis also yielded longitudinal patterns where patients' risk levels changed over time. Four types of disease trajectories were identified: decline, preserved state, improvement, and mixed-progress. This stage is a unique contribution of the work. The resulting fine partitioning and longitudinal insights promise to significantly assist cardiologists in tailoring personalized interventions to improve care quality. Cardiologists could utilize these results to glean previously undetected relationships between symptoms and disease evolution that would allow a more informed clinical decision-making and effective interventions.

Comparing the predictive value of screening to the use of electronic health record data for detecting future suicidal thoughts and behavior in an urban pediatric emergency department: A preliminary analysis

AIM

Brief screening and predictive modeling have garnered attention for utility at identifying individuals at risk of suicide. Although previous research has investigated these methods, little is known about how these methods compare against each other or work in combination in the pediatric population.

METHODS

Patients were aged 8-18 years old who presented from January 1, 2017, to June 30, 2019, to a Pediatric Emergency Department (PED). All patients were screened with the Ask Suicide Questionnaire (ASQ) as part of a universal screening approach. For all models, we used 5-fold cross-validation. We compared four models: Model 1 only included the ASQ; Model 2 included the ASQ and EHR data gathered at the time of ED visit (EHR data); Model 3 only included EHR data; and Model 4 included EHR data and a single item from the ASQ that asked about a lifetime history of suicide attempt. The main outcome was subsequent PED visit with suicide-related presenting problem within a 3-month follow-up period.

RESULTS

Of the N = 13,420 individuals, n = 141 had a subsequent suicide-related PED visit. Approximately 63% identified as Black. Results showed that a model based only on EHR data (Model 3) had an area under the curve (AUC) of 0.775 compared to the ASQ alone (Model 1), which had an AUC of 0.754. Combining screening and EHR data (Model 4) resulted in a 17.4% (absolute difference = 3.6%) improvement in sensitivity and 13.4% increase in AUC (absolute difference = 6.6%) compared to screening alone (Model 1).

CONCLUSION

Our findings show that predictive modeling based on EHR data is helpful either in the absence or as an addition to brief suicide screening. This is the first study to compare brief suicide screening to EHR-based predictive modeling and adds to our understanding of how best to identify youth at risk of suicidal thoughts and behaviors in clinical care settings.

Electronic tools in clinical laboratory diagnostics: key examples, limitations, and value in laboratory medicine

Electronic tools in clinical laboratory diagnostics can assist laboratory professionals, clinicians, and patients in medical diagnostic management and laboratory test interpretation. With increasing implementation of electronic health records (EHRs) and laboratory information systems worldwide, there is increasing demand for well-designed and evidence-based electronic resources. Both complex data-driven and simple interpretative electronic healthcare tools are currently available to improve the integration of clinical and laboratory information towards a more patient-centered approach to medicine. Several studies have reported positive clinical impact of electronic healthcare tool implementation in clinical laboratory diagnostics, including in the management of neonatal bilirubinemia, cardiac disease, and nutritional status. As patients have increasing access to their medical laboratory data, it is essential that accessible electronic healthcare tools are evidence-based and user-friendly for individuals of varying digital and medical literacy. Indeed, studies suggest electronic healthcare tool development processes significantly lack the involvement of relevant healthcare professionals and often present misinformation, including erroneous calculation algorithms or inappropriate interpretative recommendations. The current review provides an overview of the utility of available electronic healthcare tools in clinical laboratory diagnostics and critically reviews potential limitations and benefits of their clinical implementation. The Canadian Laboratory Initiative on Pediatric Reference Intervals (CALIPER) online database is also detailed as an example of a pediatric diagnostic tool with widespread global impact.

Iterative heuristic design of temporal graphic displays with clinical domain experts

Conventional electronic health record information displays are not optimized for efficient information processing. Graphical displays that integrate patient information can improve information processing, especially in data-rich environments such as critical care. We propose an adaptable and reusable approach to patient information display with modular graphical components (widgets). We had two study objectives. First, reduce numerous widget prototype alternatives to preferred designs. Second, derive widget design feature recommendations. Using iterative human-centered design methods, we interviewed experts to hone design features of widgets displaying frequently measured data elements, e.g., heart rate, for acute care patient monitoring and real-time clinical decision-making. Participant responses to design queries were coded to calculate feature-set agreement, average prototype score, and prototype agreement. Two iterative interview cycles covering 64 design queries and 86 prototypes were needed to reach consensus on six feature sets. Interviewers agreed that line graphs with a smoothed or averaged trendline, 24-h timeframe, and gradient coloring for urgency were useful and informative features. Moreover, users agreed that widgets should include key functions: (1) adjustable reference ranges, (2) expandable timeframes, and (3) access to details on demand. Participants stated graphical widgets would be used to identify correlating patterns and compare abnormal measures across related data elements at a specific time. Combining theoretical principles and validated design methods was an effective and reproducible approach to designing widgets for healthcare displays. The findings suggest our widget design features and recommendations match critical care clinician expectations for graphical information display of continuous and frequently updated patient data.

Building the evidence-base to reduce electronic health record-related clinician burden

Clinicians face competing pressures of being clinically productive while using imperfect electronic health record (EHR) systems and maximizing face-to-face time with patients. EHR use is increasingly associated with clinician burnout and underscores the need for interventions to improve clinicians' experiences. With an aim of addressing this need, we share evidence-based informatics approaches, pragmatic next steps, and future research directions to improve 3 of the highest contributors to EHR burden: (1) documentation, (2) chart review, and (3) inbox tasks. These approaches leverage speech recognition technologies, natural language processing, artificial intelligence, and redesign of EHR workflow and user interfaces. We also offer a perspective on how EHR vendors, healthcare system leaders, and policymakers all play an integral role while sharing responsibility in helping make evidence-based sociotechnical solutions available and easy to use.

Is Electronic Health Literacy Associated with Learning Outcomes among Medical Students in the First Clinical Year?: A Cross-Sectional Study

Medical students tend to use the internet as a primary resource when seeking health information. This study aims to assess the patterns of internet use, eHL level, and learning outcomes with eHL among medical students at Chiang Mai University. A cross-sectional study was conducted among 88 medical students in the first clinical year. The eHL level was determined using the Thai version of the electronic Health Literacy Scale or eHEALS. The patient case report scores were obtained representing the learning outcome. Linear regression was used to identify factors influencing their eHL level and case report scores. Students recognized the importance and usefulness of the internet. The mean eHEALS score was 33.45. There was a lower degree of agreement on questions regarding internet usage, having skills to evaluate the resources, and confidence in using health information to make health decisions. The eHEALS score had no statistically significant association with most variables and case report scores, but with the longer time of internet use (p-value = 0.014). Although medical students perceived that they have high eHL levels, they report lower confidence in using the information. Including critical thinking skills for electronic health information in the medical curriculum could be useful.

Electronic Health Record Closed-Loop Communication Program for Unexpected Nonemergent Findings

Background Reliance on examination reporting of unexpected imaging findings does not ensure receipt of findings or appropriate follow-up. A closed-loop communication system should include provider and patient notifications and be auditable through the electronic health record (EHR). Purpose To report the initial design of and results from using an EHR-integrated unexpected findings navigator (UFN) program that ensures closed-loop communication of unexpected nonemergent findings. Materials and Methods An EHR-integrated UFN program was designed to enable identification and communication of unexpected findings and aid in next steps in findings management. Three navigators (with prior training as radiologic technologists and sonographers) facilitated communication and documentation of results to providers and patients. Twelve months (October 2019 to October 2020) of results were retrospectively reviewed to evaluate patient demographics and program metrics. Descriptive statistics and correlation analysis were performed by using commercially available software. Results A total of 3542 examinations were reported within 12 months, representing 0.5% of all examinations performed (total of 749 649); the median patient age was 62 years (range, 1 day to 98 years; interquartile range, 23 years). Most patients were female (2029 of 3542 [57%]). Almost half of the examinations submitted were from chest radiography and CT (1618 of 3542 [46%]), followed by MRI and CT of the abdomen and pelvis (1123 of 3542 [32%]). The most common unexpected findings were potential neoplasms (391 of 3542 [11%]). The median time between examination performance and patient notification was 12 days (range, 0-136 days; interquartile range, 13 days). A total of 2127 additional imaging studies were performed, and 1078 patients were referred to primary care providers and specialists. Most radiologists (89%, 63 of 71 respondents) and providers (65%, 28 of 43 respondents) found the system useful and used it most frequently during regular business hours. Conclusion An electronic health record-integrated, navigator-facilitated, closed-loop communication program for unexpected radiologic findings led to near-complete success in notification of providers and patients and facilitated the next steps in findings management. © RSNA, 2021 See also the editorial by Safdar in this issue.

Evaluating Simplified Web Interfaces of Risk Models for Clinical Use: Pilot Survey Study

BACKGROUND

In this pilot study, we investigated sociotechnical factors that affect intention to use a simplified web model to support clinical decision making.

OBJECTIVE

We investigated factors that are known to affect technology adoption using the unified theory of acceptance and use of technology (UTAUT2) model. The goal was to pilot and test a tool to better support complex clinical assessments.

METHODS

Based on the results of a previously published work, we developed a web-based mobile user interface, WebModel, to allow users to work with regression equations and their predictions to evaluate the impact of various characteristics or treatments on key outcomes (eg, survival time) for chronic obstructive pulmonary disease. The WebModel provides a way to combat information overload and more easily compare treatment options. It limits the number of web forms presented to a user to between 1 and 20, rather than the dozens of detailed calculations typically required. The WebModel uses responsive design and can be used on multiple devices. To test the WebModel, we designed a questionnaire to probe the efficacy of the WebModel and assess the usability and usefulness of the system. The study was live for one month, and participants had access to it over that time. The questionnaire was administered online, and data from 674 clinical users who had access to the WebModel were captured. SPSS and R were used for statistical analysis.

RESULTS

The regression model developed from UTAUT2 constructs was a fit. Specifically, five of the seven factors were significant positive coefficients in the regression: performance expectancy (β=.2730; t=7.994; P<.001), effort expectancy (β=.1473; t=3.870; P=.001), facilitating conditions (β=.1644; t=3.849; P<.001), hedonic motivation (β=.2321; t=3.991; P<.001), and habit (β=.2943; t=12.732). Social influence was not a significant factor, while price value had a significant negative influence on intention to use the WebModel.

CONCLUSIONS

Our results indicate that multiple influences impact positive response to the system, many of which relate to the efficiency of the interface to provide clear information. Although we found that the price value was a negative factor, it is possible this was due to the removal of health workers from purchasing decisions. Given that this was a pilot test, and that the system was not used in a clinical setting, we could not examine factors related to actual workflow, patient safety, or social influence. This study shows that the concept of a simplified WebModel could be effective and efficient in reducing information overload in complex clinical decision making. We recommend further study to test this in a clinical setting and gather qualitative data from users regarding the value of the tool in practice.

Electronic Health Record Usability: Associations With Nurse and Patient Outcomes in Hospitals

BACKGROUND

Electronic health record (EHR) usability issues represent an emerging threat to the wellbeing of nurses and patients; however, few large studies have examined these relationships.

OBJECTIVE

To examine associations between EHR usability and nurse job (burnout, job dissatisfaction, and intention to leave) and surgical patient (inpatient mortality and 30-day readmission) outcomes.

METHODS

A cross-sectional analysis of linked American Hospital Association, state patient discharge, and nurse survey data was conducted. The sample included 343 hospitals, 1,281,848 surgical patients, and 12,004 nurses. Logistic regression models were used to assess relationships between EHR usability and outcomes, before and after accounting for EHR adoption level (comprehensive vs. basic or less) and other confounders.

RESULTS

In fully adjusted models, nurses who worked in hospitals with poorer EHR usability had significantly higher odds of burnout [odds ratio (OR), 1.41; 95% confidence interval (CI), 1.21-1.64], job dissatisfaction (OR, 1.61; 95% CI, 1.37-1.90) and intention to leave (OR, 1.31; 95% CI, 1.09-1.58) compared with nurses working in hospitals with better usability. Surgical patients treated in hospitals with poorer EHR usability had significantly higher odds of inpatient mortality (OR, 1.21; 95% CI, 1.09-1.35) and 30-day readmission (OR, 1.06; 95% CI, 1.01-1.12) compared with patients in hospitals with better usability. Comprehensive EHR adoption was associated with higher odds of nurse burnout (OR, 1.14; 95% CI, 1.01-1.28).

CONCLUSION

Employing EHR systems with suboptimal usability was associated with higher odds of adverse nurse job outcomes and surgical patient mortality and readmission. EHR usability may be more important to nurse job and patient outcomes than comprehensive EHR adoption.

Leveraging Machine Learning and Artificial Intelligence to Improve Peripheral Artery Disease Detection, Treatment, and Outcomes

Peripheral artery disease is an atherosclerotic disorder which, when present, portends poor patient outcomes. Low diagnosis rates perpetuate poor management, leading to limb loss and excess rates of cardiovascular morbidity and death. Machine learning algorithms and artificially intelligent systems have shown great promise in application to many areas in health care, such as accurately detecting disease, predicting patient outcomes, and automating image interpretation. Although the application of these technologies to peripheral artery disease are in their infancy, their promises are tremendous. In this review, we provide an introduction to important concepts in the fields of machine learning and artificial intelligence, detail the current state of how these technologies have been applied to peripheral artery disease, and discuss potential areas for future care enhancement with advanced analytics.

A visual representation of microbiological culture data improves comprehension: a randomized controlled trial

OBJECTIVE

While the judicious use of antibiotics takes past microbiological culture results into consideration, this data's typical format in the electronic health record (EHR) may be unwieldy when incorporated into clinical decision-making. We hypothesize that a visual representation of sensitivities may aid in their comprehension.

MATERIALS AND METHODS

A prospective parallel unblinded randomized controlled trial was undertaken at an academic urban tertiary care center. Providers managing emergency department (ED) patients receiving antibiotics and having previous culture sensitivity testing were included. Providers were randomly selected to use standard EHR functionality or a visual representation of patients' past culture data as they answered questions about previous sensitivities. Concordance between provider responses and past cultures was assessed using the kappa statistic. Providers were surveyed about their decision-making and the usability of the tool using Likert scales.

RESULTS

518 ED encounters were screened from 3/5/2018 to 9/30/18, with providers from 144 visits enrolled and analyzed in the intervention arm and 129 in the control arm. Providers using the visualization tool had a kappa of 0.69 (95% CI: 0.65-0.73) when asked about past culture results while the control group had a kappa of 0.16 (95% CI: 0.12-0.20). Providers using the tool expressed improved understanding of previous cultures and found the tool easy to use (P < .001). Secondary outcomes showed no differences in prescribing practices.

CONCLUSION

A visual representation of culture sensitivities improves comprehension when compared to standard text-based representations.

Notification System for Overdue Radiology Recommendations Improves Rates of Follow-Up and Diagnosis

OBJECTIVE. The purpose of this study was to quantify improved rates of follow-up and additional important diagnoses made after notification for overdue workups recommended by radiologists. MATERIALS AND METHODS. Standard reports from imaging studies performed at our institution from October through November 2016 were searched for the words "recommend" or "advised," yielding 9784 studies. Of these, 5245 were excluded, yielding 4539 studies; reports for 1599 of these 4539 consecutive studies were reviewed to identify firm or soft recommendations or findings requiring immediate management. If recommended follow-ups were incomplete within 1 month of the advised time, providers were notified. Compliance was calculated before and after notification and was compared using a one-sample test of proportion. RESULTS. Of 1599 patients, 92 were excluded because they had findings requiring immediate management, and 684 were excluded because of soft recommendations, yielding 823 patients. Of these patients, 125 were not yet overdue for follow-up and were excluded, and 18 were excluded because of death or transfer to another institution. Of the remaining 680 patients, follow-up was completed for 503 (74.0%). A total of 177 (26.0%) of the 680 patients were overdue for follow-up, and providers were notified. Of these 177 patients, 36 (20.3%) completed their follow-ups after notification, 34 (19.2%) had follow-up designated by the provider as nonindicated, and 107 (60.5%) were lost to follow-up, yielding four clinically important diagnoses: one biopsy-proven malignancy, one growing mass, and two thyroid nodules requiring biopsy. The rate of incomplete follow-ups after communication decreased from 26.0% (177/680) to 20.7% (141/680) (95% CI, 17.7-23.9%; p = .002), with a 20.4% reduction in relative risk of noncompliance, and 39.5% (70/177) of overdue cases were resolved when nonindicated studies were included. CONCLUSION. Notification of overdue imaging recommendations reduces incomplete follow-ups and yields clinically important diagnoses.

Physician Stress During Electronic Health Record Inbox Work: In Situ Measurement With Wearable Sensors

Background

Increased work through electronic health record (EHR) messaging is frequently cited as a factor of physician burnout. However, studies to date have relied on anecdotal or self-reported measures, which limit the ability to match EHR use patterns with continuous stress patterns throughout the day.

Objective

The aim of this study is to collect EHR use and physiologic stress data through unobtrusive means that provide objective and continuous measures, cluster distinct patterns of EHR inbox work, identify physicians’ daily physiologic stress patterns, and evaluate the association between EHR inbox work patterns and physician physiologic stress.

Methods

Physicians were recruited from 5 medical centers. Participants (N=47) were given wrist-worn devices (Garmin Vivosmart 3) with heart rate sensors to wear for 7 days. The devices measured physiological stress throughout the day based on heart rate variability (HRV). Perceived stress was also measured with self-reports through experience sampling and a one-time survey. From the EHR system logs, the time attributed to different activities was quantified. By using a clustering algorithm, distinct inbox work patterns were identified and their associated stress measures were compared. The effects of EHR use on physician stress were examined using a generalized linear mixed effects model.

Results

Physicians spent an average of 1.08 hours doing EHR inbox work out of an average total EHR time of 3.5 hours. Patient messages accounted for most of the inbox work time (mean 37%, SD 11%). A total of 3 patterns of inbox work emerged: inbox work mostly outside work hours, inbox work mostly during work hours, and inbox work extending after hours that were mostly contiguous to work hours. Across these 3 groups, physiologic stress patterns showed 3 periods in which stress increased: in the first hour of work, early in the afternoon, and in the evening. Physicians in group 1 had the longest average stress duration during work hours (80 out of 243 min of valid HRV data; P=.02), as measured by physiological sensors. Inbox work duration, the rate of EHR window switching (moving from one screen to another), the proportion of inbox work done outside of work hours, inbox work batching, and the day of the week were each independently associated with daily stress duration (marginal R²=15%). Individual-level random effects were significant and explained most of the variation in stress (conditional R²=98%).

Conclusions

This study is among the first to demonstrate associations between electronic inbox work and physiological stress. We identified 3 potentially modifiable factors associated with stress: EHR window switching, inbox work duration, and inbox work outside work hours. Organizations seeking to reduce physician stress may consider system-based changes to reduce EHR window switching or inbox work duration or the incorporation of inbox management time into work hours.

The Development and Piloting of the Ambulatory Electronic Health Record Evaluation Tool: Lessons Learned

BACKGROUND

Substantial research has been performed about the impact of computerized physician order entry on medication safety in the inpatient setting; however, relatively little has been done in ambulatory care, where most medications are prescribed.

OBJECTIVE

To outline the development and piloting process of the Ambulatory Electronic Health Record (EHR) Evaluation Tool and to report the quantitative and qualitative results from the pilot.

METHODS

The Ambulatory EHR Evaluation Tool closely mirrors the inpatient version of the tool, which is administered by The Leapfrog Group. The tool was piloted with seven clinics in the United States, each using a different EHR. The tool consists of a medication safety test and a medication reconciliation module. For the medication test, clinics entered test patients and associated test orders into their EHR and recorded any decision support they received. An overall percentage score of unsafe orders detected, and order category scores were provided to clinics. For the medication reconciliation module, clinics demonstrated how their EHR electronically detected discrepancies between two medication lists.

RESULTS

For the medication safety test, the clinics correctly alerted on 54.6% of unsafe medication orders. Clinics scored highest in the drug allergy (100%) and drug-drug interaction (89.3%) categories. Lower scoring categories included drug age (39.3%) and therapeutic duplication (39.3%). None of the clinics alerted for the drug laboratory or drug monitoring orders. In the medication reconciliation module, three (42.8%) clinics had an EHR-based medication reconciliation function; however, only one of those clinics could demonstrate it during the pilot.

CONCLUSION

Clinics struggled in areas of advanced decision support such as drug age, drug laboratory, and drub monitoring. Most clinics did not have an EHR-based medication reconciliation function and this process was dependent on accessing patients' medication lists. Wider use of this tool could improve outpatient medication safety and can inform vendors about areas of improvement.

Physicians' gender and their use of electronic health records: findings from a mixed-methods usability study

OBJECTIVE

Physician burnout associated with EHRs is a major concern in health care. A comprehensive assessment of differences among physicians in the areas of EHR performance, efficiency, and satisfaction has not been conducted. The study sought to study relationships among physicians' performance, efficiency, perceived workload, satisfaction, and usability in using the electronic health record (EHR) with comparisons by age, gender, professional role, and years of experience with the EHR.

MATERIALS AND METHODS

Mixed-methods assessments of the medical intensivists' EHR use and perceptions. Using simulated cases, we employed standardized scales, performance measures, and extensive interviews. NASA Task Load Index (TLX), System Usability Scale (SUS), and Questionnaire on User Interface Satisfaction surveys were deployed.

RESULTS

The study enrolled 25 intensive care unit (ICU) physicians (11 residents, 9 fellows, 5 attendings); 12 (48%) were men, with a mean age of 33 (range, 28-55) years and a mean of 4 (interquartile range, 2.0-5.5) years of Epic experience. Overall task performance scores were similar for men (90% ± 9.3%) and women (92% ± 4.4%), with no statistically significant differences (P = .374). However, female physicians demonstrated higher efficiency in completion time (difference = 7.1 minutes; P = .207) and mouse clicks (difference = 54; P = .13). Overall, men reported significantly higher perceived EHR workload stress compared with women (difference = 17.5; P < .001). Men reported significantly higher levels of frustration with the EHR compared with women (difference = 33.15; P < .001). Women reported significantly higher satisfaction with the ease of use of the EHR interface than men (difference = 0.66; P =.03). The women's perceived overall usability of the EHR is marginally higher than that of the men (difference = 10.31; P =.06).

CONCLUSIONS

Among ICU physicians, we measured significant gender-based differences in perceived EHR workload stress, satisfaction, and usability-corresponding to objective patterns in EHR efficiency. Understanding the reasons for these differences may help reduce burnout and guide improvements to physician performance, efficiency, and satisfaction with EHR use.

DESIGN

Mixed-methods assessments of the medical intensivists' EHR use and perceptions. Using simulated cases, we employed standardized scales, performance measures, and extensive interviews.

Expanding Telemonitoring in a Virtual World: A Case Study of the Expansion of a Heart Failure Telemonitoring Program During the COVID-19 Pandemic

Background

To minimize the spread and risk of a COVID-19 outbreak, societal norms have been challenged with respect to how essential services are delivered. With pressures to reduce the number of in-person ambulatory visits, innovative models of telemonitoring have been used during the pandemic as a necessary alternative to support access to care for patients with chronic conditions. The pandemic has led health care organizations to consider the adoption of telemonitoring interventions for the first time, while others have seen existing programs rapidly expand.

Objective

At the Toronto General Hospital in Ontario, Canada, the rapid expansion of a telemonitoring program began on March 9, 2020, in response to COVID-19. The objective of this study was to understand the experiences related to the expanded role of a telemonitoring program under the changing conditions of the pandemic.

Methods

A single-case qualitative study was conducted with 3 embedded units of analysis. Semistructured interviews probed the experiences of patients, clinicians, and program staff from the Medly telemonitoring program at a heart function clinic in Toronto, Canada. Data were analyzed using inductive thematic analysis as well as Eakin and Gladstone’s value-adding approach to enhance the analytic interpretation of the study findings.

Results

A total of 29 participants were interviewed, including patients (n=16), clinicians (n=9), and operational staff (n=4). Four themes were identified: (1) providing care continuity through telemonitoring; (2) adapting telemonitoring operations for a more virtual health care system; (3) confronting virtual workflow challenges; and (4) fostering a meaningful patient-provider relationship. Beyond supporting virtual visits, the program’s ability to provide a more comprehensive picture of the patient’s health was valued. However, issues relating to the lack of system integration and alert-driven interactions jeopardized the perceived sustainability of the program.

Conclusions

With the reduction of in-person visits during the pandemic, virtual services such as telemonitoring have demonstrated significant value. Based on our study findings, we offer recommendations to proactively adapt and scale telemonitoring programs under the changing conditions of an increasingly virtual health care system. These include revisiting the scope and expectations of telemedicine interventions, streamlining virtual patient onboarding processes, and personalizing the collection of patient information to build a stronger virtual relationship and a more holistic assessment of patient well-being.

Survey of psychiatrist use of digital technology in clinical practice

Background

Psychiatrists were surveyed to obtain an overview of how they currently use technology in clinical practice, with a focus on psychiatrists who treat patients with bipolar disorder.

Methods

Data were obtained using an online-only survey containing 46 questions, completed by a convenience sample of 209 psychiatrists in 19 countries. Descriptive statistics, and analyses of linear associations and to remove country heterogeneity were calculated.

Results

Virtually all psychiatrists seek information online with many benefits, but some experience information overload. 75.2% of psychiatrists use an EMR/EHR at work, and 64.6% communicate with patients using a new technology, primarily email (48.8%). 66.0% do not ask patients if they use the Internet in relation to bipolar disorder. 67.3% of psychiatrists feel it is too early to tell if patient online information seeking about bipolar disorder is improving the quality of care. 66.3% of psychiatrists think technology-based treatments will improve the quality of care for some or many patients. However, 60.0% of psychiatrists do not recommend technology-based treatments to patients, and those who recommend select a variety of treatments. Psychiatrists use technology more frequently when the patients live in urban rather than rural or suburban areas. Only 23.9% of psychiatrists have any formal training in technology.

Conclusions

Digital technology is routinely used by psychiatrists in clinical practice. There is near unanimous agreement about the benefits of psychiatrist online information-seeking, but research on information overload is needed. There is less agreement about the appropriate use of other clinical technologies, especially those involving patients. It is too early to tell if technology-based treatments or patient Internet activities will improve the quality of care. The digital divide remains between use of technology for psychiatrists with patients living in urban and rural or suburban areas. Psychiatrists need more formal training in technology to understand risks, benefits and limitations of clinical products.

Evaluation for Primary Hyperparathyroidism in Patients Who Present With Nephrolithiasis

BACKGROUND

The incidence of primary hyperparathyroidism (PHP) is likely underestimated. Nephrolithiasis may indicate PHP with indication for parathyroidectomy. We sought to determine the proportion of patients with an index diagnosis of nephrolithiasis that have serum calcium levels measured, parathyroid hormone (PTH) levels measured if hypercalcemic, and time to referral for definitive management if PHP is diagnosed.

METHODS

A single-institution retrospective review was performed of adult patients presenting with nephrolithiasis between July 1, 2016 and December 31, 2018. Exclusion criteria included currently admitted patients, prior nephrolithiasis, congenital or acquired urinary tract anomalies, and patients on calciuretics. Records were assessed for serum calcium and PTH measurement, as well as referrals. Univariate statistical analysis was performed.

RESULTS

Of 1782 patients with nephrolithiasis screened, 968 met inclusion criteria. Patients were 49.8% female, 88.9% white. Mean age was 53 y. Within this cohort, 620 (64.0%) patients had a calcium measured, with a mean elapsed time from presentation of 27 d (interquartile range [IQR] 0-8). Twelve patients (1.58%) with calcium measured were hypercalcemic and eight (66.7%) had PTH measured with a mean elapsed time from presentation of 183 d (IQR 72-310), all had elevated or non-suppressed PTH. Five (62.5%) were referred to surgeons with mean elapsed referral time of 270 d (IQR 95-492).

CONCLUSIONS

Many with index nephrolithiasis are not assessed for hypercalcemia or hyperparathyroidism. Patients with serum calcium and PTH values indicating PHP diagnosis may have significant delay to parathyroidectomy. Targeted interventions with electronic health record alerts or automated reflex testing may improve care in this domain.

Variation in electronic test results management and its implications for patient safety: A multisite investigation

Objective

The management and follow-up of diagnostic test results is a major patient safety concern. The aim of this qualitative study was to explore how clinicians manage test results on an everyday basis (work-as-done) in a health information technology–enabled emergency department setting. The objectives were to identify (1) variations in work-as-done in test results management and (2) the strategies clinicians use to ensure optimal management of diagnostic test results.

Materials and Methods

Qualitative interviews (n = 26) and field observations were conducted across 3 Australian emergency departments. Interview data coded for results management (ie, tracking, acknowledgment, and follow-up), and artifacts, were reviewed to identify variations in descriptions of work-as-done. Thematic analysis was performed to identify common themes.

Results

Despite using the same test result management application, there were variations in how the system was used. We identified 5 themes relating to electronic test results management: (1) tracking test results, (2) use and understanding of system functionality, (3) visibility of result actions and acknowledgment, (4) results inbox use, and (5) challenges associated with the absence of an inbox for results notifications for advanced practice nurses.

Discussion

Our findings highlight that variations in work-as-done can function to overcome perceived impediments to managing test results in a HIT-enabled environment and thus identify potential risks in the process. By illuminating work-as-done, we identified strategies clinicians use to enhance test result management including paper-based manual processes, cognitive reminders, and adaptive use of electronic medical record functionality.

Conclusions

Test results tracking and follow-up is a priority area in need of health information technology development and training to improve team-based collaboration/communication of results follow-up and diagnostic safety.

Development and validation of prediction rules to target care intensification in veteran patients with diabetes

Background

Diabetes affects 30.3 million people in the USA. Among these people, a major risk factor for microvascular complications is having a glycated haemoglobin (HbA1c) value of ≥75 mmol/mol; therefore, it would be helpful to identify patients who will obtain future HbA1c values of <75 mmol/mol.

Objectives

To develop and validate two prediction rules among patients with diabetes having a baseline HbA1c value of ≥75 mmol/mol: (1) HbA1c measurement ever <75 mmol/mol and (2) final HbA1c measurement of <75 mmol/mol.

Methods

Retrospective cohort study using a registry extracting data from the Department of Veterans Affairs’s (VA’s) electronic health records system. Baseline was 1 Jul 2013–30 June 2014; patients were followed up until 31 July 2016.

Results

Our population consisted of 145 659 patients. Across models, predictors were age, sex, minority status, baseline HbA1c value, time, HbA1c≥75 mmol/mol, receiving insulin treatment and consecutive number of HbA1c values of 75 mmol/mol. The overall likelihood of a patient ever having an HbA1c<75 mmol/mol was 73.65%; with the rule, predicted probabilities were 38.94%, 50.75% and 78.88%. The overall likelihood of patients having a final HbA1c measurement of <75 mmol/mol was 55.35%; the rule provided predicted probabilities of 29.93%, 50.17% and 68.58%.

Conclusions

Within each rule, there were similar observed and predicted tertile probabilities; maintaining HbA1c values of <75 mmol/mol resulted in probability shifts in the majority of patients. We recommend psychosocial screening for 15% of patients for whom there is less than one-third chance of maintaining HbA1c<75 mmol/mol. We plan to conduct additional research to see whether this approach helps.

Assessment of Health Information Technology-Related Outpatient Diagnostic Delays in the US Veterans Affairs Health Care System: A Qualitative Study of Aggregated Root Cause Analysis Data

IMPORTANCE

Diagnostic delay in the outpatient setting is an emerging safety priority that health information technology (HIT) should help address. However, diagnostic delays have persisted, and new safety concerns associated with the use of HIT have emerged.

OBJECTIVE

To analyze HIT-related outpatient diagnostic delays within a large, integrated health care system.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study involved qualitative content analysis of safety concerns identified in aggregated root cause analysis (RCA) data related to HIT and outpatient diagnostic delays. The setting was the US Department of Veterans Affairs using all RCAs submitted to the Veterans Affairs (VA) National Center for Patient Safety from January 1, 2013, to July 31, 2018.

MAIN OUTCOMES AND MEASURES

Common themes associated with the role of HIT-related safety concerns were identified and categorized according to the Health IT Safety framework for measuring, monitoring, and improving HIT safety. This framework includes 3 related domains (ie, safe HIT, safe use of HIT, and using HIT to improve safety) situated within an 8-dimensional sociotechnical model accounting for interacting technical and nontechnical variables associated with safety. Hence, themes identified enhanced understanding of the sociotechnical context and domain of HIT safety involved.

RESULTS

Of 214 RCAs categorized by the terms delay and outpatient submitted during the study period, 88 were identified as involving diagnostic delays and HIT, from which 172 unique HIT-related safety concerns were extracted (mean [SD], 1.97 [1.53] per RCA). Most safety concerns (82.6% [142 of 172]) involved problems with safe use of HIT, predominantly sociotechnical factors associated with people, workflow and communication, and a poorly designed human-computer interface. Fewer safety concerns involved problems with safe HIT (14.5% [25 of 172]) or using HIT to improve safety (0.3% [5 of 172]). The following 5 key high-risk areas for diagnostic delays emerged: managing electronic health record inbox notifications and communication, clinicians gathering key diagnostic information, technical problems, data entry problems, and failure of a system to track test results.

CONCLUSIONS AND RELEVANCE

This qualitative study of a national RCA data set suggests that interventions to reduce outpatient diagnostic delays could aim to improve test result management, interoperability, data visualization, and order entry, as well as to decrease information overload.

Association of Electronic Health Record Use With Physician Fatigue and Efficiency

IMPORTANCE

The use of electronic health records (EHRs) is directly associated with physician burnout. An underlying factor associated with burnout may be EHR-related fatigue owing to insufficient user-centered interface design and suboptimal usability.

OBJECTIVE

To examine the association between EHR use and fatigue, as measured by pupillometry, and efficiency, as measured by mouse clicks, time, and number of EHR screens, among intensive care unit (ICU) physicians completing a simulation activity in a prominent EHR.

DESIGN, SETTING, AND PARTICIPANTS

A cross-sectional, simulation-based EHR usability assessment of a leading EHR system was conducted from March 20 to April 5, 2018, among 25 ICU physicians and physician trainees at a southeastern US academic medical center. Participants completed 4 simulation patient cases in the EHR that involved information retrieval and task execution while wearing eye-tracking glasses. Fatigue was quantified through continuous eye pupil data; EHR efficiency was characterized through task completion time, mouse clicks, and EHR screen visits. Data were analyzed from June 1, 2018, to August 31, 2019.

MAIN OUTCOMES AND MEASURES

Primary outcomes were physician fatigue, measured by pupillometry (with lower scores indicating greater fatigue), and EHR efficiency, measured by task completion times, number of mouse clicks, and number of screens visited during EHR simulation.

RESULTS

The 25 ICU physicians (13 women; mean [SD] age, 32.1 [6.1] years) who completed a simulation exercise involving 4 patient cases (mean [SD] completion time, 34:43 [11:41] minutes) recorded a total of 14 hours and 27 minutes of EHR activity. All physician participants experienced physiological fatigue at least once during the exercise, and 20 of 25 participants (80%) experienced physiological fatigue within the first 22 minutes of EHR use. Physicians who experienced EHR-related fatigue in 1 patient case were less efficient in the subsequent patient case, as demonstrated by longer task completion times (r = -0.521; P = .007), higher numbers of mouse clicks (r = -0.562; P = .003), and more EHR screen visits (r = -0.486; P = .01).

CONCLUSIONS AND RELEVANCE

This study reports high rates of fatigue among ICU physicians during short periods of EHR simulation, which were negatively associated with EHR efficiency and included a carryover association across patient cases. More research is needed to investigate the underlying causes of EHR-associated fatigue, to support user-centered EHR design, and to inform safe EHR use policies and guidelines.

Frequency of Passive EHR Alerts in the ICU: Another Form of Alert Fatigue?

The intensive care unit (ICU) is a complex environment in terms of data density and alerts, with alert fatigue, a recognized barrier to patient safety. The Electronic Health Record (EHR) is a major source of these alerts. Although studies have looked at the incidence and impact of active EHR alerts, little research has studied the impact of passive data alerts on patient safety.

Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow

The overwhelming number of medication alerts generated by clinical decision support systems (CDSS) has led to inappropriate alert overrides, which may lead to unintended patient harm. This review highlights the factors affecting the alert appropriateness of CDSS and barriers to the fit of CDSS alert with clinical workflow. A literature review was conducted to identify features and functions pertinent to CDSS alert appropriateness using the five rights of CDSS. Moreover, a process improvement method, namely, Lean, was used as a tool to optimise clinical workflows, and the appropriate design for CDSS alert using a human automation interaction (HAI) model was recommended. Evaluating the appropriateness of CDSS alert and its impact on workflow provided insights into how alerts can be designed and triggered effectively to support clinical workflow. The application of Lean methods and tools to analyse alert efficiencies in supporting workflow in this study provides an in-depth understanding of alert-workflow fit problems and their root cause, which is required for improving CDSS design. The application of the HAI model is recommended in the design of CDSS alerts to support various levels and stages of alert automations, namely, information acquisition and analysis, decision action and action implementation.

Leveraging Eye Tracking to Prioritize Relevant Medical Record Data: Comparative Machine Learning Study

BACKGROUND

Electronic medical record (EMR) systems capture large amounts of data per patient and present that data to physicians with little prioritization. Without prioritization, physicians must mentally identify and collate relevant data, an activity that can lead to cognitive overload. To mitigate cognitive overload, a Learning EMR (LEMR) system prioritizes the display of relevant medical record data. Relevant data are those that are pertinent to a context-defined as the combination of the user, clinical task, and patient case. To determine which data are relevant in a specific context, a LEMR system uses supervised machine learning models of physician information-seeking behavior. Since obtaining information-seeking behavior data via manual annotation is slow and expensive, automatic methods for capturing such data are needed.

OBJECTIVE

The goal of the research was to propose and evaluate eye tracking as a high-throughput method to automatically acquire physician information-seeking behavior useful for training models for a LEMR system.

METHODS

Critical care medicine physicians reviewed intensive care unit patient cases in an EMR interface developed for the study. Participants manually identified patient data that were relevant in the context of a clinical task: preparing a patient summary to present at morning rounds. We used eye tracking to capture each physician's gaze dwell time on each data item (eg, blood glucose measurements). Manual annotations and gaze dwell times were used to define target variables for developing supervised machine learning models of physician information-seeking behavior. We compared the performance of manual selection and gaze-derived models on an independent set of patient cases.

RESULTS

A total of 68 pairs of manual selection and gaze-derived machine learning models were developed from training data and evaluated on an independent evaluation data set. A paired Wilcoxon signed-rank test showed similar performance of manual selection and gaze-derived models on area under the receiver operating characteristic curve (P=.40).

CONCLUSIONS

We used eye tracking to automatically capture physician information-seeking behavior and used it to train models for a LEMR system. The models that were trained using eye tracking performed like models that were trained using manual annotations. These results support further development of eye tracking as a high-throughput method for training clinical decision support systems that prioritize the display of relevant medical record data.

Association of the Meaningful Use Electronic Health Record Incentive Program With Health Information Technology Venture Capital Funding

Importance

Although the Health Information Technology for Economic and Clinical Health (HITECH) Act has accelerated electronic health record (EHR) adoption since its passage, clinician satisfaction with EHRs remains low, and the association of HITECH with health care information technology (IT) entrepreneurship has remained largely unstudied.

Objective

To determine whether the passage of the HITECH Act was associated with an increase in key measures of health care IT entrepreneurship.

Design, Setting, and Participants

This economic evaluation of venture capital (VC) activity in the US from 2000 to 2019 examined funding trends in health care IT, EHR-related companies, and all VC investments before and after the passage of HITECH. A difference-in-differences analysis compared investments in health care IT companies with those of companies in 3 categories: general health care (non-IT), IT (non-health care), and all US VC transactions. Data were analyzed from September 2018 to August 2019.

Exposures

Venture capital funding received by US companies before and after the HITECH Act.

Main Outcomes and Measures

Venture capital investment in health care IT companies and the proportion of those investments going to seed-stage companies, a proxy for very early-stage entrepreneurship and innovation.

Results

The data included 70 982 investments, of which 9425 (13.3%) were seed stage, 10 706 (15.1%) were early stage, and 50 851 (71.6%) were growth stage. After passage of the HITECH Act, investment in both health care IT companies and EHR-related companies increased at a rate much faster (13.0% and 11.4%, respectively) than VC as a whole (6.9%). In addition, the proportion of investments going to seed-stage health care IT companies increased compared with both overall VC investments and non-IT health care investments. Health care IT companies saw increased probabilities of transactions being seed-stage of 5.1% (SE, 2.2%; 95% CI, 0.8% to 9.3%; P = .02) compared with the entire sample of VC transactions and 13.6% (SE, 1.9%; 95% CI, 9.9% to 17.2%; P < .001) compared with non-IT health care VC transactions. Health care IT had essentially 0 increased probability of a transaction being seed stage compared with IT companies outside health care (-0.8% probability; SE, 2.4%; 95% CI, -5.4% to 3.9%; P = .75).

Conclusions and Relevance

Although widespread clinician dissatisfaction with EHR systems remains a challenge, the HITECH Act's incentive program may have catalyzed early-stage entrepreneurship in health care IT, suggesting an important role for incentives in promoting innovation.