1
|
Sittig DF, Wright A. A guide to mitigating audit log-related risk in medical professional liability cases. J Healthc Risk Manag 2023; 43:37-47. [PMID: 37486791 DOI: 10.1002/jhrm.21553] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 06/13/2023] [Indexed: 07/26/2023] [Imported: 08/02/2023]
Abstract
Following the American Recovery and Reinvestment Act in 2009, use of electronic health records (EHRs) has become ubiquitous. Accordingly, one should expect most medical professional liability cases to involve review of patient records produced from EHRs. When questions arise regarding who was involved in care of a patient, what they knew and when, or the meaning, completeness, integrity, validity, timeliness, confidentiality, accuracy, or legitimacy of data, or ways that the EHR's user interface or automated clinical decision support tools may have contributed to the alleged events, one often turns to the EHR and its audit log. This manuscript discusses lines of defense incorporated into the design, development, implementation, and use of EHRs to ensure their integrity and the types of EHR transaction logs (e.g., audit log) that exist. Using these logs can help one answer questions that often arise in medical malpractice cases. Finally, there are "best practices" surrounding EHR audit logs that health care organizations should implement. When used appropriately, EHRs and their audit logs provide another source of information to help hospital risk managers, legal counsel, and EHR expert witnesses to investigate adverse incidents and, if needed, prosecute or defend clinicians and/or health care organizations involved in the patient's care.
Collapse
Affiliation(s)
- Dean F Sittig
- Center for Healthcare Quality & Safety, McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
- Informatics-Review LLC, Lake Oswego, Oregon, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
2
|
Murphy DR, Zimolzak AJ, Upadhyay DK, Wei L, Jolly P, Offner A, Sittig DF, Korukonda S, Rekha RM, Singh H. Developing electronic clinical quality measures to assess the cancer diagnostic process. J Am Med Inform Assoc 2023; 30:1526-1531. [PMID: 37257883 PMCID: PMC10436145 DOI: 10.1093/jamia/ocad089] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2022] [Revised: 04/12/2023] [Accepted: 05/08/2023] [Indexed: 06/02/2023] [Imported: 08/02/2023] Open
Abstract
OBJECTIVE Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance. MATERIALS AND METHODS We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors. RESULTS We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157). CONCLUSIONS We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.
Collapse
Affiliation(s)
- Daniel R Murphy
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Andrew J Zimolzak
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Divvy K Upadhyay
- Division of Quality, Safety and Patient Experience, Geisinger, Danville, Pennsylvania, USA
| | - Li Wei
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Preeti Jolly
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Alexis Offner
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Dean F Sittig
- Department of Clinical and Health Informatics, The University of Texas Health Science Center at Houston’s School of Biomedical Informatics, Houston, Texas, USA
- The UT-Memorial Hermann Center for Healthcare Quality & Safety, Houston, Texas, USA
| | - Saritha Korukonda
- Investigator-Initiated Research Operations, Geisinger, Danville, Pennsylvania, USA
| | - Riyaa Murugaesh Rekha
- Division of Quality, Safety and Patient Experience, Geisinger, Danville, Pennsylvania, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| |
Collapse
|
3
|
Sittig DF, Boxwala A, Wright A, Zott C, Desai P, Dhopeshwarkar R, Swiger J, Lomotan EA, Dobes A, Dullabh P. A lifecycle framework illustrates eight stages necessary for realizing the benefits of patient-centered clinical decision support. J Am Med Inform Assoc 2023; 30:1583-1589. [PMID: 37414544 PMCID: PMC10436138 DOI: 10.1093/jamia/ocad122] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Revised: 06/06/2023] [Accepted: 06/23/2023] [Indexed: 07/08/2023] [Imported: 08/02/2023] Open
Abstract
The design, development, implementation, use, and evaluation of high-quality, patient-centered clinical decision support (PC CDS) is necessary if we are to achieve the quintuple aim in healthcare. We developed a PC CDS lifecycle framework to promote a common understanding and language for communication among researchers, patients, clinicians, and policymakers. The framework puts the patient, and/or their caregiver at the center and illustrates how they are involved in all the following stages: Computable Clinical Knowledge, Patient-specific Inference, Information Delivery, Clinical Decision, Patient Behaviors, Health Outcomes, Aggregate Data, and patient-centered outcomes research (PCOR) Evidence. Using this idealized framework reminds key stakeholders that developing, deploying, and evaluating PC-CDS is a complex, sociotechnical challenge that requires consideration of all 8 stages. In addition, we need to ensure that patients, their caregivers, and the clinicians caring for them are explicitly involved at each stage to help us achieve the quintuple aim.
Collapse
Affiliation(s)
- Dean F Sittig
- McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| | | | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Courtney Zott
- NORC at the University of Chicago, Bethesda, Maryland, USA
| | - Priyanka Desai
- NORC at the University of Chicago, Bethesda, Maryland, USA
| | | | - James Swiger
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - Edwin A Lomotan
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - Angela Dobes
- Crohn’s & Colitis Foundation, New York, New York, USA
| | | |
Collapse
|
4
|
Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, McCoy AB, Sittig DF, Wright A. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc 2023:7136722. [PMID: 37087108 DOI: 10.1093/jamia/ocad072] [Citation(s) in RCA: 49] [Impact Index Per Article: 49.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 03/28/2023] [Accepted: 04/11/2023] [Indexed: 04/24/2023] [Imported: 08/02/2023] Open
Abstract
OBJECTIVE To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. RESULTS Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. CONCLUSION AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Collapse
Affiliation(s)
- Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Aileen P Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Barron L Patterson
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Jonathan P Wanderer
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Robert W Turer
- Department of Emergency Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA
- Clinical Informatics Center, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Scott D Nelson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
5
|
Sittig DF, Wright A. Identifying a Clinical Informatics or Electronic Health Record Expert Witness for Medical Professional Liability Cases. Appl Clin Inform 2023; 14:290-295. [PMID: 36706791 PMCID: PMC10033222 DOI: 10.1055/a-2018-9932] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2022] [Accepted: 01/21/2023] [Indexed: 01/29/2023] [Imported: 08/02/2023] Open
Abstract
BACKGROUND The health care field is experiencing widespread electronic health record (EHR) adoption. New medical professional liability (i.e., malpractice) cases will likely involve the review of data extracted from EHRs as well as EHR workflows, audit logs, and even the potential role of the EHR in causing harm. OBJECTIVES Reviewing printed versions of a patient's EHRs can be difficult due to differences in printed versus on-screen presentations, redundancies, and the way printouts are often grouped by document or information type rather than chronologically. Simply recreating an accurate timeline often requires experts with training and experience in designing, developing, using, and reviewing EHRs and audit logs. Additional expertise is required if questions arise about data's meaning, completeness, accuracy, and timeliness or ways that the EHR's user interface or automated clinical decision support tools may have contributed to alleged events. Such experts often come from the sociotechnical field of clinical informatics that studies the design, development, implementation, use, and evaluation of information and communications technology, specifically, EHRs. Identifying well-qualified EHR experts to aid a legal team is challenging. METHODS Based on literature review and experience reviewing cases, we identified seven criteria to help in this assessment. RESULTS The criteria are education in clinical informatics; clinical informatics knowledge; experience with EHR design, development, implementation, and use; communication skills; academic publications on clinical informatics; clinical informatics certification; and membership in informatics-related professional organizations. CONCLUSION While none of these criteria are essential, understanding the breadth and depth of an individual's qualifications in each of these areas can help identify a high-quality, clinical informatics expert witness.
Collapse
Affiliation(s)
- Dean F. Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, United States
- Informatics-Review LLC, Lake Oswego, Oregon, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| |
Collapse
|
6
|
Wright A, Schreiber R, Bates DW, Aaron S, Ai A, Cholan RA, Desai A, Divo M, Dorr DA, Hickman TT, Hussain S, Just S, Koh B, Lipsitz S, Mcevoy D, Rosenbloom T, Russo E, Ting DYC, Weitkamp A, Sittig DF. A multi-site randomized trial of a clinical decision support intervention to improve problem list completeness. J Am Med Inform Assoc 2023; 30:899-906. [PMID: 36806929 PMCID: PMC10114117 DOI: 10.1093/jamia/ocad020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 01/31/2023] [Accepted: 02/14/2023] [Indexed: 02/22/2023] [Imported: 08/02/2023] Open
Abstract
OBJECTIVE To improve problem list documentation and care quality. MATERIALS AND METHODS We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.
Collapse
Affiliation(s)
- Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Digital, Mass General Brigham, Boston, Massachusetts, USA.,HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Richard Schreiber
- Physician Informatics and Department of Internal Medicine, Penn State Health Holy Spirit Medical Center, Camp Hill, Pennsylvania, USA
| | - David W Bates
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Skye Aaron
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Angela Ai
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Raja Arul Cholan
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon, USA
| | - Akshay Desai
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Miguel Divo
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - David A Dorr
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon, USA
| | - Thu-Trang Hickman
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Community Health, Mass General Brigham, Boston, Massachusetts, USA
| | - Salman Hussain
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Shari Just
- HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Brian Koh
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Stuart Lipsitz
- Department of Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Dustin Mcevoy
- Digital, Mass General Brigham, Boston, Massachusetts, USA
| | - Trent Rosenbloom
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Elise Russo
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | | | - Asli Weitkamp
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| |
Collapse
|
7
|
Cram P, Cram E, Antos J, Sittig DF, Anand A, Li Y. Changes in Availability of and Prices for Shoppable Services at US News and World Report Honor Roll Hospitals: a Longitudinal Cross-Sectional Study. J Gen Intern Med 2023; 38:542-544. [PMID: 35790664 PMCID: PMC9905366 DOI: 10.1007/s11606-022-07719-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/13/2022] [Accepted: 06/17/2022] [Indexed: 10/17/2022] [Imported: 08/02/2023]
Affiliation(s)
- Peter Cram
- Department of Internal Medicine, University of Texas Medical Branch, Galveston, TX, USA.
| | | | - Joseph Antos
- American Enterprise Institute, Washington, DC, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Ajay Anand
- Goergen Institute for Data Science, University of Rochester, Rochester, NY, USA
| | - Yue Li
- Department of Public Health Sciences, University of Rochester, Rochester, NY, USA
| |
Collapse
|
8
|
Morris AH, Horvat C, Stagg B, Grainger DW, Lanspa M, Orme J, Clemmer TP, Weaver LK, Thomas FO, Grissom CK, Hirshberg E, East TD, Wallace CJ, Young MP, Sittig DF, Suchyta M, Pearl JE, Pesenti A, Bombino M, Beck E, Sward KA, Weir C, Phansalkar S, Bernard GR, Thompson BT, Brower R, Truwit J, Steingrub J, Hiten RD, Willson DF, Zimmerman JJ, Nadkarni V, Randolph AG, Curley MAQ, Newth CJL, Lacroix J, Agus MSD, Lee KH, deBoisblanc BP, Moore FA, Evans RS, Sorenson DK, Wong A, Boland MV, Dere WH, Crandall A, Facelli J, Huff SM, Haug PJ, Pielmeier U, Rees SE, Karbing DS, Andreassen S, Fan E, Goldring RM, Berger KI, Oppenheimer BW, Ely EW, Pickering BW, Schoenfeld DA, Tocino I, Gonnering RS, Pronovost PJ, Savitz LA, Dreyfuss D, Slutsky AS, Crapo JD, Pinsky MR, James B, Berwick DM. Computer clinical decision support that automates personalized clinical care: a challenging but needed healthcare delivery strategy. J Am Med Inform Assoc 2022; 30:178-194. [PMID: 36125018 PMCID: PMC9748596 DOI: 10.1093/jamia/ocac143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2022] [Revised: 07/27/2022] [Accepted: 08/22/2022] [Indexed: 12/15/2022] [Imported: 08/02/2023] Open
Abstract
How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.
Collapse
Affiliation(s)
- Alan H Morris
- Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Christopher Horvat
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Brian Stagg
- Department of Ophthalmology and Visual Sciences, Moran Eye Center, University of Utah, Salt Lake City, Utah, USA
| | - David W Grainger
- Department of Biomedical Engineering, University of Utah, Salt Lake City, Utah, USA
| | - Michael Lanspa
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - James Orme
- Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Terry P Clemmer
- Department of Internal Medicine (Critical Care), Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Lindell K Weaver
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Frank O Thomas
- Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA
| | - Colin K Grissom
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Ellie Hirshberg
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Thomas D East
- SYNCRONYS - Chief Executive Officer, Albuquerque, New Mexico, USA
| | - Carrie Jane Wallace
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Michael P Young
- Department of Critical Care, Renown Regional Medical Center, Reno, Nevada, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Mary Suchyta
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - James E Pearl
- Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA
- Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Antinio Pesenti
- Faculty of Medicine and Surgery—Anesthesiology, University of Milan, Milano, Lombardia, Italy
| | - Michela Bombino
- Department of Emergency and Intensive Care, San Gerardo Hospital, Monza (MB), Italy
| | - Eduardo Beck
- Faculty of Medicine and Surgery - Anesthesiology, University of Milan, Ospedale di Desio, Desio, Lombardia, Italy
| | - Katherine A Sward
- Department of Biomedical Informatics, College of Nursing, University of Utah, Salt Lake City, Utah, USA
| | - Charlene Weir
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
| | - Shobha Phansalkar
- Wolters Kluwer Health—Clinical Solutions—Medical Informatics, Wolters Kluwer Health, Newton, Massachusetts, USA
| | - Gordon R Bernard
- Vanderbilt University School of Medicine, Nashville, Tennessee, USA
| | - B Taylor Thompson
- Pulmonary and Critical Care Division, Department of Internal Medicine, Harvard Medical School, Boston, Massachusetts, USA
| | - Roy Brower
- Department of Medicine, Johns Hopkins School of Medicine, Baltimore, Maryland, USA
| | - Jonathon Truwit
- Department of Internal Medicine, Pulmonary and Critical Care, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Jay Steingrub
- Department of Internal Medicine, Pulmonary and Critical Care, University of Massachusetts Medical School, Baystate Campus, Springfield, Massachusetts, USA
| | - R Duncan Hiten
- Department of Internal Medicine, Pulmonary and Critical Care, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Douglas F Willson
- Pediatric Critical Care, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia, USA
| | - Jerry J Zimmerman
- Division of Pediatric Critical Care Medicine, Department of Pediatrics, University of Washington School of Medicine, Seattle, Washington, USA
| | - Vinay Nadkarni
- Department of Anesthesiology and Critical Care Medicine, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | - Adrienne G Randolph
- Department of Anesthesiology, Critical Care and Pain Medicine, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Martha A Q Curley
- University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, USA
| | - Christopher J L Newth
- Childrens Hospital Los Angeles, Department of Anesthesiology and Critical Care, University of Southern California Keck School of Medicine, Los Angeles, California, USA
| | - Jacques Lacroix
- Division of Pediatric Critical Care Medicine, Department of Pediatrics, Université de Montréal Faculté de Médecine, Montreal, Quebec, Canada
| | - Michael S D Agus
- Division of Medical Pediatric Critical Care, Department of Pediatrics, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Kang Hoe Lee
- Department of Intensive Care Medicine, Ng Teng Fong Hospital and National University Centre of Transplantation, National University Singapore Yong Loo Lin School of Medicine, Singapore
| | - Bennett P deBoisblanc
- Department of Internal Medicine, Pulmonary and Critical Care, Louisiana State University Health Sciences Center, New Orleans, Louisiana, USA
| | - Frederick Alan Moore
- Department of Surgery, University of Florida College of Medicine, Gainesville, Florida, USA
| | - R Scott Evans
- Department of Medical Informatics, Intermountain Healthcare, and Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
| | - Dean K Sorenson
- Department of Medical Informatics, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Anthony Wong
- Department of Data Science Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Illinois, USA
| | - Michael V Boland
- Department of Ophthalmology, Massachusetts Ear and Eye Infirmary, Harvard Medical School, Boston, Massachusetts, USA
| | - Willard H Dere
- Endocrinology and Metabolism Division, Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA
| | - Alan Crandall
- Department of Ophthalmology and Visual Sciences, Moran Eye Center, University of Utah, Salt Lake City, Utah, USA
- Posthumous
| | - Julio Facelli
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
| | - Stanley M Huff
- Department of Medical Informatics, Intermountain Healthcare, Department of Biomedical Informatics, University of Utah, and Graphite Health, Salt Lake City, Utah, USA
| | - Peter J Haug
- Department of Medical Informatics, Intermountain Healthcare, and Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
| | - Ulrike Pielmeier
- Aalborg University Faculty of Engineering and Science - Department of Health Science and Technology, Respiratory and Critical Care Group, Aalborg, Nordjylland, Denmark
| | - Stephen E Rees
- Aalborg University Faculty of Engineering and Science - Department of Health Science and Technology, Respiratory and Critical Care Group, Aalborg, Nordjylland, Denmark
| | - Dan S Karbing
- Aalborg University Faculty of Engineering and Science - Department of Health Science and Technology, Respiratory and Critical Care Group, Aalborg, Nordjylland, Denmark
| | - Steen Andreassen
- Aalborg University Faculty of Engineering and Science - Department of Health Science and Technology, Respiratory and Critical Care Group, Aalborg, Nordjylland, Denmark
| | - Eddy Fan
- Internal Medicine, Pulmonary and Critical Care Division, Institute of Health Policy, Management and Evaluation, University of Toronto Faculty of Medicine, Toronto, Ontario, Canada
| | - Roberta M Goldring
- Department of Internal Medicine, Pulmonary and Critical Care, New York University School of Medicine, New York, New York, USA
| | - Kenneth I Berger
- Department of Internal Medicine, Pulmonary and Critical Care, New York University School of Medicine, New York, New York, USA
| | - Beno W Oppenheimer
- Department of Internal Medicine, Pulmonary and Critical Care, New York University School of Medicine, New York, New York, USA
| | - E Wesley Ely
- Internal Medicine, Pulmonary and Critical Care, Critical Illness, Brain Dysfunction, and Survivorship (CIBS) Center, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Tennessee Valley Veteran’s Affairs Geriatric Research Education Clinical Center (GRECC), Nashville, Tennessee, USA
| | - Brian W Pickering
- Department of Anesthesiology, Mayo Clinic, Rochester, Minnesota, USA
| | - David A Schoenfeld
- Biostatistics Center, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Irena Tocino
- Department of Radiology, Yale University School of Medicine, New Haven, Connecticut, USA
| | - Russell S Gonnering
- Department of Ophthalmology and Visual Sciences, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Peter J Pronovost
- Department of Anesthesiology and Critical Care Medicine, University Hospitals, Highland Hills, Case Western Reserve University School of Medicine, Cleveland, Ohio, USA
| | - Lucy A Savitz
- Northwest Center for Health Research, Kaiser Permanente, Oakland, California, USA
| | - Didier Dreyfuss
- Assistance Publique—Hôpitaux de Paris, Université de Paris, Sorbonne Université - INSERM unit UMR S_1155 (Common and Rare Kidney Diseases), Paris, France
| | - Arthur S Slutsky
- Interdepartmental Division of Critical Care Medicine, Keenan Research Center, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, University of Toronto, Toronto, Ontario, Canada
| | - James D Crapo
- Department of Internal Medicine, National Jewish Health, Denver, Colorado, USA
| | - Michael R Pinsky
- Department of Critical Care Medicine, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USA
| | - Brent James
- Department of Internal Medicine, Clinical Excellence Research Center (CERC), Stanford University School of Medicine, Stanford, California, USA
| | - Donald M Berwick
- Institute for Healthcare Improvement, Cambridge, Massachusetts, USA
| |
Collapse
|
9
|
Sittig DF, Sherman JD, Eckelman MJ, Draper A, Singh H. i-CLIMATE: a "clinical climate informatics" action framework to reduce environmental pollution from healthcare. J Am Med Inform Assoc 2022; 29:2153-2160. [PMID: 35997550 PMCID: PMC9667163 DOI: 10.1093/jamia/ocac137] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2022] [Revised: 06/21/2022] [Accepted: 08/08/2022] [Indexed: 11/12/2022] [Imported: 08/02/2023] Open
Abstract
Addressing environmental pollution and climate change is one of the biggest sociotechnical challenges of our time. While information technology has led to improvements in healthcare, it has also contributed to increased energy usage, destructive natural resource extraction, piles of e-waste, and increased greenhouse gases. We introduce a framework "Information technology-enabled Clinical cLimate InforMAtics acTions for the Environment" (i-CLIMATE) to illustrate how clinical informatics can help reduce healthcare's environmental pollution and climate-related impacts using 5 actionable components: (1) create a circular economy for health IT, (2) reduce energy consumption through smarter use of health IT, (3) support more environmentally friendly decision-making by clinicians and health administrators, (4) mobilize healthcare workforce environmental stewardship through informatics, and (5) Inform policies and regulations for change. We define Clinical Climate Informatics as a field that applies data, information, and knowledge management principles to operationalize components of the i-CLIMATE Framework.
Collapse
Affiliation(s)
- Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Jodi D Sherman
- Department of Anesthesiology, Yale School of Medicine, New Haven, Connecticut, USA
- Department of Environmental Sciences, Center on Climate Change and Health, Yale School of Public Health, New Haven, Connecticut, USA
| | - Matthew J Eckelman
- Department of Civil & Environmental Engineering, Northeastern University, Boston, Massachusetts, USA
| | - Andrew Draper
- Health Data Informatics and Analytics, University of Denver, HCA Continental Division, GreenCIO.org, Denver, Colorado, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| |
Collapse
|
10
|
Jing X, Indani A, Hubig N, Min H, Gong Y, Cimino JJ, Sittig DF, Rennert L, Robinson D, Biondich P, Wright A, Nøhr C, Law T, Faxvaag A, Gimbel R. A Systematic Approach to Configuring MetaMap for Optimal Performance. Methods Inf Med 2022; 61:e51-e63. [PMID: 35613942 PMCID: PMC9788913 DOI: 10.1055/a-1862-0421] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
BACKGROUND MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward. OBJECTIVE To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance. METHODS MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated. RESULTS The percentages of exact matches and missing gold-standard terms were 0.6-0.79 and 0.09-0.3 for one behavior option, and 0.56-0.8 and 0.09-0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively. CONCLUSION We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.
Collapse
Affiliation(s)
- Xia Jing
- Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson University, Clemson, South Carolina, United States,Address for correspondence Xia Jing, MD, PhD Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson UniversityEdwards Hall 511, Clemson, SC 29634United States
| | - Akash Indani
- School of Computing, College of Engineering, Computing and Applied Sciences, Clemson University, Clemson, South Carolina, United States
| | - Nina Hubig
- School of Computing, College of Engineering, Computing and Applied Sciences, Clemson University, Clemson, South Carolina, United States
| | - Hua Min
- Department of Health Administration and Policy, College of Health and Human Services, George Mason University, Fairfax, Virginia, United States
| | - Yang Gong
- School of Biomedical Informatics, The University of Texas Health Sciences Center at Houston, Houston, Texas, United States
| | - James J. Cimino
- Informatics Institute, The University of Alabama at Birmingham, Birmingham, Alabama, United States
| | - Dean F. Sittig
- School of Biomedical Informatics, The University of Texas Health Sciences Center at Houston, Houston, Texas, United States
| | - Lior Rennert
- Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson University, Clemson, South Carolina, United States
| | | | - Paul Biondich
- Department of Pediatrics, Clem McDonald Biomedical Informatics Center, Regenstrief Institute, Indiana University School of Medicine, Indianapolis, Indiana, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Christian Nøhr
- Department of Planning, Faculty of Engineering, Aalborg University, Aalborg, Denmark
| | - Timothy Law
- Ohio Musculoskeletal and Neurologic Institute, Ohio University, Athens, Ohio, United States
| | - Arild Faxvaag
- Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology, Trondheim, Norway
| | - Ronald Gimbel
- Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson University, Clemson, South Carolina, United States
| |
Collapse
|
11
|
Wright A, Nelson S, Rubins D, Schreiber R, Sittig DF. Clinical decision support malfunctions related to medication routes: a case series. J Am Med Inform Assoc 2022; 29:1972-1975. [PMID: 36040207 PMCID: PMC9552204 DOI: 10.1093/jamia/ocac150] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2022] [Revised: 08/08/2022] [Accepted: 08/25/2022] [Indexed: 11/28/2022] [Imported: 08/02/2023] Open
Abstract
Objective To identify common medication route-related causes of clinical decision support (CDS) malfunctions and best practices for avoiding them. Materials and Methods Case series of medication route-related CDS malfunctions from diverse healthcare provider organizations. Results Nine cases were identified and described, including both false-positive and false-negative alert scenarios. A common cause was the inclusion of nonsystemically available medication routes in value sets (eg, eye drops, ear drops, or topical preparations) when only systemically available routes were appropriate. Discussion These value set errors are common, occur across healthcare provider organizations and electronic health record (EHR) systems, affect many different types of medications, and can impact the accuracy of CDS interventions. New knowledge management tools and processes for auditing existing value sets and supporting the creation of new value sets can mitigate many of these issues. Furthermore, value set issues can adversely affect other aspects of the EHR, such as quality reporting and population health management. Conclusion Value set issues related to medication routes are widespread and can lead to CDS malfunctions. Organizations should make appropriate investments in knowledge management tools and strategies, such as those outlined in our recommendations.
Collapse
Affiliation(s)
- Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Scott Nelson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - David Rubins
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Harvard Medical School, Boston, Massachusetts, USA.,Partners eCare, Partners HealthCare, Boston, Massachusetts, USA
| | - Richard Schreiber
- Penn State Health Holy Spirit Hospital Medical Center, Camp Hill, Pennsylvania, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| |
Collapse
|
12
|
Zimolzak AJ, Singh H, Murphy DR, Wei L, Memon SA, Upadhyay DK, Korukonda S, Zubkoff L, Sittig DF. Translating electronic health record-based patient safety algorithms from research to clinical practice at multiple sites. BMJ Health Care Inform 2022; 29:bmjhci-2022-100565. [PMID: 35851287 PMCID: PMC9289019 DOI: 10.1136/bmjhci-2022-100565] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2022] [Accepted: 06/19/2022] [Indexed: 01/07/2023] [Imported: 08/02/2023] Open
Abstract
Introduction Researchers are increasingly developing algorithms that impact patient care, but algorithms must also be implemented in practice to improve quality and safety. Objective We worked with clinical operations personnel at two US health systems to implement algorithms to proactively identify patients without timely follow-up of abnormal test results that warrant diagnostic evaluation for colorectal or lung cancer. We summarise the steps involved and lessons learned. Methods Twelve sites were involved across two health systems. Implementation involved extensive software documentation, frequent communication with sites and local validation of results. Additionally, we used automated edits of existing code to adapt it to sites’ local contexts. Results All sites successfully implemented the algorithms. Automated edits saved sites significant work in direct code modification. Documentation and communication of changes further aided sites in implementation. Conclusion Patient safety algorithms developed in research projects were implemented at multiple sites to monitor for missed diagnostic opportunities. Automated algorithm translation procedures can produce more consistent results across sites.
Collapse
Affiliation(s)
- Andrew J Zimolzak
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Daniel R Murphy
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Li Wei
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Sahar A Memon
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
- Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
| | - Divvy K Upadhyay
- Division of Quality, Safety and Patient Experience, Geisinger, Danville, PA, USA
| | | | - Lisa Zubkoff
- Geriatric Research Education and Clinical Center, Birmingham VA Medical Center, Birmingham, Alabama, USA
- Division of Preventive Medicine, The University of Alabama at Birmingham, Birmingham, Alabama, USA
| | - Dean F Sittig
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA
| |
Collapse
|
13
|
Dullabh P, Heaney-Huls K, Hovey L, Sandberg S, Lobach DF, Boxwala A, Desai P, Sittig DF. The Technology Landscape of Patient-Centered Clinical Decision Support - Where Are We and What Is Needed? Stud Health Technol Inform 2022; 290:350-353. [PMID: 35673033 DOI: 10.3233/shti220094] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] [Imported: 08/02/2023]
Abstract
Patient Centered Outcomes Research (PCOR) and health care delivery system transformation require investments in development of tools and techniques for rapid dissemination of clinical and operational best practices. This paper explores the current technology landscape for patient-centered clinical decision support (PC CDS) and what is needed to make it more shareable, standards-based, and publicly available with the goal of improving patient care and clinical outcomes. The landscape assessment used three sources of information: (1) a 22-member technical expert panel; (2) a literature review of peer-reviewed and grey literature; and (3) key informant interviews with PC CDS stakeholders. We identified ten salient technical considerations that span all phases of PC CDS development; our findings suggest there has been significant progress in the development and implementation of PC CDS but challenges remain.
Collapse
Affiliation(s)
| | | | - Lauren Hovey
- NORC at the University of Chicago, Bethesda, MD, United States
| | - Shana Sandberg
- NORC at the University of Chicago, Bethesda, MD, United States
| | | | - Aziz Boxwala
- Elimu Informatics, El Cerrito, CA, United States
| | - Priyanka Desai
- NORC at the University of Chicago, Bethesda, MD, United States
| | - Dean F Sittig
- University of Texas Health Science Center, Houston, TX, United States
| |
Collapse
|
14
|
Cifra CL, Tigges CR, Miller SL, Curl N, Monson CD, Dukes KC, Reisinger HS, Pennathur PR, Sittig DF, Singh H. Reporting Outcomes of Pediatric Intensive Care Unit Patients to Referring Physicians via an Electronic Health Record-Based Feedback System. Appl Clin Inform 2022; 13:495-503. [PMID: 35545126 PMCID: PMC9095343 DOI: 10.1055/s-0042-1748147] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
BACKGROUND Many critically ill children are initially evaluated in front-line settings by clinicians with variable pediatric training before they are transferred to a pediatric intensive care unit (PICU). Because clinicians learn from past performance, communicating outcomes of patients back to front-line clinicians who provide pediatric emergency care could be valuable; however, referring clinicians do not consistently receive this important feedback. OBJECTIVES Our aim was to determine the feasibility, usability, and clinical relevance of a semiautomated electronic health record (EHR)-supported system developed at a single institution to deliver timely and relevant PICU patient outcome feedback to referring emergency department (ED) physicians. METHODS Guided by the Health Information Technology Safety Framework, we iteratively designed, implemented, and evaluated a semiautomated electronic feedback system leveraging the EHR in one institution. After conducting interviews and focus groups with stakeholders to understand the PICU-ED health care work system, we designed the EHR-supported feedback system by translating stakeholder, organizational, and usability objectives into feedback process and report requirements. Over 6 months, we completed three cycles of implementation and evaluation, wherein we analyzed EHR access logs, reviewed feedback reports sent, performed usability testing, and conducted physician interviews to determine the system's feasibility, usability, and clinical relevance. RESULTS The EHR-supported feedback process is feasible with timely delivery and receipt of feedback reports. Usability testing revealed excellent Systems Usability Scale scores. According to physicians, the process was well-integrated into their clinical workflows and conferred minimal additional workload. Physicians also indicated that delivering and receiving consistent feedback was relevant to their clinical practice. CONCLUSION An EHR-supported system to deliver timely and relevant PICU patient outcome feedback to referring ED physicians was feasible, usable, and important to physicians. Future work is needed to evaluate impact on clinical practice and patient outcomes and to investigate applicability to other clinical settings involved in similar care transitions.
Collapse
Affiliation(s)
- Christina L Cifra
- Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, Iowa, United States
| | - Cody R Tigges
- Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, Iowa, United States
| | - Sarah L Miller
- Department of Emergency Medicine, University of Iowa Carver College of Medicine, Iowa City, Iowa, United States
| | - Nathaniel Curl
- Emergency Medicine, UnityPoint Health-Trinity Medical Center, Rock Island, Illinois, United States
| | - Christopher D Monson
- Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, Iowa, United States
| | - Kimberly C Dukes
- Center for Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center, Iowa City, Iowa, United States.,Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City, Iowa, United States
| | - Heather S Reisinger
- Center for Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center, Iowa City, Iowa, United States.,Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City, Iowa, United States.,Institute for Clinical and Translational Science, University of Iowa, Iowa City, Iowa, United States
| | - Priyadarshini R Pennathur
- Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City, Iowa, United States.,Department of Industrial and Systems Engineering, College of Engineering, University of Iowa, Iowa City, Iowa, United States
| | - Dean F Sittig
- School of Biomedical Informatics, Center for Healthcare Quality and Safety, University of Texas Health Science Center, Houston, Texas, United States
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness, and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, United States
| |
Collapse
|
15
|
Sittig DF, Lakhani P, Singh H. Applying requisite imagination to safeguard electronic health record transitions. J Am Med Inform Assoc 2022; 29:1014-1018. [PMID: 35022741 PMCID: PMC9006683 DOI: 10.1093/jamia/ocab291] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2021] [Revised: 12/17/2021] [Accepted: 12/29/2021] [Indexed: 02/05/2023] Open
Abstract
Over the next decade, many health care organizations (HCOs) will transition from one electronic health record (EHR) to another; some forced by hospital acquisition and others by choice in search of better EHRs. Herein, we apply principles of Requisite Imagination, or the ability to imagine key aspects of the future one is planning, to offer 6 recommendations on how to proactively safeguard these transitions. First, HCOs should implement a proactive leadership structure that values communication. Second, HCOs should implement proactive risk assessment and testing processes. Third, HCOs should anticipate and reduce unwarranted variation in their EHR and clinical processes. Fourth, HCOs should establish a culture of conscious inquiry with routine system monitoring. Fifth, HCOs should foresee and reduce information access problems. Sixth, HCOs should support their workforce through difficult EHR transitions. Proactive approaches using Requisite Imagination principles outlined here can help ensure safe, effective, and economically sound EHR transitions.
Collapse
Affiliation(s)
- Dean F Sittig
- University of Texas/Memorial Hermann Center for Healthcare Quality & Safety, School of Biomedical Informatics, University of Texas Health Science Center, Houston, TX, USA
| | - Priti Lakhani
- Formerly at Office of Electronic Health Record Modernization, U.S. Department of Veterans Affairs, Washington, DC, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, TX, USA
| |
Collapse
|
16
|
Dullabh P, Heaney-Huls K, Lobach DF, Hovey LS, Sandberg SF, Desai PJ, Lomotan E, Swiger J, Harrison MI, Dymek C, Sittig DF, Boxwala A. The technical landscape for patient-centered CDS: progress, gaps, and challenges. J Am Med Inform Assoc 2022; 29:1101-1105. [PMID: 35263437 PMCID: PMC9093031 DOI: 10.1093/jamia/ocac029] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2021] [Revised: 01/31/2022] [Accepted: 02/23/2022] [Indexed: 02/05/2023] Open
Abstract
Supporting healthcare decision-making that is patient-centered and evidence-based requires investments in the development of tools and techniques for dissemination of patient-centered outcomes research findings via methods such as clinical decision support (CDS). This article explores the technical landscape for patient-centered CDS (PC CDS) and the gaps in making PC CDS more shareable, standards-based, and publicly available, with the goal of improving patient care and clinical outcomes. This landscape assessment used: (1) a technical expert panel; (2) a literature review; and (3) interviews with 18 CDS stakeholders. We identified 7 salient technical considerations that span 5 phases of PC CDS development. While progress has been made in the technical landscape, the field must advance standards for translating clinical guidelines into PC CDS, the standardization of CDS insertion points into the clinical workflow, and processes to capture, standardize, and integrate patient-generated health data.
Collapse
Affiliation(s)
| | | | | | - Lauren S Hovey
- NORC at the University of Chicago, Bethesda, Maryland, USA
| | | | | | - Edwin Lomotan
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - James Swiger
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - Michael I Harrison
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - Chris Dymek
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | | |
Collapse
|
17
|
Affiliation(s)
- Dean F Sittig
- University of Texas/Memorial Hermann Center for Healthcare Quality and Safety, School of Biomedical Informatics, University of Texas Health Science Center at Houston
| | - Patricia Sengstack
- Vanderbilt University School of Nursing, Frist Nursing Informatics Center, Nashville, Tennessee
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness, and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas
| |
Collapse
|
18
|
Keebler JR, Rosen MA, Sittig DF, Thomas E, Salas E. Human Factors and Ergonomics in Healthcare: Industry Demands and a Path Forward. Hum Factors 2022; 64:250-258. [PMID: 35000407 DOI: 10.1177/00187208211073623] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
This article reviews three industry demands that will impact the future of Human Factors and Ergonomics in Healthcare settings. These demands include the growing population of older adults, the increasing use of telemedicine, and a focus on patient-centered care. Following, we discuss a path forward through improved medical teams, error management, and safety testing of medical devices and tools. Future challenges are discussed.
Collapse
Affiliation(s)
- Joseph R Keebler
- Department of Human Factors and Behavioral Neurobiology, 2830Embry-Riddle Aeronautical University, Daytona Beach, FL, USA
| | - Michael A Rosen
- Department of Anesthesiology, Armstrong Institute for Patient Safety, Johns Hopkins University, Baltimore, MD, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Eric Thomas
- UT Health Memorial Center for Healthcare Quality and Safety, McGovern Medical School, University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Eduardo Salas
- Department of Psychological Sciences, Rice University, Houston, TX, USA
| |
Collapse
|
19
|
Keebler JR, Salas E, Rosen MA, Sittig DF, Thomas E. Preface: Special Issue on Human Factors in Healthcare. Hum Factors 2022; 64:5. [PMID: 34986657 DOI: 10.1177/00187208211073577] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Affiliation(s)
- Joseph R Keebler
- Department of Human Factors and Behavioral Neurobiology, 2830Embry-Riddle Aeronautical University, Daytona Beach, FL, USA
| | - Eduardo Salas
- Department of Psychological Sciences, 3990Rice University, Houston, TX, USA
| | - Michael A Rosen
- Department of Anesthesiology, Armstrong Institute for Patient Safety, 1466Johns Hopkins University, Baltimore, MD, USA
| | - Dean F Sittig
- School of Biomedical Informatics, 182519University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Eric Thomas
- Healthcare Quality and Safety, McGovern Medical School, 182519University of Texas Health Science Center at Houston, Houston, TX, USA
| |
Collapse
|
20
|
Abstract
Patient care in intensive care environments is complex, time-sensitive, and data-rich, factors that make these settings particularly well-suited to clinical decision support (CDS). A wide range of CDS interventions have been used in intensive care unit environments. The field needs well-designed studies to identify the most effective CDS approaches. Evolving artificial intelligence and machine learning models may reduce information-overload and enable teams to take better advantage of the large volume of patient data available to them. It is vital to effectively integrate new CDS into clinical workflows and to align closely with the cognitive processes of frontline clinicians.
Collapse
Affiliation(s)
- Robert El-Kareh
- University of California, San Diego, 9500 Gilman Drive, #0881 La Jolla, CA 92093-0881, USA.
| | - Dean F Sittig
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, UT-Memorial Hermann Center for Healthcare Quality & Safety, Houston, TX 77030, USA. https://twitter.com/DeanSittig
| |
Collapse
|
21
|
Affiliation(s)
- Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Sciences Center at Houston, Houston, Texas, United States
| | - Carolyn Petersen
- Mayo Clinic College of Medicine and Science, Rochester, Minnesota, United States
| | - Stephen M Downs
- Center for Biomedical Informatics, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
| | - Jenna S Lehmann
- Applied Clinical Informatics Editorial Office, Nashville, Tennessee, United States
| | - Christoph U Lehmann
- Applied Clinical Informatics Editorial Office, Nashville, Tennessee, United States.,Clinical Informatics Center, UT Southwestern, Dallas, Texas, United States
| |
Collapse
|
22
|
Dullabh P, Sandberg SF, Heaney-Huls K, Hovey LS, Lobach DF, Boxwala A, Desai PJ, Berliner E, Dymek C, Harrison MI, Swiger J, Sittig DF. OUP accepted manuscript. J Am Med Inform Assoc 2022; 29:1233-1243. [PMID: 35534996 PMCID: PMC9196686 DOI: 10.1093/jamia/ocac059] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2021] [Revised: 03/12/2022] [Accepted: 04/22/2022] [Indexed: 02/05/2023] Open
Abstract
Objective We conducted a horizon scan to (1) identify challenges in patient-centered clinical decision support (PC CDS) and (2) identify future directions for PC CDS. Materials and Methods We engaged a technical expert panel, conducted a scoping literature review, and interviewed key informants. We qualitatively analyzed literature and interview transcripts, mapping findings to the 4 phases for translating evidence into PC CDS interventions (Prioritizing, Authoring, Implementing, and Measuring) and to external factors. Results We identified 12 challenges for PC CDS development. Lack of patient input was identified as a critical challenge. The key informants noted that patient input is critical to prioritizing topics for PC CDS and to ensuring that CDS aligns with patients’ routine behaviors. Lack of patient-centered terminology standards was viewed as a challenge in authoring PC CDS. We found a dearth of CDS studies that measured clinical outcomes, creating significant gaps in our understanding of PC CDS’ impact. Across all phases of CDS development, there is a lack of patient and provider trust and limited attention to patients’ and providers’ concerns. Discussion These challenges suggest opportunities for advancing PC CDS. There are opportunities to develop industry-wide practices and standards to increase transparency, standardize terminologies, and incorporate patient input. There is also opportunity to engage patients throughout the PC CDS research process to ensure that outcome measures are relevant to their needs. Conclusion Addressing these challenges and embracing these opportunities will help realize the promise of PC CDS—placing patients at the center of the healthcare system.
Collapse
Affiliation(s)
- Prashila Dullabh
- Corresponding Author: Prashila Dullabh, MD, NORC at the University of Chicago, 4350 East-West Hwy 8th Floor, Bethesda, MD 20814, USA;
| | | | | | - Lauren S Hovey
- NORC at the University of Chicago, Bethesda, Maryland, USA
| | | | | | | | | | - Chris Dymek
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - Michael I Harrison
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - James Swiger
- Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| |
Collapse
|
23
|
Mahajan P, Mollen C, Alpern ER, Baird-Cox K, Boothman RC, Chamberlain JM, Cosby K, Epstein HM, Gegenheimer-Holmes J, Gerardi M, Giardina TD, Patel VL, Ruddy R, Saleem J, Shaw KN, Sittig DF, Singh H. An Operational Framework to Study Diagnostic Errors in Emergency Departments: Findings From A Consensus Panel. J Patient Saf 2021; 17:570-575. [PMID: 31790012 DOI: 10.1097/pts.0000000000000624] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
OBJECTIVE To create an operational definition and framework to study diagnostic error in the emergency department setting. METHODS We convened a 17-member multidisciplinary panel with expertise in general and pediatric emergency medicine, nursing, patient safety, informatics, cognitive psychology, social sciences, human factors, and risk management and a patient/caregiver advocate. We used a modified nominal group technique to develop a shared understanding to operationally define diagnostic errors in emergency care and modify the National Academies of Sciences, Engineering, and Medicine's conceptual process framework to this setting. RESULTS The expert panel defined diagnostic errors as "a divergence from evidence-based processes that increases the risk of poor outcomes despite the availability of sufficient information to provide a timely and accurate explanation of the patient's health problem(s)." Diagnostic processes include tasks related to (a) acuity recognition, information and synthesis, evaluation coordination, and (b) communication with patients/caregivers and other diagnostic team members. The expert panel also modified the National Academies of Sciences, Engineering, and Medicine's diagnostic process framework to incorporate influence of mode of arrival, triage level, and interventions during emergency care and underscored the importance of outcome feedback to emergency department providers to promote learning and improvement related to diagnosis. CONCLUSIONS The proposed operational definition and modified diagnostic process framework can potentially inform the development of measurement tools and strategies to study the epidemiology and interventions to improve emergency care diagnosis.
Collapse
Affiliation(s)
| | - Cynthia Mollen
- Division of Pediatric Emergency Medicine, Department Pediatrics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - Elizabeth R Alpern
- Division of Pediatric Emergency Medicine, Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois
| | | | - Richard C Boothman
- Department of Surgery, University of Michigan Medical School, Ann Arbor, Michigan
| | - James M Chamberlain
- Division of Pediatric Emergency Medicine, Department of Pediatrics, Children's National Health System, Washington, District of Columbia
| | - Karen Cosby
- Emergency Medicine, Cook County Hospital (Stroger) and Rush Medical School, Chicago, Illinois
| | - Helene M Epstein
- Member of the Board of Directors, Brightpoint Care, New York, New York
| | | | - Michael Gerardi
- Emergency Medicine, Morristown Medical Center and Goryeb Children's Hospital, Morristown, New Jersey
| | - Traber D Giardina
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas
| | - Vimla L Patel
- Center for Cognitive Studies in Medicine and Public Health, The New York Academy of Medicine, New York, New York
| | - Richard Ruddy
- University of Cincinnati College of Medicine, Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
| | - Jason Saleem
- Industrial Engineering, University of Louisville, Louisville, Kentucky
| | - Kathy N Shaw
- Division of Pediatric Emergency Medicine, Department Pediatrics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas
| |
Collapse
|
24
|
Cram P, Cram E, Antos J, Sittig DF, Anand A, Li Y. Availability of prices for shoppable services on hospital internet sites. Am J Manag Care 2021; 27:e426-e428. [PMID: 34889585 DOI: 10.37765/ajmc.2021.88690] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
OBJECTIVES A regulation from CMS required that, starting January 1, 2021, all US hospitals publicly display the cash price and minimum and minimum negotiated charge for 300 "shoppable services." We evaluated compliance with CMS requirements among highly respected US hospitals. STUDY DESIGN We conducted a cross-sectional study of hospital websites. METHODS We evaluated the public websites of the 20 hospitals listed in the 2020-2021 US News & World Report honor roll between February 1 and February 14, 2021. We selected 2 imaging studies (brain MRI and abdominal ultrasound) and 3 hospital services (cardiac valve surgery, total joint replacement, and vaginal childbirth). For each service and hospital, we determined whether the discounted cash price and minimum negotiated charge were displayed and, if displayed, what the prices were. RESULTS Among our 20 hospitals, 13 (65%) displayed the cash prices for the MRI and ultrasound, 8 (40%) for valve surgery, 10 (50%) for joint replacement, and 10 (50%) for childbirth. Only 1 (5%) displayed the minimum negotiated price for the 2 imaging studies and none for any of the hospital services. The mean (range) cash price for MRI was $3793 ($464-$6215) and for ultrasound was $767 ($136-$1391). The mean (range) cash price for cardiac surgery was $236,125 ($72,250-$349,782); for joint replacement, $46,008 ($22,170-$71,985); and for childbirth, $19,568 ($7314-$29,068). CONCLUSIONS In an early assessment, a significant percentage of US hospitals were not in compliance with new price transparency legislation. Moreover, there is wide variation in prices among hospitals for identical services. These price differences suggest the potential for significant cost savings for patients.
Collapse
Affiliation(s)
- Peter Cram
- Department of Internal Medicine, University of Texas Medical Branch, 301 University Blvd, Ste 4-124, Rte 0569, Galveston, TX 77555-0569.
| | | | | | | | | | | |
Collapse
|
25
|
Affiliation(s)
- Dean F Sittig
- University of Texas/Memorial Hermann Center for Healthcare Quality and Safety and School of Biomedical Informatics, University of Texas Health Science Center, Houston
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, and Baylor College of Medicine, Houston, Texas
| |
Collapse
|
26
|
Murphy DR, Savoy A, Satterly T, Sittig DF, Singh H. Dashboards for visual display of patient safety data: a systematic review. BMJ Health Care Inform 2021; 28:bmjhci-2021-100437. [PMID: 34615664 PMCID: PMC8496385 DOI: 10.1136/bmjhci-2021-100437] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2021] [Accepted: 09/22/2021] [Indexed: 02/05/2023] Open
Abstract
Background Methods to visualise patient safety data can support effective monitoring of safety events and discovery of trends. While quality dashboards are common, use and impact of dashboards to visualise patient safety event data remains poorly understood. Objectives To understand development, use and direct or indirect impacts of patient safety dashboards. Methods We conducted a systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched PubMed, EMBASE and CINAHL for publications between 1 January 1950 and 30 August 2018 involving use of dashboards to display data related to safety targets defined by the Agency for Healthcare Research and Quality’s Patient Safety Net. Two reviewers independently reviewed search results for inclusion in analysis and resolved disagreements by consensus. We collected data on development, use and impact via standardised data collection forms and analysed data using descriptive statistics. Results Literature search identified 4624 results which were narrowed to 33 publications after applying inclusion and exclusion criteria and consensus across reviewers. Publications included only time series and case study designs and were inpatient focused and emergency department focused. Information on direct impact of dashboards was limited, and only four studies included informatics or human factors principles in development or postimplementation evaluation. Discussion Use of patient-safety dashboards has grown over the past 15 years, but impact remains poorly understood. Dashboard design processes rarely use informatics or human factors principles to ensure that the available content and navigation assists task completion, communication or decision making. Conclusion Design and usability evaluation of patient safety dashboards should incorporate informatics and human factors principles. Future assessments should also rigorously explore their potential to support patient safety monitoring including direct or indirect impact on patient safety.
Collapse
Affiliation(s)
- Daniel R Murphy
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
| | - April Savoy
- Purdue School of Engineering and Technology, Indiana University Purdue University at Indianapolis, Indianapolis, Indiana, USA.,Department of Veterans Affairs, Richard L Roudebush VA Medical Center, Indianapolis, Indiana, USA.,Center for Health Services Research, Regenstrief Institute, Inc, Indianapolis, Indiana, USA
| | - Tyler Satterly
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
| | - Dean F Sittig
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, Texas, USA .,The UT-Memorial Hermann Center for Healthcare Quality & Safety, Houston, Texas, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey VA Medical Center, Houston, Texas, USA
| |
Collapse
|
27
|
Vaghani V, Wei L, Mushtaq U, Sittig DF, Bradford A, Singh H. Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments. J Am Med Inform Assoc 2021; 28:2202-2211. [PMID: 34279630 DOI: 10.1093/jamia/ocab121] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2021] [Revised: 05/26/2021] [Accepted: 06/23/2021] [Indexed: 02/05/2023] Open
Abstract
OBJECTIVE Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs). METHODS Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit. RESULTS We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%). CONCLUSION In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.
Collapse
Affiliation(s)
- Viralkumar Vaghani
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Li Wei
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Umair Mushtaq
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Dean F Sittig
- University of Texas-Memorial Hermann Center for Healthcare Quality & Safety, School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Andrea Bradford
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| |
Collapse
|
28
|
D'Amore JD, McCrary LK, Denson J, Li C, Vitale CJ, Tokachichu P, Sittig DF, McCoy AB, Wright A. Clinical data sharing improves quality measurement and patient safety. J Am Med Inform Assoc 2021; 28:1534-1542. [PMID: 33712850 PMCID: PMC8279795 DOI: 10.1093/jamia/ocab039] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Revised: 01/23/2021] [Accepted: 02/15/2021] [Indexed: 02/05/2023] Open
Abstract
OBJECTIVE Accurate and robust quality measurement is critical to the future of value-based care. Having incomplete information when calculating quality measures can cause inaccuracies in reported patient outcomes. This research examines how quality calculations vary when using data from an individual electronic health record (EHR) and longitudinal data from a health information exchange (HIE) operating as a multisource registry for quality measurement. MATERIALS AND METHODS Data were sampled from 53 healthcare organizations in 2018. Organizations represented both ambulatory care practices and health systems participating in the state of Kansas HIE. Fourteen ambulatory quality measures for 5300 patients were calculated using the data from an individual EHR source and contrasted to calculations when HIE data were added to locally recorded data. RESULTS A total of 79% of patients received care at more than 1 facility during the 2018 calendar year. A total of 12 994 applicable quality measure calculations were compared using data from the originating organization vs longitudinal data from the HIE. A total of 15% of all quality measure calculations changed (P < .001) when including HIE data sources, affecting 19% of patients. Changes in quality measure calculations were observed across measures and organizations. DISCUSSION These results demonstrate that quality measures calculated using single-site EHR data may be limited by incomplete information. Effective data sharing significantly changes quality calculations, which affect healthcare payments, patient safety, and care quality. CONCLUSIONS Federal, state, and commercial programs that use quality measurement as part of reimbursement could promote more accurate and representative quality measurement through methods that increase clinical data sharing.
Collapse
Affiliation(s)
- John D D'Amore
- Informatics Department, Diameter Health, Farmington, Connecticut, USA
| | | | - Jody Denson
- Kansas Health Information Network, Topeka, Kansas, USA
| | - Chun Li
- Informatics Department, Diameter Health, Farmington, Connecticut, USA
| | | | | | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
29
|
Morris AH, Stagg B, Lanspa M, Orme J, Clemmer TP, Weaver LK, Thomas F, Grissom CK, Hirshberg E, East TD, Wallace CJ, Young MP, Sittig DF, Pesenti A, Bombino M, Beck E, Sward KA, Weir C, Phansalkar SS, Bernard GR, Taylor Thompson B, Brower R, Truwit JD, Steingrub J, Duncan Hite R, Willson DF, Zimmerman JJ, Nadkarni VM, Randolph A, Curley MAQ, Newth CJL, Lacroix J, Agus MSD, Lee KH, deBoisblanc BP, Scott Evans R, Sorenson DK, Wong A, Boland MV, Grainger DW, Dere WH, Crandall AS, Facelli JC, Huff SM, Haug PJ, Pielmeier U, Rees SE, Karbing DS, Andreassen S, Fan E, Goldring RM, Berger KI, Oppenheimer BW, Wesley Ely E, Gajic O, Pickering B, Schoenfeld DA, Tocino I, Gonnering RS, Pronovost PJ, Savitz LA, Dreyfuss D, Slutsky AS, Crapo JD, Angus D, Pinsky MR, James B, Berwick D. Enabling a learning healthcare system with automated computer protocols that produce replicable and personalized clinician actions. J Am Med Inform Assoc 2021; 28:1330-1344. [PMID: 33594410 PMCID: PMC8661391 DOI: 10.1093/jamia/ocaa294] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2020] [Accepted: 11/10/2020] [Indexed: 02/05/2023] Open
Abstract
Clinical decision-making is based on knowledge, expertise, and authority, with clinicians approving almost every intervention-the starting point for delivery of "All the right care, but only the right care," an unachieved healthcare quality improvement goal. Unaided clinicians suffer from human cognitive limitations and biases when decisions are based only on their training, expertise, and experience. Electronic health records (EHRs) could improve healthcare with robust decision-support tools that reduce unwarranted variation of clinician decisions and actions. Current EHRs, focused on results review, documentation, and accounting, are awkward, time-consuming, and contribute to clinician stress and burnout. Decision-support tools could reduce clinician burden and enable replicable clinician decisions and actions that personalize patient care. Most current clinical decision-support tools or aids lack detail and neither reduce burden nor enable replicable actions. Clinicians must provide subjective interpretation and missing logic, thus introducing personal biases and mindless, unwarranted, variation from evidence-based practice. Replicability occurs when different clinicians, with the same patient information and context, come to the same decision and action. We propose a feasible subset of therapeutic decision-support tools based on credible clinical outcome evidence: computer protocols leading to replicable clinician actions (eActions). eActions enable different clinicians to make consistent decisions and actions when faced with the same patient input data. eActions embrace good everyday decision-making informed by evidence, experience, EHR data, and individual patient status. eActions can reduce unwarranted variation, increase quality of clinical care and research, reduce EHR noise, and could enable a learning healthcare system.
Collapse
Affiliation(s)
- Alan H Morris
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine
- Department of Biomedical Informatics
| | - Brian Stagg
- Department of Ophthalmology and Visual Sciences and John Moran Eye Center
| | - Michael Lanspa
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - James Orme
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine
- Department of Biomedical Informatics
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Terry P Clemmer
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine
- Department of Biomedical Informatics
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
- Emeritus
| | - Lindell K Weaver
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine
- Department of Biomedical Informatics
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Frank Thomas
- Department of Value Engineering, University of Utah Hospitals and Clinics, Salt Lake City, Utah, USA
- Emeritus
| | - Colin K Grissom
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine
- Department of Biomedical Informatics
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Ellie Hirshberg
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Intermountain Healthcare, Salt Lake City, Utah, USA
| | - Thomas D East
- SYNCRONYS, and University of New Mexico Health Sciences Library & Informatics, Albuquerque, New Mexico, USA
| | - Carrie Jane Wallace
- Department of Ophthalmology and Visual Sciences and John Moran Eye Center
- Emeritus
| | - Michael P Young
- Critical Care Division, Renown Medical Center, School of Medicine, University of Nevada, Reno, Nevada, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Antonio Pesenti
- Dipartimento di Anestesia, Rianimazione ed Emergenza-Urgenza, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico, Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy
| | - Michela Bombino
- Department of Emergency and Intensive Care Medicine, ASST-Monza San Gerardo Hospital, Milan, Italy
| | - Eduardo Beck
- Ospedale di Desio—ASST Monza, UOC Anestesia e Rianimazione, Milan, Italy
| | | | - Charlene Weir
- Department of Biomedical Informatics
- School of Nursing
| | | | - Gordon R Bernard
- Pulmonary, Critical Care, and Allergy Division, Department of Internal Medicine
| | - B Taylor Thompson
- Pulmonary, Critical Care, and Sleep Division , Department of Internal Medicine
| | - Roy Brower
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Jonathon D Truwit
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Jay Steingrub
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, University of Massachusetts Medical School-Baystate, Springfield, Massachusetts, USA
| | - R Duncan Hite
- Pulmonary, Critical Care, and Sleep Division, Department of Internal Medicine, College of Medicine, University of Cincinnati, Cincinnati, Ohio, USA
| | - Douglas F Willson
- Division of Pediatric Critical Care, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia, USA
| | - Jerry J Zimmerman
- Division of Pediatric Critical Care Medicine, Department of Pediatrics, University of Washington School of Medicine, Seattle, Washington, USA
| | - Vinay M Nadkarni
- Department of Anesthesia and Critical Care Medicine
- Department of Pediatrics, Perelman School of Medicine
| | | | - Martha A. Q Curley
- Department of Pediatrics, Perelman School of Medicine
- School of Nursing, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Christopher J. L Newth
- Department of Pediatrics, University of Southern California, Los Angeles, California, USA
| | - Jacques Lacroix
- Division of Pediatric Critical Care Medicine, Department of Pediatrics, CHU Sainte-Justine and Université de Montréal, Montréal, Canada
| | | | - Kang H Lee
- Asian American Liver Centre, Gleneagles Hospital, Singapore, Singapore
| | - Bennett P deBoisblanc
- Section of Pulmonary/Critical Care & Allergy/Immunology, Louisiana State University School of Medicine, New Orleans, Louisiana, USA
| | | | | | - Anthony Wong
- Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois, USA
| | | | - David W Grainger
- Department of Biomedical Engineering and Department of Pharmaceutics and Pharmaceutical Chemistry, University of Utah
| | - Willard H Dere
- Department of Biomedical Engineering and Department of Pharmaceutics and Pharmaceutical Chemistry, University of Utah
| | - Alan S Crandall
- Department of Ophthalmology and Visual Sciences and John Moran Eye Center
| | - Julio C Facelli
- Department of Biomedical Informatics
- Center for Clinical and Translational Science, School of Medicine
| | | | | | - Ulrike Pielmeier
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Stephen E Rees
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Dan S Karbing
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Steen Andreassen
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Eddy Fan
- Institute of Health Policy, Management and Evaluation
| | - Roberta M Goldring
- Pulmonary, Critical Care, and Sleep Division, NYU School of Medicine, New York, New York, USA
| | - Kenneth I Berger
- Pulmonary, Critical Care, and Sleep Division, NYU School of Medicine, New York, New York, USA
| | - Beno W Oppenheimer
- Pulmonary, Critical Care, and Sleep Division, NYU School of Medicine, New York, New York, USA
| | - E Wesley Ely
- Pulmonary, Critical Care, and Allergy Division, Department of Internal Medicine
- Critical Illness, Brain Dysfunction, and Survivorship (CIBS) Center, Vanderbilt University Medical Center
- Tennessee Valley Veterans Affairs Geriatric Research Education Clinical Center (GRECC), Nashville, Tennessee, USA
| | - Ognjen Gajic
- Pulmonary , Critical Care, and Sleep Division, Department of Internal Medicine
| | - Brian Pickering
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic School of Medicine, Rochester, Minnesota, USA
| | - David A Schoenfeld
- Department of Biostatistics, T.H. Chan School of Public Health, Harvard Medical School, Boston, Massachusetts, USA
| | - Irena Tocino
- Department of Radiology, Yale University School of Medicine, New Haven, Connecticut, USA
| | - Russell S Gonnering
- Department of Ophthalmology and Visual Sciences, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Peter J Pronovost
- Critical Care, Department of Anesthesia, Chief Clinical Transformation Officer, University Hospitals, Highland Hills, Case Western Reserve University, Cleveland, OH, USA
| | - Lucy A Savitz
- Kaiser Permanente Northwest Center for Health Research, Portland, OR, USA
| | - Didier Dreyfuss
- Assistance Publique – Hôpitaux de Paris, Université de Paris, INSERM unit UMR S_1155 (Common and Rare Kidney Diseases), Sorbonne Université, Paris, France
| | - Arthur S Slutsky
- Keenan Research Center, Li Ka Shing Knowledge Institute / ST. Michaels' Hospital and Interdepartmental Division of Critical Care Medicine, University of Toronto, Toronto, ON, Canada
| | - James D Crapo
- Department of Internal Medicine, National Jewish Health, Denver, Colorado, USA
| | - Derek Angus
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Michael R Pinsky
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Brent James
- Clinical Excellence Research Center (CERC), Department of Medicine, Stanford University School of Medicine, Palo Alto, California, USA
| | - Donald Berwick
- Institute for Healthcare Improvement, Boston, Massachusetts, USA
| |
Collapse
|
30
|
Lehmann CU, Lehmann JS, Petersen C, Sittig DF. To Applied Clinical Informatics Authors and Reviewers: Thank You for All Your Help! Appl Clin Inform 2021; 12:417-424. [PMID: 33979875 DOI: 10.1055/s-0041-1729958] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Affiliation(s)
| | - Jenna S Lehmann
- Applied Clinical Informatics Editorial Office, Nashville, Tennessee, United States
| | - Carolyn Petersen
- Mayo Clinic College of Medicine and Science, Rochester, Minnesota, United States
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Sciences Center at Houston, Houston, Texas, United States
| |
Collapse
|
31
|
Cifra CL, Sittig DF, Singh H. Bridging the feedback gap: a sociotechnical approach to informing clinicians of patients' subsequent clinical course and outcomes. BMJ Qual Saf 2021; 30:591-597. [PMID: 33958442 PMCID: PMC8237185 DOI: 10.1136/bmjqs-2020-012464] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2020] [Revised: 03/27/2021] [Accepted: 04/28/2021] [Indexed: 02/05/2023]
Affiliation(s)
- Christina L Cifra
- Department of Pediatrics, University of Iowa Roy J and Lucille A Carver College of Medicine, Iowa City, Iowa, USA
| | - Dean F Sittig
- School of Biomedical Informatics, Center for Healthcare Quality and Safety, University of Texas Health Science Center, Houston, Texas, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| |
Collapse
|
32
|
Petersen C, Smith J, Freimuth RR, Goodman KW, Jackson GP, Kannry J, Liu H, Madhavan S, Sittig DF, Wright A. Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper. J Am Med Inform Assoc 2021; 28:677-684. [PMID: 33447854 DOI: 10.1093/jamia/ocaa319] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2020] [Accepted: 12/01/2020] [Indexed: 02/07/2023] Open
Abstract
The development and implementation of clinical decision support (CDS) that trains itself and adapts its algorithms based on new data-here referred to as Adaptive CDS-present unique challenges and considerations. Although Adaptive CDS represents an expected progression from earlier work, the activities needed to appropriately manage and support the establishment and evolution of Adaptive CDS require new, coordinated initiatives and oversight that do not currently exist. In this AMIA position paper, the authors describe current and emerging challenges to the safe use of Adaptive CDS and lay out recommendations for the effective management and monitoring of Adaptive CDS.
Collapse
Affiliation(s)
- Carolyn Petersen
- Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, Minnesota, USA
| | - Jeffery Smith
- The Office of the National Coordinator for Health Information Technology, Washington, DC, USA
| | - Robert R Freimuth
- Division of Digital Health Sciences, Center for Individualized Medicine, Mayo Clinic, Rochester, Minnesota, USA
| | - Kenneth W Goodman
- Institute for Bioethics and Health Policy, University of Miami Miller School of Medicine, Miami, Florida, USA
| | - Gretchen Purcell Jackson
- IBM Watson Health, Cambridge, Massachusetts, USA.,Department of Pediatric Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Joseph Kannry
- Mount Sinai Health System, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Hongfang Liu
- Division of Digital Health Sciences, Mayo Clinic, Rochester, Minnesota, USA
| | - Subha Madhavan
- Department of Oncology, Georgetown Lombardi Comprehensive Cancer Center, Innovation Center for Biomedical Informatics, Georgetown University Medical Center, Washington, DC, USA
| | - Dean F Sittig
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, UT-Memorial Hermann Center for Healthcare Quality & Safety, Houston, Texas, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
33
|
Wright A, Aaron S, McCoy AB, El-Kareh R, Fort D, Kassakian SZ, Longhurst CA, Malhotra S, McEvoy DS, Monsen CB, Schreiber R, Weitkamp AO, Willett DL, Sittig DF. Algorithmic Detection of Boolean Logic Errors in Clinical Decision Support Statements. Appl Clin Inform 2021; 12:182-189. [PMID: 33694144 DOI: 10.1055/s-0041-1722918] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
OBJECTIVE Clinical decision support (CDS) can contribute to quality and safety. Prior work has shown that errors in CDS systems are common and can lead to unintended consequences. Many CDS systems use Boolean logic, which can be difficult for CDS analysts to specify accurately. We set out to determine the prevalence of certain types of Boolean logic errors in CDS statements. METHODS Nine health care organizations extracted Boolean logic statements from their Epic electronic health record (EHR). We developed an open-source software tool, which implemented the Espresso logic minimization algorithm, to identify three classes of logic errors. RESULTS Participating organizations submitted 260,698 logic statements, of which 44,890 were minimized by Espresso. We found errors in 209 of them. Every participating organization had at least two errors, and all organizations reported that they would act on the feedback. DISCUSSION An automated algorithm can readily detect specific categories of Boolean CDS logic errors. These errors represent a minority of CDS errors, but very likely require correction to avoid patient safety issues. This process found only a few errors at each site, but the problem appears to be widespread, affecting all participating organizations. CONCLUSION Both CDS implementers and EHR vendors should consider implementing similar algorithms as part of the CDS authoring process to reduce the number of errors in their CDS interventions.
Collapse
Affiliation(s)
- Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States.,Division of General Internal Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States.,Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, United States.,Partners eCare, Partners HealthCare System, Boston, Massachusetts, United States
| | - Skye Aaron
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Robert El-Kareh
- Department of Medicine, UC San Diego Health, University of California, San Diego, San Diego, California, United States
| | - Daniel Fort
- Center for Outcomes and Health Services Research, Ochsner Health System, New Orleans, Louisiana, United States
| | - Steven Z Kassakian
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, United States
| | - Christopher A Longhurst
- Department of Medicine, UC San Diego Health, University of California, San Diego, San Diego, California, United States
| | - Sameer Malhotra
- Department of Healthcare Policy and Research, Weill Cornell Medicine, New York, New York, United States.,Department of Internal Medicine, NewYork-Presbyterian Hospital, New York, New York, United States
| | - Dustin S McEvoy
- Partners eCare, Partners HealthCare System, Boston, Massachusetts, United States
| | - Craig B Monsen
- Center for Informatics, Atrius Health, Boston, Massachusetts, United States
| | - Richard Schreiber
- Physician Informatics and Department of Internal Medicine, Geisinger Holy Spirit, Camp Hill, Pennsylvania, United States
| | - Asli O Weitkamp
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - DuWayne L Willett
- Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, United States
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, United States
| |
Collapse
|
34
|
Tajgardoon M, Cooper GF, King AJ, Clermont G, Hochheiser H, Hauskrecht M, Sittig DF, Visweswaran S. Modeling physician variability to prioritize relevant medical record information. JAMIA Open 2020; 3:602-610. [PMID: 33623894 PMCID: PMC7886572 DOI: 10.1093/jamiaopen/ooaa058] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2020] [Revised: 10/05/2020] [Accepted: 11/02/2020] [Indexed: 02/05/2023] Open
Abstract
Objective Patient information can be retrieved more efficiently in electronic medical record (EMR) systems by using machine learning models that predict which information a physician will seek in a clinical context. However, information-seeking behavior varies across EMR users. To explicitly account for this variability, we derived hierarchical models and compared their performance to nonhierarchical models in identifying relevant patient information in intensive care unit (ICU) cases. Materials and methods Critical care physicians reviewed ICU patient cases and selected data items relevant for presenting at morning rounds. Using patient EMR data as predictors, we derived hierarchical logistic regression (HLR) and standard logistic regression (LR) models to predict their relevance. Results In 73 pairs of HLR and LR models, the HLR models achieved an area under the receiver operating characteristic curve of 0.81, 95% confidence interval (CI) [0.80-0.82], which was statistically significantly higher than that of LR models (0.75, 95% CI [0.74-0.76]). Further, the HLR models achieved statistically significantly lower expected calibration error (0.07, 95% CI [0.06-0.08]) than LR models (0.16, 95% CI [0.14-0.17]). Discussion The physician reviewers demonstrated variability in selecting relevant data. Our results show that HLR models perform significantly better than LR models with respect to both discrimination and calibration. This is likely due to explicitly modeling physician-related variability. Conclusion Hierarchical models can yield better performance when there is physician-related variability as in the case of identifying relevant information in the EMR.
Collapse
Affiliation(s)
- Mohammadamin Tajgardoon
- Intelligent Systems Program, School of Computing and Information, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Gregory F Cooper
- Intelligent Systems Program, School of Computing and Information, University of Pittsburgh, Pittsburgh, Pennsylvania, USA.,Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Andrew J King
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Gilles Clermont
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Harry Hochheiser
- Intelligent Systems Program, School of Computing and Information, University of Pittsburgh, Pittsburgh, Pennsylvania, USA.,Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Milos Hauskrecht
- Intelligent Systems Program, School of Computing and Information, University of Pittsburgh, Pittsburgh, Pennsylvania, USA.,Department of Computer Science, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Dean F Sittig
- Department of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| | - Shyam Visweswaran
- Intelligent Systems Program, School of Computing and Information, University of Pittsburgh, Pittsburgh, Pennsylvania, USA.,Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| |
Collapse
|
35
|
Rogith D, Satterly T, Singh H, Sittig DF, Russo E, Smith MW, Roosan D, Bhise V, Murphy DR. Application of Human Factors Methods to Understand Missed Follow-up of Abnormal Test Results. Appl Clin Inform 2020; 11:692-698. [PMID: 33086395 DOI: 10.1055/s-0040-1716537] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
OBJECTIVE This study demonstrates application of human factors methods for understanding causes for lack of timely follow-up of abnormal test results ("missed results") in outpatient settings. METHODS We identified 30 cases of missed test results by querying electronic health record data, developed a critical decision method (CDM)-based interview guide to understand decision-making processes, and interviewed physicians who ordered these tests. We analyzed transcribed responses using a contextual inquiry (CI)-based methodology to identify contextual factors contributing to missed results. We then developed a CI-based flow model and conducted a fault tree analysis (FTA) to identify hierarchical relationships between factors that delayed action. RESULTS The flow model highlighted barriers in information flow and decision making, and the hierarchical model identified relationships between contributing factors for delayed action. Key findings including underdeveloped methods to track follow-up, as well as mismatches, in communication channels, timeframes, and expectations between patients and physicians. CONCLUSION This case report illustrates how human factors-based approaches can enable analysis of contributing factors that lead to missed results, thus informing development of preventive strategies to address them.
Collapse
Affiliation(s)
- Deevakar Rogith
- The University of Texas Health Science Center at Houston School of Biomedical Informatics, Houston, Texas, United States
| | - Tyler Satterly
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veteran Affairs Medical Center, Houston, Texas, United States.,Department of Medicine, Section of Health Services Research, Baylor College of Medicine, Houston, Texas, United States
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veteran Affairs Medical Center, Houston, Texas, United States.,Department of Medicine, Section of Health Services Research, Baylor College of Medicine, Houston, Texas, United States
| | - Dean F Sittig
- The University of Texas Health Science Center at Houston School of Biomedical Informatics, Houston, Texas, United States.,UT-Memorial Hermann Center for Healthcare Quality and Safety, Houston, Texas, United States
| | - Elise Russo
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Michael W Smith
- Department of Industrial and Mechanical Engineering, Universidad de las Americas Puebla, Cholula, Mexico
| | - Don Roosan
- Department of Pharmacy Practice and Administration, College of Pharmacy Western University of Health Sciences, Pomona, California, United States
| | - Viraj Bhise
- Department of Medicine, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire, United States
| | - Daniel R Murphy
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veteran Affairs Medical Center, Houston, Texas, United States.,Department of Medicine, Section of Health Services Research, Baylor College of Medicine, Houston, Texas, United States
| |
Collapse
|
36
|
Singh H, Sittig DF, Gandhi TK. Fighting a common enemy: a catalyst to close intractable safety gaps. BMJ Qual Saf 2020; 30:141-145. [PMID: 32675326 PMCID: PMC7841492 DOI: 10.1136/bmjqs-2020-011390] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2020] [Revised: 06/16/2020] [Accepted: 06/19/2020] [Indexed: 02/05/2023]
Affiliation(s)
- Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | | |
Collapse
|
37
|
Sittig DF, Ash JS, Wright A, Chase D, Gebhardt E, Russo EM, Tercek C, Mohan V, Singh H. How can we partner with electronic health record vendors on the complex journey to safer health care? J Healthc Risk Manag 2020; 40:34-43. [PMID: 32648286 DOI: 10.1002/jhrm.21434] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
The Office of the National Coordinator for Health Information Technology released the Safety Assurance Factors for EHR Resilience (SAFER) guides in 2014. Our group developed these guides covering key facets of both electronic health record (EHR) infrastructure (eg, system configuration, contingency planning for downtime, and system-to-system interfaces) and clinical processes (eg, computer-based provider order entry with clinical decision support, test result reporting, patient identification, and clinician-to-clinician communication). The SAFER guides encourage healthy relationships between EHR vendors and users. We conducted a qualitative study over 12 months. We visited 9 health care organizations ranging in size from 1-doctor outpatient clinics to large, multisite, multihospital integrated delivery networks. We interviewed and observed clinicians, IT professionals, and administrators. From the interview transcripts and observation field notes, we identified overarching themes: technical functionality, usability, standards, testing, workflow processes, personnel to support implementation and use, infrastructure, and clinical content. In addition, we identified health care organization-EHR vendor working relationships: marine drill sergeant, mentor, development partner, seller, and parasite. We encourage health care organizations and EHR vendors to develop healthy working relationships to help address the tasks required to design, develop, implement, and maintain EHRs required to achieve safer and higher quality health care.
Collapse
Affiliation(s)
- Dean F Sittig
- UT-Memorial Hermann Center for Healthcare Quality and Safety, University of Texas Health Science Center at Houston, 6410 Fannin St. UTP 1100.43, Houston, TX, 77030
| | - Joan S Ash
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Mail Code BICC, Portland, OR, 97239-3098
| | - Adam Wright
- Biomedical Informatics, 2525 West End Avenue, Suite 1475, Room 14109, Nashville, TN, 37203.,Brigham and Women's Hospital, Boston, MA
| | - Dian Chase
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Mail Code BICC, Portland, OR, 97239-3098
| | - Eric Gebhardt
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Mail Code BICC, Portland, OR, 97239-3098
| | - Elise M Russo
- Michael E. DeBakey VA Medical Center, Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, 2002 Holcombe Blvd. 152, Houston, TX, 77030
| | - Colleen Tercek
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Mail Code BICC, Portland, OR, 97239-3098
| | - Vishnu Mohan
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Mail Code BICC, Portland, OR, 97239-3098
| | - Hardeep Singh
- Michael E. DeBakey VA Medical Center, Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, 2002 Holcombe Blvd. 152, Houston, TX, 77030
| |
Collapse
|
38
|
Affiliation(s)
- Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness, and Safety (IQuESt) at the Michael E. DeBakey VA Medical Center and Baylor College of Medicine, Houston, Texas
| |
Collapse
|
39
|
Singh H, Sittig DF. A Sociotechnical Framework for Safety-Related Electronic Health Record Research Reporting: The SAFER Reporting Framework. Ann Intern Med 2020; 172:S92-S100. [PMID: 32479184 DOI: 10.7326/m19-0879] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
Electronic health record (EHR)-based interventions to improve patient safety are complex and sensitive to who, what, where, why, when, and how they are delivered. Success or failure depends not only on the characteristics and behaviors of individuals who are targeted by an intervention, but also on the technical characteristics of the intervention and the culture and environment of the health system that implements it. Current reporting guidelines do not capture the complexity of sociotechnical factors (technical and nontechnical factors, such as workflow and organizational issues) that confound or influence these interventions. This article proposes a methodological reporting framework for EHR interventions targeting patient safety and builds on an 8-dimension sociotechnical model previously developed by the authors for design, development, implementation, use, and evaluation of health information technology. The Safety-related EHR Research (SAFER) Reporting Framework enables reporting of patient safety-focused EHR-based interventions while accounting for the multifaceted, dynamic sociotechnical context affecting intervention implementation, effectiveness, and generalizability. As an example, an EHR-based intervention to improve communication and timely follow-up of subcritical abnormal test results to operationalize the framework is presented. For each dimension, reporting should include what sociotechnical changes were made to implement an EHR-related intervention to improve patient safety, why the intervention did or did not lead to safety improvements, and how this intervention can be applied or exported to other health care organizations. A foundational list of research and reporting recommendations to address implementation, effectiveness, and generalizability of EHR-based interventions needed to effectively reduce preventable patient harm is provided. The SAFER Reporting Framework is not meant to replace previous research reporting guidelines, but rather provides a sociotechnical adjunct that complements their use.
Collapse
Affiliation(s)
- Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, Texas (H.S.)
| | - Dean F Sittig
- University of Texas Memorial Hermann Center for Healthcare Quality & Safety, School of Biomedical Informatics, University of Texas Health Science Center at Houston, Texas (D.F.S.)
| |
Collapse
|
40
|
Powell L, Sittig DF, Chrouser K, Singh H. Assessment of Health Information Technology-Related Outpatient Diagnostic Delays in the US Veterans Affairs Health Care System: A Qualitative Study of Aggregated Root Cause Analysis Data. JAMA Netw Open 2020; 3:e206752. [PMID: 32584406 PMCID: PMC7317596 DOI: 10.1001/jamanetworkopen.2020.6752] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/05/2023] Open
Abstract
IMPORTANCE Diagnostic delay in the outpatient setting is an emerging safety priority that health information technology (HIT) should help address. However, diagnostic delays have persisted, and new safety concerns associated with the use of HIT have emerged. OBJECTIVE To analyze HIT-related outpatient diagnostic delays within a large, integrated health care system. DESIGN, SETTING, AND PARTICIPANTS This cohort study involved qualitative content analysis of safety concerns identified in aggregated root cause analysis (RCA) data related to HIT and outpatient diagnostic delays. The setting was the US Department of Veterans Affairs using all RCAs submitted to the Veterans Affairs (VA) National Center for Patient Safety from January 1, 2013, to July 31, 2018. MAIN OUTCOMES AND MEASURES Common themes associated with the role of HIT-related safety concerns were identified and categorized according to the Health IT Safety framework for measuring, monitoring, and improving HIT safety. This framework includes 3 related domains (ie, safe HIT, safe use of HIT, and using HIT to improve safety) situated within an 8-dimensional sociotechnical model accounting for interacting technical and nontechnical variables associated with safety. Hence, themes identified enhanced understanding of the sociotechnical context and domain of HIT safety involved. RESULTS Of 214 RCAs categorized by the terms delay and outpatient submitted during the study period, 88 were identified as involving diagnostic delays and HIT, from which 172 unique HIT-related safety concerns were extracted (mean [SD], 1.97 [1.53] per RCA). Most safety concerns (82.6% [142 of 172]) involved problems with safe use of HIT, predominantly sociotechnical factors associated with people, workflow and communication, and a poorly designed human-computer interface. Fewer safety concerns involved problems with safe HIT (14.5% [25 of 172]) or using HIT to improve safety (0.3% [5 of 172]). The following 5 key high-risk areas for diagnostic delays emerged: managing electronic health record inbox notifications and communication, clinicians gathering key diagnostic information, technical problems, data entry problems, and failure of a system to track test results. CONCLUSIONS AND RELEVANCE This qualitative study of a national RCA data set suggests that interventions to reduce outpatient diagnostic delays could aim to improve test result management, interoperability, data visualization, and order entry, as well as to decrease information overload.
Collapse
Affiliation(s)
- Lauren Powell
- Veterans Affairs (VA) National Center for Patient Safety, Ann Arbor, Michigan
| | - Dean F Sittig
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston
| | | | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness, and Safety (IQuESt) at the Michael E. DeBakey VA Medical Center and Baylor College of Medicine, Houston, Texas
| |
Collapse
|
41
|
King AJ, Cooper GF, Clermont G, Hochheiser H, Hauskrecht M, Sittig DF, Visweswaran S. Leveraging Eye Tracking to Prioritize Relevant Medical Record Data: Comparative Machine Learning Study. J Med Internet Res 2020; 22:e15876. [PMID: 32238342 PMCID: PMC7163414 DOI: 10.2196/15876] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2019] [Revised: 12/04/2019] [Accepted: 01/23/2020] [Indexed: 02/05/2023] Open
Abstract
BACKGROUND Electronic medical record (EMR) systems capture large amounts of data per patient and present that data to physicians with little prioritization. Without prioritization, physicians must mentally identify and collate relevant data, an activity that can lead to cognitive overload. To mitigate cognitive overload, a Learning EMR (LEMR) system prioritizes the display of relevant medical record data. Relevant data are those that are pertinent to a context-defined as the combination of the user, clinical task, and patient case. To determine which data are relevant in a specific context, a LEMR system uses supervised machine learning models of physician information-seeking behavior. Since obtaining information-seeking behavior data via manual annotation is slow and expensive, automatic methods for capturing such data are needed. OBJECTIVE The goal of the research was to propose and evaluate eye tracking as a high-throughput method to automatically acquire physician information-seeking behavior useful for training models for a LEMR system. METHODS Critical care medicine physicians reviewed intensive care unit patient cases in an EMR interface developed for the study. Participants manually identified patient data that were relevant in the context of a clinical task: preparing a patient summary to present at morning rounds. We used eye tracking to capture each physician's gaze dwell time on each data item (eg, blood glucose measurements). Manual annotations and gaze dwell times were used to define target variables for developing supervised machine learning models of physician information-seeking behavior. We compared the performance of manual selection and gaze-derived models on an independent set of patient cases. RESULTS A total of 68 pairs of manual selection and gaze-derived machine learning models were developed from training data and evaluated on an independent evaluation data set. A paired Wilcoxon signed-rank test showed similar performance of manual selection and gaze-derived models on area under the receiver operating characteristic curve (P=.40). CONCLUSIONS We used eye tracking to automatically capture physician information-seeking behavior and used it to train models for a LEMR system. The models that were trained using eye tracking performed like models that were trained using manual annotations. These results support further development of eye tracking as a high-throughput method for training clinical decision support systems that prioritize the display of relevant medical record data.
Collapse
Affiliation(s)
- Andrew J King
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, United States
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA, United States
| | - Gregory F Cooper
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, United States
- Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, United States
| | - Gilles Clermont
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA, United States
| | - Harry Hochheiser
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, United States
- Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, United States
| | - Milos Hauskrecht
- Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, United States
| | - Dean F Sittig
- Department of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, United States
| | - Shyam Visweswaran
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, United States
- Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, United States
| |
Collapse
|
42
|
McCoy AB, Sittig DF, Lin J, Wright A. Identification and Ranking of Biomedical Informatics Researcher Citation Statistics through a Google Scholar Scraper. AMIA Annu Symp Proc 2020; 2019:655-663. [PMID: 32308860 PMCID: PMC7153158] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
To overcome limitations of previously developed scientific productivity ranking services, we created the Biomedical Informatics Researchers ranking website (rank.informatics-review.com). The website is composed of four key components that work together to create the automatically updating ranking website: 1) list of biomedical informatics researchers, 2) Google Scholar scraper, 3) display page, and 4) updater. The interactive website has facilitated identification of leaders in each of the key citation statistics categories (i.e., number of citations, h-index, and i10-index), and it has allowed other groups, such as tenure and promotions committees, to more effectively and efficiently evaluate researchers and interpret the various citation statistics reported by candidates. Creation of the biomedical informatics researcher ranking website highlights the vast differences in scholarly productivity among members of the biomedical informatics research community. Future efforts are underway to add new functionality to the website and to expand the work to identify top papers in biomedical informatics.
Collapse
Affiliation(s)
| | - Dean F Sittig
- The University of Texas School of Biomedical Informatics at Houston, Houston, TX
| | - Jimmy Lin
- University of Waterloo, Waterloo, ON, Canada
| | - Adam Wright
- Brigham and Women's Hospital, Harvard University, Boston, MA, USA
| |
Collapse
|
43
|
Brunner MC, Sheehan SE, Yanke EM, Sittig DF, Safdar N, Hill B, Lee KS, Orwin JF, Vanness DJ, Hildebrand CJ, Bruno MA, Erickson TJ, Zea R, Moberg DP. Joint Design with Providers of Clinical Decision Support for Value-Based Advanced Shoulder Imaging. Appl Clin Inform 2020; 11:142-152. [PMID: 32074651 DOI: 10.1055/s-0040-1701256] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
BACKGROUND Provider orders for inappropriate advanced imaging, while rarely altering patient management, contribute enough to the strain on available health care resources, and therefore the United States Congress established the Appropriate Use Criteria Program. OBJECTIVES To examine whether co-designing clinical decision support (CDS) with referring providers will reduce barriers to adoption and facilitate more appropriate shoulder ultrasound (US) over magnetic resonance imaging (MRI) in diagnosing Veteran shoulder pain, given similar efficacies and only 5% MRI follow-up rate after shoulder US. METHODS We used a theory-driven, convergent parallel mixed-methods approach to prospectively (1) determine medical providers' reasons for selecting MRI over US in diagnosing shoulder pain and identify barriers to ordering US, (2) co-design CDS, informed by provider interviews, to prompt appropriate US use, and (3) assess CDS impact on shoulder imaging use. CDS effectiveness in guiding appropriate shoulder imaging was evaluated through monthly monitoring of ordering data at our quaternary care Veterans Hospital. Key outcome measures were appropriate MRI/US use rates and transition to ordering US by both musculoskeletal specialist and generalist providers. We assessed differences in ordering using a generalized estimating equations logistic regression model. We compared continuous measures using mixed effects analysis of variance with log-transformed data. RESULTS During December 2016 to March 2018, 569 (395 MRI, 174 US) shoulder advanced imaging examinations were ordered by 111 providers. CDS "co-designed" in collaboration with providers increased US from 17% (58/335) to 50% (116/234) of all orders (p < 0.001), with concomitant decrease in MRI. Ordering appropriateness more than doubled from 31% (105/335) to 67% (157/234) following CDS (p < 0.001). Interviews confirmed that generalist providers want help in appropriately ordering advanced imaging. CONCLUSION Partnering with medical providers to co-design CDS reduced barriers and prompted appropriate transition to US from MRI for shoulder pain diagnosis, promoting evidence-based practice. This approach can inform the development and implementation of other forms of CDS.
Collapse
Affiliation(s)
- Michael C Brunner
- Department of Radiology, William S. Middleton Memorial Veterans Hospital, Madison, Wisconsin, United States.,Department of Radiology, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Scott E Sheehan
- Department of Radiology, William S. Middleton Memorial Veterans Hospital, Madison, Wisconsin, United States.,Department of Radiology, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Eric M Yanke
- Department of Medicine, William S. Middleton Memorial Veteran Hospital, Madison, Wisconsin, United States
| | - Dean F Sittig
- Department of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, United States
| | - Nasia Safdar
- Department of Medicine, William S. Middleton Memorial Veteran Hospital, Madison, Wisconsin, United States.,Department of Medicine, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Barbara Hill
- Population Health Sciences, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Kenneth S Lee
- Department of Radiology, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| | - John F Orwin
- Department of Orthopedics and Rehabilitation, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| | - David J Vanness
- Department of Health Policy and Administration, Pennsylvania State University, University Park, Pennsylvania, United States
| | - Christopher J Hildebrand
- Department of Medicine, William S. Middleton Memorial Veteran Hospital, Madison, Wisconsin, United States.,Department of Medicine, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Michael A Bruno
- Department of Radiology, The Penn State Milton S. Hershey Medical Center and Penn State College of Medicine, Hershey, Pennsylvania, United States
| | - Timothy J Erickson
- Department of Physical Medicine and Rehabilitation, William S. Middleton Memorial Veterans Hospital, Madison, Wisconsin, United States
| | - Ryan Zea
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, Wisconsin, United States
| | - D Paul Moberg
- Population Health Sciences, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, United States
| |
Collapse
|
44
|
Dullabh P, Hovey L, Heaney-Huls K, Rajendran N, Wright A, Sittig DF. Application Programming Interfaces in Health Care: Findings from a Current-State Sociotechnical Assessment. Appl Clin Inform 2020; 11:59-69. [PMID: 31968383 PMCID: PMC6976305 DOI: 10.1055/s-0039-1701001] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
Objective
Interest in application programming interfaces (APIs) is increasing as key stakeholders look for technical solutions to interoperability challenges. We explored three thematic areas to assess the current state of API use for data access and exchange in health care: (1) API use cases and standards; (2) challenges and facilitators for read and write capabilities; and (3) outlook for development of write capabilities.
Methods
We employed four methods: (1) literature review; (2) expert interviews with 13 API stakeholders; (3) review of electronic health record (EHR) app galleries; and (4) a technical expert panel. We used an eight-dimension sociotechnical model to organize our findings.
Results
The API ecosystem is complicated and cuts across five of the eight sociotechnical model dimensions: (1) app marketplaces support a range of use cases, the majority of which target providers' needs, with far fewer supporting patient access to data; (2) current focus on read APIs with limited use of write APIs; (3) where standards are used, they are largely Fast Healthcare Interoperability Resources (FHIR); (4) FHIR-based APIs support exchange of electronic health information within the common clinical data set; and (5) validating external data and data sources for clinical decision making creates challenges to provider workflows.
Conclusion
While the use of APIs in health care is increasing rapidly, it is still in the pilot stages. We identified five key issues with implications for the continued advancement of API use: (1) a robust normative FHIR standard; (2) expansion of the common clinical data set to other data elements; (3) enhanced support for write implementation; (4) data provenance rules; and (5) data governance rules. Thus, while APIs are being touted as a solution to interoperability challenges, they remain an emerging technology that is only one piece of a multipronged approach to data access and use.
Collapse
Affiliation(s)
- Prashila Dullabh
- NORC at the University of Chicago, Chicago, Illinois, United States
| | - Lauren Hovey
- NORC at the University of Chicago, Chicago, Illinois, United States
| | | | - Nithya Rajendran
- NORC at the University of Chicago, Chicago, Illinois, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, United States
| |
Collapse
|
45
|
Sheehan SE, Safdar N, Singh H, Sittig DF, Bruno MA, Keller K, Kinnard S, Brunner MC. Detection and Remediation of Misidentification Errors in Radiology Examination Ordering. Appl Clin Inform 2020; 11:79-87. [PMID: 31995835 PMCID: PMC6989264 DOI: 10.1055/s-0039-3402730] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2019] [Accepted: 12/06/2019] [Indexed: 02/08/2023] Open
Abstract
BACKGROUND Despite progress in patient safety, misidentification errors in radiology such as ordering imaging on the wrong anatomic side persist. If undetected, these errors can cause patient harm for multiple reasons, in addition to producing erroneous electronic health records (EHR) data. OBJECTIVES We describe the pilot testing of a quality improvement methodology using electronic trigger tools and preimaging checklists to detect "wrong-side" misidentification errors in radiology examination ordering, and to measure staff adherence to departmental policy in error remediation. METHODS We retrospectively applied and compared two methods for the detection of "wrong-side" misidentification errors among a cohort of all imaging studies ordered during a 1-year period (June 1, 2015-May 31, 2016) at our tertiary care hospital. Our methods included: (1) manual review of internal quality improvement spreadsheet records arising from the prospective performance of preimaging safety checklists, and (2) automated error detection via the development and validation of an electronic trigger tool which identified discrepant side indications within EHR imaging orders. RESULTS Our combined methods detected misidentification errors in 6.5/1,000 of study cohort imaging orders. Our trigger tool retrospectively identified substantially more misidentification errors than were detected prospectively during preimaging checklist performance, with a high positive predictive value (PPV: 88.4%, 95% confidence interval: 85.4-91.4). However, two third of errors detected during checklist performance were not detected by the trigger tool, and checklist-detected errors were more often appropriately resolved (p < 0.00001, 95% confidence interval: 2.0-6.9; odds ratio: 3.6). CONCLUSION Our trigger tool enabled the detection of substantially more imaging ordering misidentification errors than preimaging safety checklists alone, with a high PPV. Many errors were only detected by the preimaging checklist; however, suggesting that additional trigger tools may need to be developed and used in conjunction with checklist-based methods to ensure patient safety.
Collapse
Affiliation(s)
- Scott E. Sheehan
- Department of Radiology, William S. Middleton Veterans Hospital, Madison, Wisconsin, United States
| | - Nasia Safdar
- Department of Medicine, William S. Middleton Memorial Veterans Hospital and University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center and Department of Medicine, Baylor College of Medicine, Houston, Texas, United States
| | - Dean F. Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, United States
| | - Michael A. Bruno
- Department of Radiology, Penn State Hershey, Hershey, Pennsylvania, United States
| | - Kelli Keller
- Department of Radiology, William S. Middleton Veterans Hospital, Madison, Wisconsin, United States
| | - Samantha Kinnard
- Department of Radiology, William S. Middleton Veterans Hospital, Madison, Wisconsin, United States
| | - Michael C. Brunner
- Department of Radiology, William S. Middleton Memorial Veterans Hospital and University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, United States
| |
Collapse
|
46
|
King AJ, Cooper GF, Clermont G, Hochheiser H, Hauskrecht M, Sittig DF, Visweswaran S. Using machine learning to selectively highlight patient information. J Biomed Inform 2019; 100:103327. [PMID: 31676461 DOI: 10.1016/j.jbi.2019.103327] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2019] [Revised: 08/20/2019] [Accepted: 10/28/2019] [Indexed: 02/05/2023]
Abstract
BACKGROUND Electronic medical record (EMR) systems need functionality that decreases cognitive overload by drawing the clinician's attention to the right data, at the right time. We developed a Learning EMR (LEMR) system that learns statistical models of clinician information-seeking behavior and applies those models to direct the display of data in future patients. We evaluated the performance of the system in identifying relevant patient data in intensive care unit (ICU) patient cases. METHODS To capture information-seeking behavior, we enlisted critical care medicine physicians who reviewed a set of patient cases and selected data items relevant to the task of presenting at morning rounds. Using patient EMR data as predictors, we built machine learning models to predict their relevancy. We prospectively evaluated the predictions of a set of high performing models. RESULTS On an independent evaluation data set, 25 models achieved precision of 0.52, 95% CI [0.49, 0.54] and recall of 0.77, 95% CI [0.75, 0.80] in identifying relevant patient data items. For data items missed by the system, the reviewers rated the effect of not seeing those data from no impact to minor impact on patient care in about 82% of the cases. CONCLUSION Data-driven approaches for adaptively displaying data in EMR systems, like the LEMR system, show promise in using information-seeking behavior of clinicians to identify and highlight relevant patient data.
Collapse
Affiliation(s)
- Andrew J King
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA; Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA, USA
| | - Gregory F Cooper
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA; Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA
| | - Gilles Clermont
- Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA, USA
| | - Harry Hochheiser
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA; Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA
| | - Milos Hauskrecht
- Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA; Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA
| | - Dean F Sittig
- Department of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Shyam Visweswaran
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA; Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA.
| |
Collapse
|
47
|
Murphy DR, Giardina TD, Satterly T, Sittig DF, Singh H. An Exploration of Barriers, Facilitators, and Suggestions for Improving Electronic Health Record Inbox-Related Usability: A Qualitative Analysis. JAMA Netw Open 2019; 2:e1912638. [PMID: 31584683 PMCID: PMC6784746 DOI: 10.1001/jamanetworkopen.2019.12638] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/05/2023] Open
Abstract
IMPORTANCE Managing messages in the electronic health record (EHR) inbox consumes substantial amounts of physician time. Certain factors associated with inbox management, such as poor usability and excessive and unnecessary inbox messages, have been associated with physician burnout. Additionally, inbox design, usability, and workflows are associated with physicians' situational awareness (ie, perception, comprehension, and projection of clinical status) and efficiency of processing EHR inbox messages. Understanding factors associated with inbox usability could improve future EHR inbox designs and workflows, thus reducing risk of burnout while improving patient safety. OBJECTIVE To determine barriers, facilitators, and suggestions associated with EHR inbox-related usability. DESIGN, SETTING, AND PARTICIPANTS This qualitative study included cognitive walkthroughs of EHR inbox management with 25 physicians (17 primary care physicians and 8 specialists) at 6 large health care organizations using 4 different EHR systems between May 6, 2015, and September 19, 2016. While processing EHR inbox messages, participants identified facilitators and barriers associated with EHR inbox situational awareness and processing efficiency and potential interventions to address such barriers. A qualitative analysis was performed on transcribed recordings using an inductive thematic approach with an 8-dimension sociotechnical model as a theoretical lens from May 6, 2015, to August 15, 2019. RESULTS The cognitive walkthroughs identified 60 barriers, 32 facilitators, and 28 suggestions for improving the EHR inbox. Emergent data fit within 5 major themes: message processing complexity, inbox interface design, cognitive load, team communication, and inbox message content. Within these themes, similar barriers were identified across sites, such as poor usability due the high numbers of clicks needed to accomplish actions. In certain instances, an identified facilitator at one site provided the exact solution needed to address a barrier identified at another site. CONCLUSIONS AND RELEVANCE This qualitative study found that usability of the EHR inbox is often suboptimal and variable across sites, suggesting lack of shared best practices related to information management. Implementation of optimized design features and workflows will require EHR developers and health care organizations to collectively share this responsibility. Development of regional or national consortia to support collaborative sharing and implementation of EHR system best practices across EHR developers and health care organizations could also improve safety and efficiency and reduce physician burnout.
Collapse
Affiliation(s)
- Daniel R. Murphy
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
- Department of Medicine, Baylor College of Medicine, Houston, Texas
| | - Traber D. Giardina
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
- Department of Medicine, Baylor College of Medicine, Houston, Texas
| | - Tyler Satterly
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
- Department of Medicine, Baylor College of Medicine, Houston, Texas
| | - Dean F. Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston
- Center for Healthcare Quality and Safety, University of Texas Health Science Center at Houston, Houston
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas
- Department of Medicine, Baylor College of Medicine, Houston, Texas
| |
Collapse
|
48
|
Wright A, McEvoy DS, Aaron S, McCoy AB, Amato MG, Kim H, Ai A, Cimino JJ, Desai BR, El-Kareh R, Galanter W, Longhurst CA, Malhotra S, Radecki RP, Samal L, Schreiber R, Shelov E, Sirajuddin AM, Sittig DF. Structured override reasons for drug-drug interaction alerts in electronic health records. J Am Med Inform Assoc 2019; 26:934-942. [PMID: 31329891 PMCID: PMC6748816 DOI: 10.1093/jamia/ocz033] [Citation(s) in RCA: 23] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2019] [Revised: 02/28/2019] [Accepted: 03/06/2019] [Indexed: 02/05/2023] Open
Abstract
Objective The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records. Materials and Methods We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices. Results Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: “will monitor or take precautions,” “not clinically significant,” and “benefit outweighs risk.” Discussion We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved. Conclusions Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.
Collapse
Affiliation(s)
- Adam Wright
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA.,Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA.,Partners eCare, Partners HealthCare, Boston, Massachusetts, USA
| | - Dustin S McEvoy
- Partners eCare, Partners HealthCare, Boston, Massachusetts, USA
| | - Skye Aaron
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Mary G Amato
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Department of Pharmacy Practice, Massachusetts College of Pharmacy and Health Sciences University, Boston, Massachusetts, USA
| | - Hyun Kim
- Clinical Pharmacogenomics Service, Boston Children's Hospital, Boston, Massachusetts, USA
| | - Angela Ai
- University of Wisconsin School of Medicine and Public Health, University of Wisconsin Madison, Madison, Wisconsin, USA
| | - James J Cimino
- Informatics Institute and Department of Medicine, University of Alabama at Birmingham School of Medicine, Birmingham, Alabama, USA
| | - Bimal R Desai
- Division of General Pediatrics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | - Robert El-Kareh
- Department of Medicine, UC San Diego Health, University of California, San Diego, San Diego, California, USA
| | - William Galanter
- Department of Medicine, University of Illinois at Chicago, Chicago, Illinois, USA
| | - Christopher A Longhurst
- Department of Medicine, UC San Diego Health, University of California, San Diego, San Diego, California, USA
| | - Sameer Malhotra
- Department of Healthcare Policy and Research, Weill Cornell Medicine, New York, New York, USA
| | - Ryan P Radecki
- Department of Emergency Medicine, Northwest Permanente, Portland, Oregon, USA
| | - Lipika Samal
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA
| | - Richard Schreiber
- Physician Informatics and Department of Internal Medicine, Geisinger Holy Spirit, Camp Hill, Pennsylvania, USA
| | - Eric Shelov
- Division of General Pediatrics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | | | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA
| |
Collapse
|
49
|
Murphy DR, Satterly T, Giardina TD, Sittig DF, Singh H. Practicing Clinicians' Recommendations to Reduce Burden from the Electronic Health Record Inbox: a Mixed-Methods Study. J Gen Intern Med 2019; 34:1825-1832. [PMID: 31292905 PMCID: PMC6712240 DOI: 10.1007/s11606-019-05112-5] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/06/2018] [Revised: 03/25/2019] [Accepted: 04/20/2019] [Indexed: 02/05/2023]
Abstract
BACKGROUND Workload from electronic health record (EHR) inbox notifications leads to information overload and contributes to job dissatisfaction and physician burnout. Better understanding of physicians' inbox requirements and workflows could optimize inbox designs, enhance efficiency, and reduce safety risks from information overload. DESIGN We conducted a mixed-methods study to identify strategies to enhance EHR inbox design and workflow. First, we performed a secondary analysis of national survey data of all Department of Veterans Affairs (VA) primary care practitioners (PCP) to identify major themes in responses to a free-text question soliciting suggestions to improve EHR inbox design and workflows. We then conducted expert interviews of clinicians at five health care systems (1 VA and 4 non-VA settings using 4 different EHRs) to understand existing optimal strategies to improve efficiency and situational awareness related to EHR inbox use. Themes from survey data were cross-validated with interview findings. RESULTS We analyzed responses from 2104 PCPs who completed the free-text inbox question (of 5001 PCPs who responded to survey) and used an inductive approach to identify five themes: (1) Inbox notification content should be actionable for patient care and relevant to recipient clinician, (2) Inboxes should reduce risk of losing messages, (3) Inbox functionality should be optimized to improve efficiency of processing notifications, (4) Team support should be leveraged to help with EHR inbox notification burden, (5) Sufficient time should be provided to all clinicians to process EHR inbox notifications. We subsequently interviewed 15 VA and non-VA clinicians and identified 11 unique strategies, each corresponding directly with one of these five themes. CONCLUSION Feedback from practicing end-user clinicians provides robust evidence to improve content and design of the EHR inbox and related clinical workflows and organizational policies. Several strategies we identified could improve clinicians' EHR efficiency and satisfaction as well as empower them to work with their local administrators, health IT personnel, and EHR developers to improve these systems.
Collapse
Affiliation(s)
- Daniel R Murphy
- Center for Innovations in Quality, Effectiveness and Safety (IQuESt) (152), Michael E. DeBakey Veterans Affairs Medical Center (MEDVAMC), 2002 Holcombe Boulevard, Houston, TX, 77030, USA.
- Department of Medicine, Baylor College of Medicine, Houston, TX, USA.
| | - Tyler Satterly
- Center for Innovations in Quality, Effectiveness and Safety (IQuESt) (152), Michael E. DeBakey Veterans Affairs Medical Center (MEDVAMC), 2002 Holcombe Boulevard, Houston, TX, 77030, USA
- Department of Medicine, Baylor College of Medicine, Houston, TX, USA
| | - Traber D Giardina
- Center for Innovations in Quality, Effectiveness and Safety (IQuESt) (152), Michael E. DeBakey Veterans Affairs Medical Center (MEDVAMC), 2002 Holcombe Boulevard, Houston, TX, 77030, USA
- Department of Medicine, Baylor College of Medicine, Houston, TX, USA
| | - Dean F Sittig
- University of Texas Health Science Center at Houston's School of Biomedical Informatics and the UT-Memorial Hermann Center for Healthcare Quality & Safety, Houston, TX, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety (IQuESt) (152), Michael E. DeBakey Veterans Affairs Medical Center (MEDVAMC), 2002 Holcombe Boulevard, Houston, TX, 77030, USA
- Department of Medicine, Baylor College of Medicine, Houston, TX, USA
| |
Collapse
|
50
|
Gensheimer SG, Wu AW, Snyder CF; PRO-EHR Users’ Guide Steering Group., PRO-EHR Users’ Guide Working Group. Oh, the Places We'll Go: Patient-Reported Outcomes and Electronic Health Records. Patient 2018; 11:591-8. [PMID: 29968179 DOI: 10.1007/s40271-018-0321-9] [Citation(s) in RCA: 76] [Impact Index Per Article: 15.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
The growing measurement of patient-reported outcomes (PROs) and adoption of electronic health records (EHRs) presents an unprecedented opportunity to improve health care for patients and populations. The integration of PROs into EHRs can promote patient-centered care and advance quality improvement initiatives, research, and population health. Despite these potential benefits, there are few best practices to help organizations achieve integration. To integrate PROs into EHRs, organizations should evaluate the advantages and disadvantages of various approaches within three themes: Planning, Selection, and Engagement. Planning considerations for integration include what strategy will be used, how the integrated system will be governed, ethical and legal issues, and how data from multiple EHRs can be pooled across organizations. Selection considerations involve identifying which patient population to target for PRO data collection on the basis of the intended use of the data in the health care system, and then choosing specific outcomes and their measures. Engagement considerations include how, where, and with what frequency patients will respond to PRO measures, how to display PRO data in EHRs, how clinical teams will act upon PRO data, and how to train, support and incent clinical teams and patients to incorporate PRO data into care. There is no most effective model that will work in all contexts. Organizations wishing to integrate PROs and EHRs should assemble the multidisciplinary expertise needed to evaluate the advantages and disadvantages of the various approaches for their particular context. We specifically recommend that organizations think carefully about stakeholder participation; design their system with data sharing in mind; develop a framework to aid in PRO selection; create guidelines to support PRO interpretation and action for patients and clinicians; and ensure patients have access to their own PRO data.
Collapse
|