1
|
Thayer JG, Franklin A, Miller JM, Grundmeier RW, Rogith D, Wright A. A scoping review of rule-based clinical decision support malfunctions. J Am Med Inform Assoc 2024:ocae187. [PMID: 39078287 DOI: 10.1093/jamia/ocae187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2024] [Revised: 06/14/2024] [Accepted: 07/08/2024] [Indexed: 07/31/2024] Open
Abstract
OBJECTIVE Conduct a scoping review of research studies that describe rule-based clinical decision support (CDS) malfunctions. MATERIALS AND METHODS In April 2022, we searched three bibliographic databases (MEDLINE, CINAHL, and Embase) for literature referencing CDS malfunctions. We coded the identified malfunctions according to an existing CDS malfunction taxonomy and added new categories for factors not already captured. We also extracted and summarized information related to the CDS system, such as architecture, data source, and data format. RESULTS Twenty-eight articles met inclusion criteria, capturing 130 malfunctions. Architectures used included stand-alone systems (eg, web-based calculator), integrated systems (eg, best practices alerts), and service-oriented architectures (eg, distributed systems like SMART or CDS Hooks). No standards-based CDS malfunctions were identified. The "Cause" category of the original taxonomy includes three new types (organizational policy, hardware error, and data source) and two existing causes were expanded to include additional layers. Only 29 malfunctions (22%) described the potential impact of the malfunction on patient care. DISCUSSION While a substantial amount of research on CDS exists, our review indicates there is a limited focus on CDS malfunctions, with even less attention on malfunctions associated with modern delivery architectures such as SMART and CDS Hooks. CONCLUSION CDS malfunctions can and do occur across several different care delivery architectures. To account for advances in health information technology, existing taxonomies of CDS malfunctions must be continually updated. This will be especially important for service-oriented architectures, which connect several disparate systems, and are increasing in use.
Collapse
Affiliation(s)
- Jeritt G Thayer
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, PA 19146, United States
- McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030, United States
- Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Amy Franklin
- McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030, United States
| | - Jeffrey M Miller
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, PA 19146, United States
| | - Robert W Grundmeier
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, PA 19146, United States
- Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Deevakar Rogith
- McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37232, United States
| |
Collapse
|
2
|
Sangillo E, Jube-Desai N, El-Metwally D, Hughes Driscoll C. Impact of a Clinical Decision Support Alert on Informed Consent Documentation in the Neonatal Intensive Care Unit. Pediatr Qual Saf 2024; 9:e713. [PMID: 38322296 PMCID: PMC10843373 DOI: 10.1097/pq9.0000000000000713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 12/12/2023] [Indexed: 02/08/2024] Open
Abstract
Background Informed consent is necessary to preserve patient autonomy and shared decision-making, yet compliant consent documentation is suboptimal in the intensive care unit (ICU). We aimed to increase compliance with bundled consent documentation, which provides consent for a predefined set of common procedures in the neonatal ICU from 0% to 50% over 1 year. Methods We used the Plan-Do-Study-Act model for quality improvement. Interventions included education and performance awareness, delineation of the preferred consenting process, consent form revision, overlay tool creation, and clinical decision support (CDS) alert use within the electronic health record. Monthly audits categorized consent forms as missing, present but noncompliant, or compliant. We analyzed consent compliance on a run chart using standard run chart interpretation rules and obtained feedback on the CDS as a countermeasure. Results We conducted 564 audits over 37 months. Overall, median consent compliance increased from 0% to 86.6%. Upon initiating the CDS alert, we observed the highest monthly compliance of 93.3%, followed by a decrease to 33.3% with an inadvertent discontinuation of the CDS. Compliance subsequently increased to 73.3% after the restoration of the alert. We created a consultant opt-out selection to address negative feedback associated with CDS. There were no missing consent forms within the last 7 months of monitoring. Conclusions A multi-faceted approach led to sustained improvement in bundled consent documentation compliance in our neonatal intensive care unit, with the direct contribution of the CDS observed. A CDS intervention directed at the informed consenting process may similarly benefit other ICUs.
Collapse
Affiliation(s)
- Emily Sangillo
- From the Department of Pediatrics, University of Maryland School of Medicine, Baltimore, Md
| | - Neena Jube-Desai
- From the Department of Pediatrics, University of Maryland School of Medicine, Baltimore, Md
| | - Dina El-Metwally
- From the Department of Pediatrics, University of Maryland School of Medicine, Baltimore, Md
| | | |
Collapse
|
3
|
Black KC, Snyder NA, Zhou M, Zhu Z, Uptegraft C, Chintalapani A, Orwoll B. An Electronic Health Record Alert for Inpatient Coronavirus Disease 2019 Vaccinations Increases Vaccination Ordering and Uncovers Workflow Inefficiencies. Appl Clin Inform 2024; 15:192-198. [PMID: 38253337 PMCID: PMC10917607 DOI: 10.1055/a-2250-6305] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Accepted: 01/19/2024] [Indexed: 01/24/2024] Open
Abstract
BACKGROUND Despite mortality benefits, only 19.9% of U.S. adults are fully vaccinated against the coronavirus disease 2019 (COVID-19). The inpatient setting is an opportune environment to update vaccinations, and inpatient electronic health record (EHR) alerts have been shown to increase vaccination rates. OBJECTIVE Our objective was to evaluate whether an EHR alert could increase COVID-19 vaccinations in eligible hospitalized adults by prompting providers to order the vaccine. METHODS This was a quasiexperimental pre-post-interventional design study at an academic and community hospital in the western United States between 1 January, 2021 and 31 October, 2021. Inclusion criteria were unvaccinated hospitalized adults. A soft-stop, interruptive EHR alert prompted providers to order COVID-19 vaccines for those with an expected discharge date within 48 hours and interest in vaccination. The outcome measured was the proportion of all eligible patients for whom vaccines were ordered and administered before and after alert implementation. RESULTS Vaccine ordering rates increased from 4.0 to 13.0% at the academic hospital (odds ratio [OR]: 4.01, 95% confidence interval [CI]: 3.39-4.74, p < 0.001) and from 7.4 to 11.6% at the community hospital (OR: 1.62, 95% CI: 1.23-2.13, p < 0.001) after alert implementation. Administration increased postalert from 3.6 to 12.7% at the academic hospital (OR: 3.21, 95% CI: 2.70-3.82, p < 0.001) but was unchanged at the community hospital, 6.7 to 6.7% (OR: 0.99, 95% CI: 0.73-1.37, p = 0.994). Further analysis revealed infrequent vaccine availability at the community hospital. CONCLUSION Vaccine ordering rates improved at both sites after alert implementation. Vaccine administration rates, however, only improved at the academic hospital, likely due in part to vaccine dispensation inefficiency at the community hospital. This study demonstrates the potential impact of complex workflow patterns on new EHR alert success and provides a rationale for subsequent qualitative workflow analysis with alert implementation.
Collapse
Affiliation(s)
| | | | - Mengyu Zhou
- Department of Medicine, Oregon Health and Science University, Portland, Oregon
| | - Zhen Zhu
- Department of Medicine, Oregon Health and Science University, Portland, Oregon
| | - Colby Uptegraft
- Health Informatics Directorate, Defense Health Agency, Falls Church, Virginia
| | - Ani Chintalapani
- Department of Medicine, Oregon Health and Science University, Portland, Oregon
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon
| | - Benjamin Orwoll
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon
- Department of Pediatrics, Oregon Health and Science University, Portland, Oregon
| |
Collapse
|
4
|
Cánovas-Segura B, Morales A, Juarez JM, Campos M. Meaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework. J Biomed Inform 2023:104397. [PMID: 37245656 DOI: 10.1016/j.jbi.2023.104397] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2022] [Revised: 03/11/2023] [Accepted: 05/15/2023] [Indexed: 05/30/2023]
Abstract
Alerts are a common functionality of clinical decision support systems (CDSSs). Although they have proven to be useful in clinical practice, the alert burden can lead to alert fatigue and significantly reduce their usability and acceptance. Based on a literature review, we propose a unified framework consisting of a set of meaningful timestamps that allows the use of state-of-the-art measures for alert burden, such as alert dwell time, alert think time, and response time. In addition, it can be used to investigate other measures that could be relevant as regards dealing with this problem. Furthermore, we provide a case study concerning three different types of alerts to which the framework was successfully applied. We consider that our framework can easily be adapted to other CDSSs and that it could be useful for dealing with alert burden measurement thus contributing to its appropriate management.
Collapse
Affiliation(s)
| | - Antonio Morales
- AIKE Research Group (INTICO), University of Murcia, Murcia, Spain.
| | - Jose M Juarez
- AIKE Research Group (INTICO), University of Murcia, Murcia, Spain.
| | - Manuel Campos
- AIKE Research Group (INTICO), University of Murcia, Murcia, Spain; Murcian Bio-Health Institute (IMIB-Arrixaca), Murcia, Spain.
| |
Collapse
|
5
|
Surian D, Wang Y, Coiera E, Magrabi F. Using automated methods to detect safety problems with health information technology: a scoping review. J Am Med Inform Assoc 2023; 30:382-392. [PMID: 36374227 PMCID: PMC9846685 DOI: 10.1093/jamia/ocac220] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2022] [Revised: 10/14/2022] [Accepted: 11/01/2022] [Indexed: 11/16/2022] Open
Abstract
OBJECTIVE To summarize the research literature evaluating automated methods for early detection of safety problems with health information technology (HIT). MATERIALS AND METHODS We searched bibliographic databases including MEDLINE, ACM Digital, Embase, CINAHL Complete, PsycINFO, and Web of Science from January 2010 to June 2021 for studies evaluating the performance of automated methods to detect HIT problems. HIT problems were reviewed using an existing classification for safety concerns. Automated methods were categorized into rule-based, statistical, and machine learning methods, and their performance in detecting HIT problems was assessed. The review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Reviews statement. RESULTS Of the 45 studies identified, the majority (n = 27, 60%) focused on detecting use errors involving electronic health records and order entry systems. Machine learning (n = 22) and statistical modeling (n = 17) were the most common methods. Unsupervised learning was used to detect use errors in laboratory test results, prescriptions, and patient records while supervised learning was used to detect technical errors arising from hardware or software issues. Statistical modeling was used to detect use errors, unauthorized access, and clinical decision support system malfunctions while rule-based methods primarily focused on use errors. CONCLUSIONS A wide variety of rule-based, statistical, and machine learning methods have been applied to automate the detection of safety problems with HIT. Many opportunities remain to systematically study their application and effectiveness in real-world settings.
Collapse
Affiliation(s)
- Didi Surian
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Ying Wang
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Enrico Coiera
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Farah Magrabi
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| |
Collapse
|
6
|
Rubins D, McCoy AB, Dutta S, McEvoy DS, Patterson L, Miller A, Jackson JG, Zuccotti G, Wright A. Real-Time User Feedback to Support Clinical Decision Support System Improvement. Appl Clin Inform 2022; 13:1024-1032. [PMID: 36288748 PMCID: PMC9605820 DOI: 10.1055/s-0042-1757923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Accepted: 09/13/2022] [Indexed: 11/02/2022] Open
Abstract
OBJECTIVES To improve clinical decision support (CDS) by allowing users to provide real-time feedback when they interact with CDS tools and by creating processes for responding to and acting on this feedback. METHODS Two organizations implemented similar real-time feedback tools and processes in their electronic health record and gathered data over a 30-month period. At both sites, users could provide feedback by using Likert feedback links embedded in all end-user facing alerts, with results stored outside the electronic health record, and provide feedback as a comment when they overrode an alert. Both systems are monitored daily by clinical informatics teams. RESULTS The two sites received 2,639 Likert feedback comments and 623,270 override comments over a 30-month period. Through four case studies, we describe our use of end-user feedback to rapidly respond to build errors, as well as identifying inaccurate knowledge management, user-interface issues, and unique workflows. CONCLUSION Feedback on CDS tools can be solicited in multiple ways, and it contains valuable and actionable suggestions to improve CDS alerts. Additionally, end users appreciate knowing their feedback is being received and may also make other suggestions to improve the electronic health record. Incorporation of end-user feedback into CDS monitoring, evaluation, and remediation is a way to improve CDS.
Collapse
Affiliation(s)
- David Rubins
- Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
- Digital, Mass General Brigham, Boston, Massachusetts, United States
| | - Allison B. McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Sayon Dutta
- Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
- Digital, Mass General Brigham, Boston, Massachusetts, United States
- Department of Emergency Medicine, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, United States
| | - Dustin S. McEvoy
- Digital, Mass General Brigham, Boston, Massachusetts, United States
| | - Lorraine Patterson
- HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Amy Miller
- Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
- Digital, Mass General Brigham, Boston, Massachusetts, United States
| | - John G. Jackson
- HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Gianna Zuccotti
- Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States
- Digital, Mass General Brigham, Boston, Massachusetts, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| |
Collapse
|
7
|
Chaparro JD, Beus JM, Dziorny AC, Hagedorn PA, Hernandez S, Kandaswamy S, Kirkendall ES, McCoy AB, Muthu N, Orenstein EW. Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts. Appl Clin Inform 2022; 13:560-568. [PMID: 35613913 PMCID: PMC9132737 DOI: 10.1055/s-0042-1748856] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022] Open
Abstract
Interruptive clinical decision support systems, both within and outside of electronic health records, are a resource that should be used sparingly and monitored closely. Excessive use of interruptive alerting can quickly lead to alert fatigue and decreased effectiveness and ignoring of alerts. In this review, we discuss the evidence for effective alert stewardship as well as practices and methods we have found useful to assess interruptive alert burden, reduce excessive firings, optimize alert effectiveness, and establish quality governance at our institutions. We also discuss the importance of a holistic view of the alerting ecosystem beyond the electronic health record.
Collapse
Affiliation(s)
- Juan D Chaparro
- Division of Clinical Informatics, Nationwide Children's Hospital, Columbus, Ohio, United States.,Departments of Pediatrics and Biomedical Informatics, The Ohio State University College of Medicine, Columbus, Ohio, United States
| | - Jonathan M Beus
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States.,Children's Healthcare of Atlanta, Atlanta, Georgia, United States
| | - Adam C Dziorny
- Department of Pediatrics, University of Rochester School of Medicine, Rochester, New York, United States
| | - Philip A Hagedorn
- Department of Pediatrics, University of Cincinnati, College of Medicine, Cincinnati, Ohio, United States.,Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
| | - Sean Hernandez
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States.,Department of General Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
| | - Swaminathan Kandaswamy
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
| | - Eric S Kirkendall
- Center for Healthcare Innovation, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States.,Department of Pediatrics, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States.,Center for Biomedical Informatics, Wake Forest School of Medicine, Winston-Salem NC, United States
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States
| | - Naveen Muthu
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States.,Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Evan W Orenstein
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States.,Children's Healthcare of Atlanta, Atlanta, Georgia, United States
| |
Collapse
|
8
|
Hughes AEO, Jackups R. Clinical Decision Support for Laboratory Testing. Clin Chem 2021; 68:402-412. [PMID: 34871351 DOI: 10.1093/clinchem/hvab201] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2021] [Accepted: 08/24/2021] [Indexed: 01/16/2023]
Abstract
BACKGROUND As technology enables new and increasingly complex laboratory tests, test utilization presents a growing challenge for healthcare systems. Clinical decision support (CDS) refers to digital tools that present providers with clinically relevant information and recommendations, which have been shown to improve test utilization. Nevertheless, individual CDS applications often fail, and implementation remains challenging. CONTENT We review common classes of CDS tools grounded in examples from the literature as well as our own institutional experience. In addition, we present a practical framework and specific recommendations for effective CDS implementation. SUMMARY CDS encompasses a rich set of tools that have the potential to drive significant improvements in laboratory testing, especially with respect to test utilization. Deploying CDS effectively requires thoughtful design and careful maintenance, and structured processes focused on quality improvement and change management play an important role in achieving these goals.
Collapse
Affiliation(s)
- Andrew E O Hughes
- Department of Pathology and Immunology, Washington University School of Medicine, St. Louis, MO, USA
| | - Ronald Jackups
- Department of Pathology and Immunology, Washington University School of Medicine, St. Louis, MO, USA
| |
Collapse
|
9
|
Olakotan OO, Mohd Yusof M. The appropriateness of clinical decision support systems alerts in supporting clinical workflows: A systematic review. Health Informatics J 2021; 27:14604582211007536. [PMID: 33853395 DOI: 10.1177/14604582211007536] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2023]
Abstract
A CDSS generates a high number of inappropriate alerts that interrupt the clinical workflow. As a result, clinicians silence, disable, or ignore alerts, thereby undermining patient safety. Therefore, the effectiveness and appropriateness of CDSS alerts need to be evaluated. A systematic review was carried out to identify the factors that affect CDSS alert appropriateness in supporting clinical workflow. Seven electronic databases (PubMed, Scopus, ACM, Science Direct, IEEE, Ovid Medline, and Ebscohost) were searched for English language articles published between 1997 and 2018. Seventy six papers met the inclusion criteria, of which 26, 24, 15, and 11 papers are retrospective cohort, qualitative, quantitative, and mixed-method studies, respectively. The review highlights various factors influencing the appropriateness and efficiencies of CDSS alerts. These factors are categorized into technology, human, organization, and process aspects using a combination of approaches, including socio-technical framework, five rights of CDSS, and Lean. Most CDSS alerts were not properly designed based on human factor methods and principles, explaining high alert overrides in clinical practices. The identified factors and recommendations from the review may offer valuable insights into how CDSS alerts can be designed appropriately to support clinical workflow.
Collapse
|
10
|
Mahadevaiah G, Rv P, Bermejo I, Jaffray D, Dekker A, Wee L. Artificial intelligence-based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance. Med Phys 2021; 47:e228-e235. [PMID: 32418341 PMCID: PMC7318221 DOI: 10.1002/mp.13562] [Citation(s) in RCA: 35] [Impact Index Per Article: 11.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2019] [Revised: 04/27/2019] [Accepted: 04/27/2019] [Indexed: 01/16/2023] Open
Abstract
Background Recent advances in machine and deep learning based on an increased availability of clinical data have fueled renewed interest in computerized clinical decision support systems (CDSSs). CDSSs have shown great potential to improve healthcare, increase patient safety and reduce costs. However, the use of CDSSs is not without pitfalls, as an inadequate or faulty CDSS can potentially deteriorate the quality of healthcare and put patients at risk. In addition, the adoption of a CDSS might fail because its intended users ignore the output of the CDSS due to lack of trust, relevancy or actionability. Aim In this article, we provide guidance based on literature for the different aspects involved in the adoption of a CDSS with a special focus on machine and deep learning based systems: selection, acceptance testing, commissioning, implementation and quality assurance. Results A rigorous selection process will help identify the CDSS that best fits the preferences and requirements of the local site. Acceptance testing will make sure that the selected CDSS fulfills the defined specifications and satisfies the safety requirements. The commissioning process will prepare the CDSS for safe clinical use at the local site. An effective implementation phase should result in an orderly roll out of the CDSS to the well‐trained end‐users whose expectations have been managed. And finally, quality assurance will make sure that the performance of the CDSS is maintained and that any issues are promptly identified and solved. Conclusion We conclude that a systematic approach to the adoption of a CDSS will help avoid pitfalls, improve patient safety and increase the chances of success.
Collapse
Affiliation(s)
| | - Prasad Rv
- Philips Research India, Bangalore, 560045, India
| | - Inigo Bermejo
- Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre, Maastricht, 6229 ET, Netherlands
| | - David Jaffray
- Princess Margaret Cancer Centre, University Health Network, Toronto, ON, M5G 2M9, Canada
| | - Andre Dekker
- Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre, Maastricht, 6229 ET, Netherlands
| | - Leonard Wee
- Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre, Maastricht, 6229 ET, Netherlands
| |
Collapse
|
11
|
Kannan V, Basit MA, Bajaj P, Carrington AR, Donahue IB, Flahaven EL, Medford R, Melaku T, Moran BA, Saldana LE, Willett DL, Youngblood JE, Toomay SM. User stories as lightweight requirements for agile clinical decision support development. J Am Med Inform Assoc 2021; 26:1344-1354. [PMID: 31512730 PMCID: PMC6798563 DOI: 10.1093/jamia/ocz123] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2019] [Revised: 05/17/2019] [Accepted: 07/01/2019] [Indexed: 02/02/2023] Open
Abstract
Objective We sought to demonstrate applicability of user stories, progressively elaborated by testable acceptance criteria, as lightweight requirements for agile development of clinical decision support (CDS). Materials and Methods User stories employed the template: As a [type of user], I want [some goal] so that [some reason]. From the “so that” section, CDS benefit measures were derived. Detailed acceptance criteria were elaborated through ensuing conversations. We estimated user story size with “story points,” and depicted multiple user stories with a use case diagram or feature breakdown structure. Large user stories were split to fit into 2-week iterations. Results One example user story was: As a rheumatologist, I want to be advised if my patient with rheumatoid arthritis is not on a disease-modifying anti-rheumatic drug (DMARD), so that they receive optimal therapy and can experience symptom improvement. This yielded a process measure (DMARD use), and an outcome measure (Clinical Disease Activity Index). Following implementation, the DMARD nonuse rate decreased from 3.7% to 1.4%. Patients with a high Clinical Disease Activity Index improved from 13.7% to 7%. For a thromboembolism prevention CDS project, diagrams organized multiple user stories. Discussion User stories written in the clinician’s voice aid CDS governance and lead naturally to measures of CDS effectiveness. Estimation of relative story size helps plan CDS delivery dates. User stories prove to be practical even on larger projects. Conclusions User stories concisely communicate the who, what, and why of a CDS request, and serve as lightweight requirements for agile development to meet the demand for increasingly diverse CDS.
Collapse
Affiliation(s)
- Vaishnavi Kannan
- Clinical Informatics, University of Texas Southwestern Health System, Dallas, Texas, USA.,Health System Information Resources Department, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Mujeeb A Basit
- Clinical Informatics, University of Texas Southwestern Health System, Dallas, Texas, USA.,Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Puneet Bajaj
- Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Angela R Carrington
- Health System Information Resources Department, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Irma B Donahue
- Health System Information Resources Department, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Emily L Flahaven
- Clinical Informatics, University of Texas Southwestern Health System, Dallas, Texas, USA
| | - Richard Medford
- Clinical Informatics, University of Texas Southwestern Health System, Dallas, Texas, USA.,Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Tsedey Melaku
- Clinical Informatics, Parkland Health and Hospital System, Dallas, Texas, USA
| | - Brett A Moran
- Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA.,Clinical Informatics, Parkland Health and Hospital System, Dallas, Texas, USA
| | - Luis E Saldana
- Clinical Informatics, Texas Health Resources, Arlington, Texas, USA
| | - Duwayne L Willett
- Clinical Informatics, University of Texas Southwestern Health System, Dallas, Texas, USA.,Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Josh E Youngblood
- Health System Information Resources Department, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Seth M Toomay
- Clinical Informatics, University of Texas Southwestern Health System, Dallas, Texas, USA.,Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| |
Collapse
|
12
|
Wan PK, Satybaldy A, Huang L, Holtskog H, Nowostawski M. Reducing Alert Fatigue by Sharing Low-Level Alerts With Patients and Enhancing Collaborative Decision Making Using Blockchain Technology: Scoping Review and Proposed Framework (MedAlert). J Med Internet Res 2020; 22:e22013. [PMID: 33112253 PMCID: PMC7657729 DOI: 10.2196/22013] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2020] [Revised: 09/08/2020] [Accepted: 09/12/2020] [Indexed: 01/23/2023] Open
Abstract
Background Clinical decision support (CDS) is a tool that helps clinicians in decision making by generating clinical alerts to supplement their previous knowledge and experience. However, CDS generates a high volume of irrelevant alerts, resulting in alert fatigue among clinicians. Alert fatigue is the mental state of alerts consuming too much time and mental energy, which often results in relevant alerts being overridden unjustifiably, along with clinically irrelevant ones. Consequently, clinicians become less responsive to important alerts, which opens the door to medication errors. Objective This study aims to explore how a blockchain-based solution can reduce alert fatigue through collaborative alert sharing in the health sector, thus improving overall health care quality for both patients and clinicians. Methods We have designed a 4-step approach to answer this research question. First, we identified five potential challenges based on the published literature through a scoping review. Second, a framework is designed to reduce alert fatigue by addressing the identified challenges with different digital components. Third, an evaluation is made by comparing MedAlert with other proposed solutions. Finally, the limitations and future work are also discussed. Results Of the 341 academic papers collected, 8 were selected and analyzed. MedAlert securely distributes low-level (nonlife-threatening) clinical alerts to patients, enabling a collaborative clinical decision. Among the solutions in our framework, Hyperledger (private permissioned blockchain) and BankID (federated digital identity management) have been selected to overcome challenges such as data integrity, user identity, and privacy issues. Conclusions MedAlert can reduce alert fatigue by attracting the attention of patients and clinicians, instead of solely reducing the total number of alerts. MedAlert offers other advantages, such as ensuring a higher degree of patient privacy and faster transaction times compared with other frameworks. This framework may not be suitable for elderly patients who are not technology savvy or in-patients. Future work in validating this framework based on real health care scenarios is needed to provide the performance evaluations of MedAlert and thus gain support for the better development of this idea.
Collapse
Affiliation(s)
- Paul Kengfai Wan
- Department of Manufacturing and Civil Engineering, Norwegian University of Science and Technology, Gjøvik, Norway
| | - Abylay Satybaldy
- Department of Computer Science, Norwegian University of Science and Technology, Gjøvik, Norway
| | - Lizhen Huang
- Department of Manufacturing and Civil Engineering, Norwegian University of Science and Technology, Gjøvik, Norway
| | - Halvor Holtskog
- Department of Industrial Economics and Technology Management, Norwegian University of Science and Technology, Gjøvik, Norway
| | - Mariusz Nowostawski
- Department of Computer Science, Norwegian University of Science and Technology, Gjøvik, Norway
| |
Collapse
|
13
|
Hofer IS, Burns M, Kendale S, Wanderer JP. Realistically Integrating Machine Learning Into Clinical Practice: A Road Map of Opportunities, Challenges, and a Potential Future. Anesth Analg 2020; 130:1115-1118. [PMID: 32287118 DOI: 10.1213/ane.0000000000004575] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Affiliation(s)
- Ira S Hofer
- From the Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at University of California Los Angeles, Los Angeles, California
| | - Michael Burns
- Department of Anesthesiology, University of Michigan School of Medicine, Ann Arbor, Michigan
| | - Samir Kendale
- Department of Anesthesiology, Perioperative Care and Pain Medicine, New York University Langone School of Medicine, New York, New York
| | - Jonathan P Wanderer
- Departments of Anesthesiology and Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee
| |
Collapse
|
14
|
Olakotan OO, Yusof MM. Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow. J Biomed Inform 2020; 106:103453. [PMID: 32417444 DOI: 10.1016/j.jbi.2020.103453] [Citation(s) in RCA: 19] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2019] [Revised: 05/08/2020] [Accepted: 05/09/2020] [Indexed: 02/06/2023]
Abstract
The overwhelming number of medication alerts generated by clinical decision support systems (CDSS) has led to inappropriate alert overrides, which may lead to unintended patient harm. This review highlights the factors affecting the alert appropriateness of CDSS and barriers to the fit of CDSS alert with clinical workflow. A literature review was conducted to identify features and functions pertinent to CDSS alert appropriateness using the five rights of CDSS. Moreover, a process improvement method, namely, Lean, was used as a tool to optimise clinical workflows, and the appropriate design for CDSS alert using a human automation interaction (HAI) model was recommended. Evaluating the appropriateness of CDSS alert and its impact on workflow provided insights into how alerts can be designed and triggered effectively to support clinical workflow. The application of Lean methods and tools to analyse alert efficiencies in supporting workflow in this study provides an in-depth understanding of alert-workflow fit problems and their root cause, which is required for improving CDSS design. The application of the HAI model is recommended in the design of CDSS alerts to support various levels and stages of alert automations, namely, information acquisition and analysis, decision action and action implementation.
Collapse
Affiliation(s)
| | - Maryati Mohd Yusof
- Faculty of Information Science & Technology, Universiti Kebangsaan Malaysia, Bangi, Selangor, Malaysia.
| |
Collapse
|
15
|
Kim HJ, Kim HJ, Park Y, Lee WS, Lim Y, Kim JH. Clinical Genome Data Model (cGDM) provides Interactive Clinical Decision Support for Precision Medicine. Sci Rep 2020; 10:1414. [PMID: 31996707 PMCID: PMC6989462 DOI: 10.1038/s41598-020-58088-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2019] [Accepted: 01/09/2020] [Indexed: 02/02/2023] Open
Abstract
In light of recent developments in genomic technology and the rapid accumulation of genomic information, a major transition toward precision medicine is anticipated. However, the clinical applications of genomic information remain limited. This lag can be attributed to several complex factors, including the knowledge gap between medical experts and bioinformaticians, the distance between bioinformatics workflows and clinical practice, and the unique characteristics of genomic data, which can make interpretation difficult. Here we present a novel genomic data model that allows for more interactive support in clinical decision-making. Informational modelling was used as a basis to design a communication scheme between sophisticated bioinformatics predictions and the representative data relevant to a clinical decision. This study was conducted by a multidisciplinary working group who carried out clinico-genomic workflow analysis and attribute extraction, through Failure Mode and Effects Analysis (FMEA). Based on those results, a clinical genome data model (cGDM) was developed with 8 entities and 46 attributes. The cGDM integrates reliability-related factors that enable clinicians to access the reliability problem of each individual genetic test result as clinical evidence. The proposed cGDM provides a data-layer infrastructure supporting the intellectual interplay between medical experts and informed decision-making.
Collapse
Affiliation(s)
- Hyo Jung Kim
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Hyeong Joon Kim
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Yoomi Park
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Woo Seung Lee
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Younggyun Lim
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Ju Han Kim
- Seoul National University Biomedical Informatics (SNUBI), Division of Biomedical Informatics, Seoul National University College of Medicine, Seoul, Republic of Korea.
| |
Collapse
|
16
|
Aaron S, McEvoy DS, Ray S, Hickman TTT, Wright A. Cranky comments: detecting clinical decision support malfunctions through free-text override reasons. J Am Med Inform Assoc 2019; 26:37-43. [PMID: 30590557 PMCID: PMC6308015 DOI: 10.1093/jamia/ocy139] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2018] [Accepted: 10/08/2018] [Indexed: 11/13/2022] Open
Abstract
Background Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited. Objective Investigate whether user override comments can be used to discover malfunctions. Methods We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: “broken,” “not broken, but could be improved,” and “not broken.” We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth. Results Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738. Discussion Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts. Conclusion Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.
Collapse
Affiliation(s)
- Skye Aaron
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Dustin S McEvoy
- Partners eCare, Partners HealthCare, Boston, Massachusetts, USA
| | - Soumi Ray
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA
| | | | - Adam Wright
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, USA.,Partners eCare, Partners HealthCare, Boston, Massachusetts, USA.,Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA.,Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
17
|
Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar AM. Evaluating the Impact of Interruptive Alerts within a Health System: Use, Response Time, and Cumulative Time Burden. Appl Clin Inform 2019; 10:909-917. [PMID: 31777057 DOI: 10.1055/s-0039-1700869] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022] Open
Abstract
BACKGROUND Health systems often employ interruptive alerts through the electronic health record to improve patient care. However, concerns of "alert fatigue" have been raised, highlighting the importance of understanding the time burden and impact of these alerts on providers. OBJECTIVES Our main objective was to determine the total time providers spent on interruptive alerts in both inpatient and outpatient settings. Our secondary objectives were to analyze dwell time for individual alerts and examine both provider and alert-related factors associated with dwell time variance. METHODS We retrospectively evaluated use and response to the 75 most common interruptive ("popup") alerts between June 1st, 2015 and November 1st, 2016 in a large academic health care system. Alert "dwell time" was calculated as the time between the alert appearing on a provider's screen until it was closed. The total number of alerts and dwell times per provider per month was calculated for inpatient and outpatient alerts and compared across alert type. RESULTS The median number of alerts seen by a provider was 12 per month (IQR 4-34). Overall, 67% of inpatient and 39% of outpatient alerts were closed in under 3 seconds. Alerts related to patient safety and those requiring more than a single click to proceed had significantly longer median dwell times of 5.2 and 6.7 seconds, respectively. The median total monthly time spent by providers viewing alerts was 49 seconds on inpatient alerts and 28 seconds on outpatient alerts. CONCLUSION Most alerts were closed in under 3 seconds and a provider's total time spent on alerts was less than 1 min/mo. Alert fatigue may lie in their interruptive and noncritical nature rather than time burden. Monitoring alert interaction time can function as a valuable metric to assess the impact of alerts on workflow and potentially identify routinely ignored alerts.
Collapse
Affiliation(s)
- Pierre Elias
- Department of Biomedical Informatics, Columbia University, New York, New York, United States
| | - Eric Peterson
- Duke Clinical Research Institute, Duke University, Durham, North Carolina, United States
| | - Bob Wachter
- Department of Medicine, University of California, San Francisco, California, United States
| | - Cary Ward
- Duke Clinical Research Institute, Duke University, Durham, North Carolina, United States
| | - Eric Poon
- Duke Health Technology Solutions, Duke University School of Medicine, Duke University, Durham, North Carolina, United States
| | - Ann Marie Navar
- Duke Clinical Research Institute, Duke University, Durham, North Carolina, United States
| |
Collapse
|
18
|
Ray S, McEvoy DS, Aaron S, Hickman TT, Wright A. Using statistical anomaly detection models to find clinical decision support malfunctions. J Am Med Inform Assoc 2019; 25:862-871. [PMID: 29762678 DOI: 10.1093/jamia/ocy041] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2017] [Accepted: 04/03/2018] [Indexed: 11/13/2022] Open
Abstract
Objective Malfunctions in Clinical Decision Support (CDS) systems occur due to a multitude of reasons, and often go unnoticed, leading to potentially poor outcomes. Our goal was to identify malfunctions within CDS systems. Methods We evaluated 6 anomaly detection models: (1) Poisson Changepoint Model, (2) Autoregressive Integrated Moving Average (ARIMA) Model, (3) Hierarchical Divisive Changepoint (HDC) Model, (4) Bayesian Changepoint Model, (5) Seasonal Hybrid Extreme Studentized Deviate (SHESD) Model, and (6) E-Divisive with Median (EDM) Model and characterized their ability to find known anomalies. We analyzed 4 CDS alerts with known malfunctions from the Longitudinal Medical Record (LMR) and Epic® (Epic Systems Corporation, Madison, WI, USA) at Brigham and Women's Hospital, Boston, MA. The 4 rules recommend lead testing in children, aspirin therapy in patients with coronary artery disease, pneumococcal vaccination in immunocompromised adults and thyroid testing in patients taking amiodarone. Results Poisson changepoint, ARIMA, HDC, Bayesian changepoint and the SHESD model were able to detect anomalies in an alert for lead screening in children and in an alert for pneumococcal conjugate vaccine in immunocompromised adults. EDM was able to detect anomalies in an alert for monitoring thyroid function in patients on amiodarone. Conclusions Malfunctions/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.
Collapse
Affiliation(s)
- Soumi Ray
- Department of General Internal Medicine and Primary Care, Brigham & Women's Hospital, Boston, MA, USA.,Department of Medicine, Harvard Medical School, Boston, MA, USA
| | - Dustin S McEvoy
- Partners Healthcare, Information Systems, Somerville, MA, USA
| | - Skye Aaron
- Department of General Internal Medicine and Primary Care, Brigham & Women's Hospital, Boston, MA, USA
| | - Thu-Trang Hickman
- Department of General Internal Medicine and Primary Care, Brigham & Women's Hospital, Boston, MA, USA
| | - Adam Wright
- Department of General Internal Medicine and Primary Care, Brigham & Women's Hospital, Boston, MA, USA.,Department of Medicine, Harvard Medical School, Boston, MA, USA.,Partners Healthcare, Information Systems, Somerville, MA, USA
| |
Collapse
|
19
|
Orenstein EW, Muthu N, Weitkamp AO, Ferro DF, Zeidlhack MD, Slagle J, Shelov E, Tobias MC. Towards a Maturity Model for Clinical Decision Support Operations. Appl Clin Inform 2019; 10:810-819. [PMID: 31667818 PMCID: PMC6821535 DOI: 10.1055/s-0039-1697905] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2019] [Accepted: 08/14/2019] [Indexed: 12/21/2022] Open
Abstract
Clinical decision support (CDS) systems delivered through the electronic health record are an important element of quality and safety initiatives within a health care system. However, managing a large CDS knowledge base can be an overwhelming task for informatics teams. Additionally, it can be difficult for these informatics teams to communicate their goals with external operational stakeholders and define concrete steps for improvement. We aimed to develop a maturity model that describes a roadmap toward organizational functions and processes that help health care systems use CDS more effectively to drive better outcomes. We developed a maturity model for CDS operations through discussions with health care leaders at 80 organizations, iterative model development by four clinical informaticists, and subsequent review with 19 health care organizations. We ceased iterations when feedback from three organizations did not result in any changes to the model. The proposed CDS maturity model includes three main "pillars": "Content Creation," "Analytics and Reporting," and "Governance and Management." Each pillar contains five levels-advancing along each pillar provides CDS teams a deeper understanding of the processes CDS systems are intended to improve. A "roof" represents the CDS functions that become attainable after advancing along each of the pillars. Organizations are not required to advance in order and can develop in one pillar separately from another. However, we hypothesize that optimal deployment of preceding levels and advancing in tandem along the pillars increase the value of organizational investment in higher levels of CDS maturity. In addition to describing the maturity model and its development, we also provide three case studies of health care organizations using the model for self-assessment and determine next steps in CDS development.
Collapse
Affiliation(s)
- Evan W. Orenstein
- Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
- Division of Hospital Medicine, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
| | - Naveen Muthu
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
| | - Asli O. Weitkamp
- Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, United States
| | - Daria F. Ferro
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
| | | | - Jason Slagle
- Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, Tennessee, United States
| | - Eric Shelov
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
| | - Marc C. Tobias
- Phrase Health Inc., Philadelphia, Pennsylvania, United States
| |
Collapse
|
20
|
Thayer JG, Miller JM, Fiks AG, Tague L, Grundmeier RW. Assessing the Safety of Custom Web-Based Clinical Decision Support Systems in Electronic Health Records: A Case Study. Appl Clin Inform 2019; 10:237-246. [PMID: 30943572 DOI: 10.1055/s-0039-1683985] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/02/2023] Open
Abstract
BACKGROUND With the widespread adoption of vendor-supplied electronic health record (EHR) systems, clinical decision support (CDS) customization efforts beyond those anticipated by the vendor may require the use of technologies external to the EHR such as web services. Pursuing such customizations, however, is not without risk. Validating the expected behavior of a customized CDS system in the high-volume, complex environment of the live EHR is a challenging problem. OBJECTIVE This article identifies technology failures that impacted clinical care related to web service-based advanced custom CDS systems embedded in the complex sociotechnical context of a production EHR. METHODS In an academic health system's primary care network, we performed an inventory of incidents between January 1, 2008 and December 31, 2016 related to a customized CDS system and performed a targeted review of changes in the CDS source code. Additional feedback on the root cause of individual incidents was obtained through interviews with members of the CDS project teams. RESULTS We identified five CDS malfunctions that impaired clinical workflow. The mechanisms for these failures are mapped to four characteristics of well-behaved applications: (1) system integrity; (2) data integrity; (3) reliability; and (4) scalability. Over the 9-year period, two malfunctions of the customized CDS significantly impaired clinical workflow for a total of 5 hours. Lesser impacts-loss of individual features with straightforward workarounds-arose from three malfunctions, which affected users on 53 days. DISCUSSION Advanced customization of EHRs for the purpose of CDS can present significant risks to clinical workflow. CONCLUSION This case study highlights that advanced customization of CDS within a commercial EHR may support care for complex patient populations, but ongoing monitoring and support is required to ensure its safe use.
Collapse
Affiliation(s)
- Jeritt G Thayer
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Jeffrey M Miller
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Alexander G Fiks
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States.,Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, United States
| | - Linda Tague
- Department of Information Services, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
| | - Robert W Grundmeier
- Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States.,Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, United States
| |
Collapse
|
21
|
Lyell D, Magrabi F, Coiera E. Reduced Verification of Medication Alerts Increases Prescribing Errors. Appl Clin Inform 2019; 10:66-76. [PMID: 30699458 DOI: 10.1055/s-0038-1677009] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/31/2022] Open
Abstract
OBJECTIVE Clinicians using clinical decision support (CDS) to prescribe medications have an obligation to ensure that prescriptions are safe. One option is to verify the safety of prescriptions if there is uncertainty, for example, by using drug references. Supervisory control experiments in aviation and process control have associated errors, with reduced verification arising from overreliance on decision support. However, it is unknown whether this relationship extends to clinical decision-making. Therefore, we examine whether there is a relationship between verification behaviors and prescribing errors, with and without CDS medication alerts, and whether task complexity mediates this. METHODS A total of 120 students in the final 2 years of a medical degree prescribed medicines for patient scenarios using a simulated electronic prescribing system. CDS (correct, incorrect, and no CDS) and task complexity (low and high) were varied. Outcomes were omission (missed prescribing errors) and commission errors (accepted false-positive alerts). Verification measures were access of drug references and view time percentage of task time. RESULTS Failure to access references for medicines with prescribing errors increased omission errors with no CDS (high-complexity: χ 2(1) = 12.716; p < 0.001) and incorrect CDS (Fisher's exact; low-complexity: p = 0.002; high-complexity: p = 0.001). Failure to access references for false-positive alerts increased commission errors (low-complexity: χ 2(1) = 16.673, p < 0.001; high-complexity: χ 2(1) = 18.690, p < 0.001). Fewer participants accessed relevant references with incorrect CDS compared with no CDS (McNemar; low-complexity: p < 0.001; high-complexity: p < 0.001). Lower view time percentages increased omission (F(3, 361.914) = 4.498; p = 0.035) and commission errors (F(1, 346.223) = 2.712; p = 0.045). View time percentages were lower in CDS-assisted conditions compared with unassisted conditions (F(2, 335.743) = 10.443; p < 0.001). DISCUSSION The presence of CDS reduced verification of prescription safety. When CDS was incorrect, reduced verification was associated with increased prescribing errors. CONCLUSION CDS can be incorrect, and verification provides one mechanism to detect errors. System designers need to facilitate verification without increasing workload or eliminating the benefits of correct CDS.
Collapse
Affiliation(s)
- David Lyell
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, New South Wales, Australia
| | - Farah Magrabi
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, New South Wales, Australia
| | - Enrico Coiera
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, New South Wales, Australia
| |
Collapse
|
22
|
Jones BE, Collingridge DS, Vines CG, Post H, Holmen J, Allen TL, Haug P, Weir CR, Dean NC. CDS in a Learning Health Care System: Identifying Physicians' Reasons for Rejection of Best-Practice Recommendations in Pneumonia through Computerized Clinical Decision Support. Appl Clin Inform 2019; 10:1-9. [PMID: 30602195 PMCID: PMC6327742 DOI: 10.1055/s-0038-1676587] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2018] [Accepted: 11/09/2018] [Indexed: 12/14/2022] Open
Abstract
BACKGROUND Local implementation of guidelines for pneumonia care is strongly recommended, but the context of care that affects implementation is poorly understood. In a learning health care system, computerized clinical decision support (CDS) provides an opportunity to both improve and track practice, providing insights into the implementation process. OBJECTIVES This article examines physician interactions with a CDS to identify reasons for rejection of guideline recommendations. METHODS We implemented a multicenter bedside CDS for the emergency department management of pneumonia that integrated patient data with guideline-based recommendations. We examined the frequency of adoption versus rejection of recommendations for site-of-care and antibiotic selection. We analyzed free-text responses provided by physicians explaining their clinical reasoning for rejection, using concept mapping and thematic analysis. RESULTS Among 1,722 patient episodes, physicians rejected recommendations to send a patient home in 24%, leaving text in 53%; reasons for rejection of the recommendations included additional or alternative diagnoses beyond pneumonia, and comorbidities or signs of physiologic derangement contributing to risk of outpatient failure that were not processed by the CDS. Physicians rejected broad-spectrum antibiotic recommendations in 10%, leaving text in 76%; differences in pathogen risk assessment, additional patient information, concern about antibiotic properties, and admitting physician preferences were given as reasons for rejection. CONCLUSION While adoption of CDS recommendations for pneumonia was high, physicians rejecting recommendations frequently provided feedback, reporting alternative diagnoses, additional individual patient characteristics, and provider preferences as major reasons for rejection. CDS that collects user feedback is feasible and can contribute to a learning health system.
Collapse
Affiliation(s)
- Barbara E. Jones
- VA Salt Lake City IDEAS Center, VA Salt Lake City Healthcare System, Salt Lake City, Utah, United States
- Division of Pulmonary and Critical Care Medicine, University of Utah, Salt Lake City, Utah, United States
| | | | | | - Herman Post
- Homer Warner Center for Informatics, Intermountain Healthcare, Murray, Utah, United States
| | - John Holmen
- Homer Warner Center for Informatics, Intermountain Healthcare, Murray, Utah, United States
| | - Todd L. Allen
- Department of Emergency Medicine, Intermountain Healthcare, Murray, Utah, United States
| | - Peter Haug
- Homer Warner Center for Informatics, Intermountain Healthcare, Murray, Utah, United States
| | - Charlene R. Weir
- Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, United States
| | - Nathan C. Dean
- Division of Pulmonary and Critical Care Medicine, Intermountain Healthcare and University of Utah, Murray, Utah, United States
| |
Collapse
|
23
|
Basit MA, Baldwin KL, Kannan V, Flahaven EL, Parks CJ, Ott JM, Willett DL. Agile Acceptance Test-Driven Development of Clinical Decision Support Advisories: Feasibility of Using Open Source Software. JMIR Med Inform 2018; 6:e23. [PMID: 29653922 PMCID: PMC5924365 DOI: 10.2196/medinform.9679] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/26/2017] [Revised: 03/22/2018] [Accepted: 03/23/2018] [Indexed: 12/01/2022] Open
Abstract
Background Moving to electronic health records (EHRs) confers substantial benefits but risks unintended consequences. Modern EHRs consist of complex software code with extensive local configurability options, which can introduce defects. Defects in clinical decision support (CDS) tools are surprisingly common. Feasible approaches to prevent and detect defects in EHR configuration, including CDS tools, are needed. In complex software systems, use of test–driven development and automated regression testing promotes reliability. Test–driven development encourages modular, testable design and expanding regression test coverage. Automated regression test suites improve software quality, providing a “safety net” for future software modifications. Each automated acceptance test serves multiple purposes, as requirements (prior to build), acceptance testing (on completion of build), regression testing (once live), and “living” design documentation. Rapid-cycle development or “agile” methods are being successfully applied to CDS development. The agile practice of automated test–driven development is not widely adopted, perhaps because most EHR software code is vendor-developed. However, key CDS advisory configuration design decisions and rules stored in the EHR may prove amenable to automated testing as “executable requirements.” Objective We aimed to establish feasibility of acceptance test–driven development of clinical decision support advisories in a commonly used EHR, using an open source automated acceptance testing framework (FitNesse). Methods Acceptance tests were initially constructed as spreadsheet tables to facilitate clinical review. Each table specified one aspect of the CDS advisory’s expected behavior. Table contents were then imported into a test suite in FitNesse, which queried the EHR database to automate testing. Tests and corresponding CDS configuration were migrated together from the development environment to production, with tests becoming part of the production regression test suite. Results We used test–driven development to construct a new CDS tool advising Emergency Department nurses to perform a swallowing assessment prior to administering oral medication to a patient with suspected stroke. Test tables specified desired behavior for (1) applicable clinical settings, (2) triggering action, (3) rule logic, (4) user interface, and (5) system actions in response to user input. Automated test suite results for the “executable requirements” are shown prior to building the CDS alert, during build, and after successful build. Conclusions Automated acceptance test–driven development and continuous regression testing of CDS configuration in a commercial EHR proves feasible with open source software. Automated test–driven development offers one potential contribution to achieving high-reliability EHR configuration. Vetting acceptance tests with clinicians elicits their input on crucial configuration details early during initial CDS design and iteratively during rapid-cycle optimization.
Collapse
Affiliation(s)
- Mujeeb A Basit
- University of Texas Southwestern Medical Center, Dallas, TX, United States
| | - Krystal L Baldwin
- University of Texas Southwestern Medical Center, Dallas, TX, United States
| | - Vaishnavi Kannan
- University of Texas Southwestern Medical Center, Dallas, TX, United States
| | - Emily L Flahaven
- University of Texas Southwestern Medical Center, Dallas, TX, United States
| | - Cassandra J Parks
- University of Texas Southwestern Medical Center, Dallas, TX, United States
| | - Jason M Ott
- University of Texas Southwestern Medical Center, Dallas, TX, United States
| | - Duwayne L Willett
- University of Texas Southwestern Medical Center, Dallas, TX, United States
| |
Collapse
|
24
|
Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The Value of Monitoring Clinical Decision Support Interventions. Appl Clin Inform 2018. [PMID: 29514353 DOI: 10.1055/s-0038-1632397] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND Well-functioning clinical decision support (CDS) can facilitate provider workflow, improve patient care, promote better outcomes, and reduce costs. However, poorly functioning CDS may lead to alert fatigue, cause providers to ignore important CDS interventions, and increase provider dissatisfaction. OBJECTIVE The purpose of this article is to describe one institution's experience in implementing a program to create and maintain properly functioning CDS by systematically monitoring CDS firing rates and patterns. METHODS Four types of CDS monitoring activities were implemented as part of the CDS lifecycle. One type of monitoring occurs prior to releasing active CDS, while the other types occur at different points after CDS activation. RESULTS Two hundred and forty-eight CDS interventions were monitored over a 2-year period. The rate of detecting a malfunction or significant opportunity for improvement was 37% during preactivation and 18% during immediate postactivation monitoring. Monitoring also informed the process of responding to user feedback about alerts. Finally, an automated alert detection tool identified 128 instances of alert pattern change over the same period. A subset of cases was evaluated by knowledge engineers to identify true and false positives, the results of which were used to optimize the tool's pattern detection algorithms. CONCLUSION CDS monitoring can identify malfunctions and/or significant improvement opportunities even after careful design and robust testing. CDS monitoring provides information when responding to user feedback. Ongoing, continuous, and automated monitoring can detect malfunctions in real time, before users report problems. Therefore, CDS monitoring should be part of any systematic program of implementing and maintaining CDS.
Collapse
|