1
|
Cresswell K, de Keizer N, Magrabi F, Williams R, Rigby M, Prgomet M, Kukhareva P, Wong ZSY, Scott P, Craven CK, Georgiou A, Medlock S, Brender McNair J, Ammenwerth E. Evaluating Artificial Intelligence in Clinical Settings-Let Us Not Reinvent the Wheel. J Med Internet Res 2024; 26:e46407. [PMID: 39110494 PMCID: PMC11339570 DOI: 10.2196/46407] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Revised: 04/20/2023] [Accepted: 03/02/2024] [Indexed: 08/24/2024] Open
Abstract
Given the requirement to minimize the risks and maximize the benefits of technology applications in health care provision, there is an urgent need to incorporate theory-informed health IT (HIT) evaluation frameworks into existing and emerging guidelines for the evaluation of artificial intelligence (AI). Such frameworks can help developers, implementers, and strategic decision makers to build on experience and the existing empirical evidence base. We provide a pragmatic conceptual overview of selected concrete examples of how existing theory-informed HIT evaluation frameworks may be used to inform the safe development and implementation of AI in health care settings. The list is not exhaustive and is intended to illustrate applications in line with various stakeholder requirements. Existing HIT evaluation frameworks can help to inform AI-based development and implementation by supporting developers and strategic decision makers in considering relevant technology, user, and organizational dimensions. This can facilitate the design of technologies, their implementation in user and organizational settings, and the sustainability and scalability of technologies.
Collapse
Affiliation(s)
- Kathrin Cresswell
- Usher Institute, The University of Edinburgh, Usher Building, Edinburgh, United Kingdom
| | - Nicolette de Keizer
- Amsterdam UMC, University of Amsterdam, Medical Informatics, Amsterdam, Netherlands
- Amsterdam Public Health Research Institute, Digital Health and Quality of Care, Amsterdam, Netherlands
| | - Farah Magrabi
- Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Robin Williams
- Institute for the Study of Science, Technology and Innovation, The University of Edinburgh, Edinburgh, United Kingdom
| | - Michael Rigby
- School of Social, Political and Global Studies and School of Primary, Community and Social Care, Keele University, Keele, United Kingdom
| | - Mirela Prgomet
- Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, Australia
| | - Polina Kukhareva
- Department of Biomedical Informatics, University of Utah, Utah, UT, United States
| | | | - Philip Scott
- University of Wales Trinity St David, Swansea, United Kingdom
| | - Catherine K Craven
- University of Texas Health Science Center, San Antonio, TX, United States
| | - Andrew Georgiou
- Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Stephanie Medlock
- Amsterdam UMC, University of Amsterdam, Medical Informatics, Amsterdam, Netherlands
- Amsterdam Public Health, Methodology & Aging & Later Life, Amsterdam, Netherlands
| | - Jytte Brender McNair
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Elske Ammenwerth
- Institute of Medical Informatics, Private University for Health Sciences and Health Technology, UMIT TIROL, Hall in Tirol, Austria
| |
Collapse
|
2
|
Molloy MJ, Zackoff M, Gifford A, Hagedorn P, Tegtmeyer K, Britto MT, Dewan M. Usability Testing of Situation Awareness Clinical Decision Support in the Intensive Care Unit. Appl Clin Inform 2024; 15:327-334. [PMID: 38378044 PMCID: PMC11062760 DOI: 10.1055/a-2272-6184] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Accepted: 02/18/2024] [Indexed: 02/22/2024] Open
Abstract
OBJECTIVE Our objective was to evaluate the usability of an automated clinical decision support (CDS) tool previously implemented in the pediatric intensive care unit (PICU) to promote shared situation awareness among the medical team to prevent serious safety events within children's hospitals. METHODS We conducted a mixed-methods usability evaluation of a CDS tool in a PICU at a large, urban, quaternary, free-standing children's hospital in the Midwest. Quantitative assessment was done using the system usability scale (SUS), while qualitative assessment involved think-aloud usability testing. The SUS was scored according to survey guidelines. For think-aloud testing, task times were calculated, and means and standard deviations were determined, stratified by role. Qualitative feedback from participants and moderator observations were summarized. RESULTS Fifty-one PICU staff members, including physicians, advanced practice providers, nurses, and respiratory therapists, completed the SUS, while ten participants underwent think-aloud usability testing. The overall median usability score was 87.5 (interquartile range: 80-95), with over 96% rating the tool's usability as "good" or "excellent." Task completion times ranged from 2 to 92 seconds, with the quickest completion for reviewing high-risk criteria and the slowest for adding to high-risk criteria. Observations and participant responses from think-aloud testing highlighted positive aspects of learnability and clear display of complex information that is easily accessed, as well as opportunities for improvement in tool integration into clinical workflows. CONCLUSION The PICU Warning Tool demonstrates good usability in the critical care setting. This study demonstrates the value of postimplementation usability testing in identifying opportunities for continued improvement of CDS tools.
Collapse
Affiliation(s)
- Matthew J. Molloy
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
- Division of Hospital Medicine, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
- Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
| | - Matthew Zackoff
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
- Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
| | | | - Philip Hagedorn
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
- Division of Hospital Medicine, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
- Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
| | - Ken Tegtmeyer
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
- Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
| | - Maria T. Britto
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
- James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
| | - Maya Dewan
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
- Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
- Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
- James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
| |
Collapse
|
3
|
Ryan DK, Maclean RH, Balston A, Scourfield A, Shah AD, Ross J. Artificial intelligence and machine learning for clinical pharmacology. Br J Clin Pharmacol 2024; 90:629-639. [PMID: 37845024 DOI: 10.1111/bcp.15930] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2023] [Revised: 10/04/2023] [Accepted: 10/06/2023] [Indexed: 10/18/2023] Open
Abstract
Artificial intelligence (AI) will impact many aspects of clinical pharmacology, including drug discovery and development, clinical trials, personalized medicine, pharmacogenomics, pharmacovigilance and clinical toxicology. The rapid progress of AI in healthcare means clinical pharmacologists should have an understanding of AI and its implementation in clinical practice. As with any new therapy or health technology, it is imperative that AI tools are subject to robust and stringent evaluation to ensure that they enhance clinical practice in a safe and equitable manner. This review serves as an introduction to AI for the clinical pharmacologist, highlighting current applications, aspects of model development and issues surrounding evaluation and deployment. The aim of this article is to empower clinical pharmacologists to embrace and lead on the safe and effective use of AI within healthcare.
Collapse
Affiliation(s)
- David K Ryan
- Department of Clinical Pharmacology, University College London Hospitals NHS Foundation Trust, London, UK
| | - Rory H Maclean
- Department of Clinical Pharmacology, University College London Hospitals NHS Foundation Trust, London, UK
- Institute of Health Informatics, University College London, London, UK
| | - Alfred Balston
- Department of Clinical Pharmacology, Guy's and St Thomas' NHS Foundation Trust, London, UK
| | - Andrew Scourfield
- Department of Clinical Pharmacology, University College London Hospitals NHS Foundation Trust, London, UK
| | - Anoop D Shah
- Department of Clinical Pharmacology, University College London Hospitals NHS Foundation Trust, London, UK
- Institute of Health Informatics, University College London, London, UK
- National Institute for Health Research, University College London Hospitals Biomedical Research Centre, London, UK
| | - Jack Ross
- Department of Clinical Pharmacology, University College London Hospitals NHS Foundation Trust, London, UK
| |
Collapse
|
4
|
Perez A, Fetters MD, Creswell JW, Scerbo M, Kron FW, Gonzalez R, An L, Jimbo M, Klasnja P, Guetterman TC. Enhancing Nonverbal Communication Through Virtual Human Technology: Protocol for a Mixed Methods Study. JMIR Res Protoc 2023; 12:e46601. [PMID: 37279041 PMCID: PMC10282909 DOI: 10.2196/46601] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2023] [Revised: 04/27/2023] [Accepted: 04/27/2023] [Indexed: 06/07/2023] Open
Abstract
BACKGROUND Communication is a critical component of the patient-provider relationship; however, limited research exists on the role of nonverbal communication. Virtual human training is an informatics-based educational strategy that offers various benefits in communication skill training directed at providers. Recent informatics-based interventions aimed at improving communication have mainly focused on verbal communication, yet research is needed to better understand how virtual humans can improve verbal and nonverbal communication and further elucidate the patient-provider dyad. OBJECTIVE The purpose of this study is to enhance a conceptual model that incorporates technology to examine verbal and nonverbal components of communication and develop a nonverbal assessment that will be included in the virtual simulation for further testing. METHODS This study will consist of a multistage mixed methods design, including convergent and exploratory sequential components. A convergent mixed methods study will be conducted to examine the mediating effects of nonverbal communication. Quantitative (eg, MPathic game scores, Kinect nonverbal data, objective structured clinical examination communication score, and Roter Interaction Analysis System and Facial Action Coding System coding of video) and qualitative data (eg, video recordings of MPathic-virtual reality [VR] interventions and student reflections) will be collected simultaneously. Data will be merged to determine the most crucial components of nonverbal behavior in human-computer interaction. An exploratory sequential design will proceed, consisting of a grounded theory qualitative phase. Using theoretical, purposeful sampling, interviews will be conducted with oncology providers probing intentional nonverbal behaviors. The qualitative findings will aid the development of a nonverbal communication model that will be included in a virtual human. The subsequent quantitative strand will incorporate and validate a new automated nonverbal communication behavior assessment into the virtual human simulation, MPathic-VR, by assessing interrater reliability, code interactions, and dyadic data analysis by comparing Kinect responses (system recorded) to manually scored records for specific nonverbal behaviors. Data will be integrated using building integration to develop the automated nonverbal communication behavior assessment and conduct a quality check of these nonverbal features. RESULTS Secondary data from the MPathic-VR randomized controlled trial data set (210 medical students and 840 video recordings of interactions) were analyzed in the first part of this study. Results showed differential experiences by performance in the intervention group. Following the analysis of the convergent design, participants consisting of medical providers (n=30) will be recruited for the qualitative phase of the subsequent exploratory sequential design. We plan to complete data collection by July 2023 to analyze and integrate these findings. CONCLUSIONS The results from this study contribute to the improvement of patient-provider communication, both verbal and nonverbal, including the dissemination of health information and health outcomes for patients. Further, this research aims to transfer to various topical areas, including medication safety, informed consent processes, patient instructions, and treatment adherence between patients and providers. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/46601.
Collapse
Affiliation(s)
- Analay Perez
- Department of Educational Psychology, University of Nebraska-Lincoln, Lincoln, NE, United States
| | - Michael D Fetters
- Department of Family Medicine, University of Michigan, Ann Arbor, MI, United States
| | - John W Creswell
- Department of Family Medicine, University of Michigan, Ann Arbor, MI, United States
| | - Mark Scerbo
- Department of Psychology, Old Dominion University, Norfolk, VA, United States
| | - Frederick W Kron
- Section of General Internal Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, CT, United States
| | - Richard Gonzalez
- Department of Psychology, University of Michigan, Ann Arbor, MI, United States
| | - Lawrence An
- Department of Internal Medicine, University of Michigan, Ann Arbor, MI, United States
| | - Masahito Jimbo
- Department of Family and Community Medicine, University of Illinois College of Medicine, Chicago, IL, United States
| | - Predrag Klasnja
- School of Information, University of Michigan, Ann Arbor, MI, United States
| | - Timothy C Guetterman
- Department of Family Medicine, University of Michigan, Ann Arbor, MI, United States
| |
Collapse
|
5
|
Carr LH, Christ L, Ferro DF. The Electronic Health Record as a Quality Improvement Tool: Exceptional Potential with Special Considerations. Clin Perinatol 2023; 50:473-488. [PMID: 37201992 DOI: 10.1016/j.clp.2023.01.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/20/2023]
Abstract
The electronic health record (EHR) offers an exciting opportunity for quality improvement efforts. An understanding of the nuances of a site's EHR landscape including the best practices in clinical decision support design, basics of data capture, and acknowledgment of the potential unintended consequences of technology change is essential to ensuring effective usage of this powerful tool.
Collapse
Affiliation(s)
- Leah H Carr
- Division of Neonatology, Department of Pediatrics, Children's Hospital of Philadelphia, 3400 Civic Center Boulevard, Philadelphia, PA 19104, USA; Division of Neonatology, Department of Pediatrics, Children's Hospital of Philadelphia Newborn Care at the Hospital of the University of Pennsylvania, 3400 Spruce Street, 8 Ravdin, Philadelphia, PA 19104, USA; Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA; Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, 2716 South Street, Philadelphia, PA 19146, USA.
| | - Lori Christ
- Division of Neonatology, Department of Pediatrics, Children's Hospital of Philadelphia, 3400 Civic Center Boulevard, Philadelphia, PA 19104, USA; Division of Neonatology, Department of Pediatrics, Children's Hospital of Philadelphia Newborn Care at the Hospital of the University of Pennsylvania, 3400 Spruce Street, 8 Ravdin, Philadelphia, PA 19104, USA; Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA
| | - Daria F Ferro
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA; Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, 2716 South Street, Philadelphia, PA 19146, USA; Division of General Pediatrics, Department of Pediatrics, Children's Hospital of Philadelphia, 3400 Civic Center Boulevard, Philadelphia, PA 19104, USA
| |
Collapse
|
6
|
Zhao Y, Zeng H, Zheng H, Wu J, Kong W, Dai G. A bidirectional interaction-based hybrid network architecture for EEG cognitive recognition. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 238:107593. [PMID: 37209578 DOI: 10.1016/j.cmpb.2023.107593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Revised: 05/04/2023] [Accepted: 05/08/2023] [Indexed: 05/22/2023]
Abstract
BACKGROUND AND OBJECTIVE Extracting cognitive representation and computational representation information simultaneously from electroencephalography (EEG) data and constructing corresponding information interaction models can effectively improve the recognition capability of brain cognitive status. However, due to the huge gap in the interaction between the two types of information, existing studies have yet to consider the advantages of the interaction of both. METHODS This paper introduces a novel architecture named the bidirectional interaction-based hybrid network (BIHN) for EEG cognitive recognition. BIHN consists of two networks: a cognitive-based network named CogN (e.g., graph convolution network, GCN; capsule network, CapsNet) and a computing-based network named ComN (e.g., EEGNet). CogN is responsible for extracting cognitive representation features from EEG data, while ComN is responsible for extracting computational representation features. Additionally, a bidirectional distillation-based coadaptation (BDC) algorithm is proposed to facilitate information interaction between CogN and ComN to realize the coadaptation of the two networks through bidirectional closed-loop feedback. RESULTS Cross-subject cognitive recognition experiments were performed on the Fatigue-Awake EEG dataset (FAAD, 2-class classification) and SEED dataset (3-class classification), and hybrid network pairs of GCN + EEGNet and CapsNet + EEGNet were verified. The proposed method achieved average accuracies of 78.76% (GCN + EEGNet) and 77.58% (CapsNet + EEGNet) on FAAD and 55.38% (GCN + EEGNet) and 55.10% (CapsNet + EEGNet) on SEED, outperforming the hybrid networks without the bidirectional interaction strategy. CONCLUSIONS Experimental results show that BIHN can achieve superior performance on two EEG datasets and enhance the ability of both CogN and ComN in EEG processing as well as cognitive recognition. We also validated its effectiveness with different hybrid network pairs. The proposed method could greatly promote the development of brain-computer collaborative intelligence.
Collapse
Affiliation(s)
- Yue Zhao
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
| | - Hong Zeng
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China; Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou 310018, China.
| | - Haohao Zheng
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
| | - Jing Wu
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China
| | - Wanzeng Kong
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China; Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou 310018, China
| | - Guojun Dai
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, China; Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou 310018, China.
| |
Collapse
|
7
|
Molloy M, Hagedorn P, Dewan M. Why Does Current Clinical Decision Support Frequently Fail to Support Clinical Decisions? Pediatr Crit Care Med 2022; 23:670-672. [PMID: 36165945 PMCID: PMC9523478 DOI: 10.1097/pcc.0000000000003000] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Affiliation(s)
- Matthew Molloy
- Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, Ohio
- Division of Hospital Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
| | - Philip Hagedorn
- Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, Ohio
- Division of Hospital Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
| | - Maya Dewan
- Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, Ohio
- Division of Critical Care Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
- Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
- James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
| |
Collapse
|
8
|
Abstract
OBJECTIVES To assess the current landscape of clinical decision support (CDS) tools in PICUs in order to identify priority areas of focus in this field. DESIGN International, quantitative, cross-sectional survey. SETTING Role-specific, web-based survey administered in November and December 2020. SUBJECTS Medical directors, bedside nurses, attending physicians, and residents/advanced practice providers at Pediatric Acute Lung Injury and Sepsis Network-affiliated PICUs. INTERVENTIONS None. MEASUREMENTS AND MAIN RESULTS The survey was completed by 109 respondents from 45 institutions, primarily attending physicians from university-affiliated PICUs in the United States. The most commonly used CDS tools were people-based resources (93% used always or most of the time) and laboratory result highlighting (86%), with order sets, order-based alerts, and other electronic CDS tools also used frequently. The most important goal providers endorsed for CDS tools were a proven impact on patient safety and an evidence base for their use. Negative perceptions of CDS included concerns about diminished critical thinking and the burden of intrusive processes on providers. Routine assessment of existing CDS was rare, with infrequent reported use of observation to assess CDS impact on workflows or measures of individual alert burden. CONCLUSIONS Although providers share some consensus over CDS utility, we identified specific priority areas of research focus. Consensus across practitioners exists around the importance of evidence-based CDS tools having a proven impact on patient safety. Despite broad presence of CDS tools in PICUs, practitioners continue to view them as intrusive and with concern for diminished critical thinking. Deimplementing ineffective CDS may mitigate this burden, though postimplementation evaluation of CDS is rare.
Collapse
|
9
|
Abstract
OBJECTIVE Human factors and ergonomics (HF/E) frameworks and methods are becoming embedded in the health informatics community. There is now broad recognition that health informatics tools must account for the diverse needs, characteristics, and abilities of end users, as well as their context of use. The objective of this review is to synthesize the current nature and scope of HF/E integration into the health informatics community. METHODS Because the focus of this synthesis is on understanding the current integration of the HF/E and health informatics research communities, we manually reviewed all manuscripts published in primary HF/E and health informatics journals during 2020. RESULTS HF/E-focused health informatics studies included in this synthesis focused heavily on EHR customizations, specifically clinical decision support customizations and customized data displays, and on mobile health innovations. While HF/E methods aimed to jointly improve end user safety, performance, and satisfaction, most HF/E-focused health informatics studies measured only end user satisfaction. CONCLUSION HF/E-focused health informatics researchers need to identify and communicate methodological standards specific to health informatics, to better synthesize findings across resource intensive HF/E-focused health informatics studies. Important gaps in the HF/E design and evaluation process should be addressed in future work, including support for technology development platforms and training programs so that health informatics designers are as diverse as end users.
Collapse
|
10
|
Montagna S, Mariani S, Gamberini E, Ricci A, Zambonelli F. Complementing Agents with Cognitive Services: A Case Study in Healthcare. J Med Syst 2020; 44:188. [PMID: 32930870 PMCID: PMC7497514 DOI: 10.1007/s10916-020-01621-7] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2020] [Accepted: 07/16/2020] [Indexed: 11/24/2022]
Abstract
Personal Agents (PAs) have longly been explored as assistants to support users in their daily activities. Surprisingly, few works refer to the adoption of PAs in the healthcare domain, where they can assist physicians' activities reducing medical errors. Although literature proposes different approaches for modelling and engineering PAs, none of them discusses how they can be integrated with cognitive services in order to empower their reasoning capabilities. In this paper we present an integration model, specifically devised for healthcare applications, that enhances Belief-Desire-Intention agents reasoning with advanced cognitive capabilities. As a case study, we adopt this integrated model in the critical care path of trauma resuscitation, stepping forward to the vision of Smart Hospitals.
Collapse
Affiliation(s)
| | - Stefano Mariani
- Università degli Studi di Modena e Reggio Emilia, Reggio Emilia, Italy
| | | | | | - Franco Zambonelli
- Università degli Studi di Modena e Reggio Emilia, Reggio Emilia, Italy
| |
Collapse
|
11
|
Guetterman TC, Sakakibara R, Baireddy S, Kron FW, Scerbo MW, Cleary JF, Fetters MD. Medical Students' Experiences and Outcomes Using a Virtual Human Simulation to Improve Communication Skills: Mixed Methods Study. J Med Internet Res 2019; 21:e15459. [PMID: 31774400 PMCID: PMC6906619 DOI: 10.2196/15459] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2019] [Revised: 09/27/2019] [Accepted: 09/28/2019] [Indexed: 12/03/2022] Open
Abstract
BACKGROUND Attending to the wide range of communication behaviors that convey empathy is an important but often underemphasized concept to reduce errors in care, improve patient satisfaction, and improve cancer patient outcomes. A virtual human (VH)-based simulation, MPathic-VR, was developed to train health care providers in empathic communication with patients and in interprofessional settings and evaluated through a randomized controlled trial. OBJECTIVE This mixed methods study aimed to investigate the differential effects of a VH-based simulation developed to train health care providers in empathic patient-provider and interprofessional communication. METHODS We employed a mixed methods intervention design, involving a comparison of 2 quantitative measures-MPathic-VR-calculated scores and the objective structured clinical exam (OSCE) scores-with qualitative reflections by medical students about their experiences. This paper is a secondary, focused analysis of intervention arm data from the larger trial. Students at 3 medical schools in the United States (n=206) received simulation to improve empathic communication skills. We conducted analysis of variance, thematic text analysis, and merging mixed methods analysis. RESULTS OSCE scores were significantly improved for learners in the intervention group (mean 0.806, SD 0.201) compared with the control group (mean 0.752, SD 0.198; F1,414=6.09; P=.01). Qualitative analysis revealed 3 major positive themes for the MPathic-VR group learners: gaining useful communication skills, learning awareness of nonverbal skills in addition to verbal skills, and feeling motivated to learn more about communication. Finally, the results of the mixed methods analysis indicated that most of the variation between high, middle, and lower performers was noted about nonverbal behaviors. Medium and high OSCE scorers most often commented on the importance of nonverbal communication. Themes of motivation to learn about communication were only present in middle and high scorers. CONCLUSIONS VHs are a promising strategy for improving empathic communication in health care. Higher performers seemed most engaged to learn, particularly nonverbal skills.
Collapse
|
12
|
What do healthcare professionals need to turn risk models for type 2 diabetes into usable computerized clinical decision support systems? Lessons learned from the MOSAIC project. BMC Med Inform Decis Mak 2019; 19:163. [PMID: 31419982 PMCID: PMC6697904 DOI: 10.1186/s12911-019-0887-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2018] [Accepted: 08/02/2019] [Indexed: 01/12/2023] Open
Abstract
Background To understand user needs, system requirements and organizational conditions towards successful design and adoption of Clinical Decision Support Systems for Type 2 Diabetes (T2D) care built on top of computerized risk models. Methods The holistic and evidence-based CEHRES Roadmap, used to create eHealth solutions through participatory development approach, persuasive design techniques and business modelling, was adopted in the MOSAIC project to define the sequence of multidisciplinary methods organized in three phases, user needs, implementation and evaluation. The research was qualitative, the total number of participants was ninety, about five-seventeen involved in each round of experiment. Results Prediction models for the onset of T2D are built on clinical studies, while for T2D care are derived from healthcare registries. Accordingly, two set of DSSs were defined: the first, T2D Screening, introduces a novel routine; in the second case, T2D Care, DSSs can support managers at population level, and daily practitioners at individual level. In the user needs phase, T2D Screening and solution T2D Care at population level share similar priorities, as both deal with risk-stratification. End-users of T2D Screening and solution T2D Care at individual level prioritize easiness of use and satisfaction, while managers prefer the tools to be available every time and everywhere. In the implementation phase, three Use Cases were defined for T2D Screening, adapting the tool to different settings and granularity of information. Two Use Cases were defined around solutions T2D Care at population and T2D Care at individual, to be used in primary or secondary care. Suitable filtering options were equipped with “attractive” visual analytics to focus the attention of end-users on specific parameters and events. In the evaluation phase, good levels of user experience versus bad level of usability suggest that end-users of T2D Screening perceived the potential, but they are worried about complexity. Usability and user experience were above acceptable thresholds for T2D Care at population and T2D Care at individual. Conclusions By using a holistic approach, we have been able to understand user needs, behaviours and interactions and give new insights in the definition of effective Decision Support Systems to deal with the complexity of T2D care. Electronic supplementary material The online version of this article (10.1186/s12911-019-0887-8) contains supplementary material, which is available to authorized users.
Collapse
|
13
|
Call for Papers: HCI for Biomedical Decision-Making: From Diagnosis to Therapy. J Biomed Inform 2019. [DOI: 10.1016/j.jbi.2019.103214] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
14
|
Savoy A, Militello LG, Patel H, Flanagan ME, Russ AL, Daggy JK, Weiner M, Saleem JJ. A cognitive systems engineering design approach to improve the usability of electronic order forms for medical consultation. J Biomed Inform 2018; 85:138-148. [DOI: 10.1016/j.jbi.2018.07.021] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2017] [Revised: 06/19/2018] [Accepted: 07/29/2018] [Indexed: 12/30/2022]
|
15
|
Kannampallil TG, Denton CA, Shapiro JS, Patel VL. Efficiency of Emergency Physicians: Insights from an Observational Study using EHR Log Files. Appl Clin Inform 2018; 9:99-104. [PMID: 30184241 DOI: 10.1055/s-0037-1621705] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022] Open
Abstract
OBJECTIVE With federal mandates and incentives since the turn of this decade, electronic health records (EHR) have been widely adopted and used for clinical care. Over the last several years, we have seen both positive and negative perspectives on its use. Using an analysis of log files of EHR use, we investigated the nature of EHR use and their effect on an emergency department's (ED) throughput and efficiency. METHODS EHR logs of time spent by attending physicians on EHR-based activities over a 6-week period (n = 2,304 patients) were collected. For each patient encounter, physician activities in the EHR were categorized into four activities: documentation, review, orders, and navigation. Four ED-based performance metrics were also captured: door-to-provider time, door-to-doctor time, door-to-disposition time, and length of stay (LOS). Association between the four EHR-based activities and corresponding ED performance metrics were evaluated. RESULTS We found positive correlations between physician review of patient charts, and door-to-disposition time (r = 0.43, p < 0.05), and with LOS (r = 0.48, p < 0.05). There were no statistically significant associations between any of the other performance metrics and EHR activities. CONCLUSION The results highlight that longer time spent on reviewing information on the EHR is potentially associated with decreased ED throughput efficiency. Balancing these competing goals is often a challenge of physicians, and its implications for patient safety is discussed.
Collapse
|
16
|
Abraham J, Kannampallil TG, Jarman A, Sharma S, Rash C, Schiff G, Galanter W. Reasons for computerised provider order entry (CPOE)-based inpatient medication ordering errors: an observational study of voided orders. BMJ Qual Saf 2017; 27:299-307. [PMID: 28698381 DOI: 10.1136/bmjqs-2017-006606] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2017] [Revised: 06/02/2017] [Accepted: 06/06/2017] [Indexed: 01/04/2023]
Abstract
OBJECTIVE Medication voiding is a computerised provider order entry (CPOE)-based discontinuation mechanism that allows clinicians to identify erroneous medication orders. We investigated the accuracy of voiding as an indicator of clinician identification and interception of a medication ordering error, and investigated reasons and root contributors for medication ordering errors. METHOD Using voided orders identified with a void alert, we conducted interviews with ordering and voiding clinicians, followed by patient chart reviews. A structured coding framework was used to qualitatively analyse the reasons for medication ordering errors. We also compared clinician-CPOE-selected (at time of voiding), clinician-reported (interview) and chart review-based reasons for voiding. RESULTS We conducted follow-up interviews on 101 voided orders. The positive predictive value (PPV) of voided orders that were medication ordering errors was 93.1% (95% CI 88.1% to 98.1%, n=94). Using chart review-based reasons as the gold standard, we found that clinician-CPOE-selected reasons were less reflective (PPV=70.2%, 95% CI 61.0% to 79.4%) than clinician-reported (interview) (PPV=86.1%, 95%CI 78.2% to 94.1%) reasons for medication ordering errors. Duplicate (n=44) and improperly composed (n=41) ordering errors were common, often caused by predefined order sets and data entry issues. A striking finding was the use of intentional violations as a mechanism to notify and seek ordering assistance from pharmacy service. Nearly half of the medication ordering errors were voided by pharmacists. DISCUSSION We demonstrated that voided orders effectively captured medication ordering errors. The mismatch between clinician-CPOE-selected and the chart review-based reasons for error emphasises the need for developing standardised operational descriptions for medication ordering errors. Such standardisation can help in accurately identifying, tracking, managing and sharing erroneous orders and their root contributors between healthcare institutions, and with patient safety organisations.
Collapse
Affiliation(s)
- Joanna Abraham
- Department Biomedical and Health Information Sciences, University of Illinois, Chicago, Illinois, USA
| | - Thomas G Kannampallil
- Department of Family Medicine, University of Illinois at Chicago, Chicago, Illinois, USA
| | - Alan Jarman
- Department Biomedical and Health Information Sciences, University of Illinois, Chicago, Illinois, USA
| | - Shivy Sharma
- Department Biomedical and Health Information Sciences, University of Illinois, Chicago, Illinois, USA
| | - Christine Rash
- Department of Pharmacy Practice, University of Illinois at Chicago, Chicago, USA
| | - Gordon Schiff
- Department of General Medicine, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - William Galanter
- Department of Pharmacy Practice, University of Illinois at Chicago, Chicago, USA.,Department of Medicine, University of Illinois, Chicago, Illinois, USA.,Department of Pharmacy Systems, Outcomes and Policy, University of Illinois at Chicago, Chicago, IL, USA
| |
Collapse
|
17
|
Special issue on cognitive informatics methods for interactive clinical systems. J Biomed Inform 2017; 71:207-210. [PMID: 28602905 DOI: 10.1016/j.jbi.2017.06.003] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2017] [Accepted: 06/02/2017] [Indexed: 12/19/2022]
|
18
|
Brunner J, Chuang E, Goldzweig C, Cain CL, Sugar C, Yano EM. User-centered design to improve clinical decision support in primary care. Int J Med Inform 2017; 104:56-64. [PMID: 28599817 DOI: 10.1016/j.ijmedinf.2017.05.004] [Citation(s) in RCA: 45] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2016] [Revised: 03/28/2017] [Accepted: 05/08/2017] [Indexed: 12/28/2022]
Abstract
BACKGROUND A growing literature has demonstrated the ability of user-centered design to make clinical decision support systems more effective and easier to use. However, studies of user-centered design have rarely examined more than a handful of sites at a time, and have frequently neglected the implementation climate and organizational resources that influence clinical decision support. The inclusion of such factors was identified by a systematic review as "the most important improvement that can be made in health IT evaluations." OBJECTIVES (1) Identify the prevalence of four user-centered design practices at United States Veterans Affairs (VA) primary care clinics and assess the perceived utility of clinical decision support at those clinics; (2) Evaluate the association between those user-centered design practices and the perceived utility of clinical decision support. METHODS We analyzed clinic-level survey data collected in 2006-2007 from 170 VA primary care clinics. We examined four user-centered design practices: 1) pilot testing, 2) provider satisfaction assessment, 3) formal usability assessment, and 4) analysis of impact on performance improvement. We used a regression model to evaluate the association between user-centered design practices and the perceived utility of clinical decision support, while accounting for other important factors at those clinics, including implementation climate, available resources, and structural characteristics. We also examined associations separately at community-based clinics and at hospital-based clinics. RESULTS User-centered design practices for clinical decision support varied across clinics: 74% conducted pilot testing, 62% conducted provider satisfaction assessment, 36% conducted a formal usability assessment, and 79% conducted an analysis of impact on performance improvement. Overall perceived utility of clinical decision support was high, with a mean rating of 4.17 (±.67) out of 5 on a composite measure. "Analysis of impact on performance improvement" was the only user-centered design practice significantly associated with perceived utility of clinical decision support, b=.47 (p<.001). This association was present in hospital-based clinics, b=.34 (p<.05), but was stronger at community-based clinics, b=.61 (p<.001). CONCLUSIONS Our findings are highly supportive of the practice of analyzing the impact of clinical decision support on performance metrics. This was the most common user-centered design practice in our study, and was the practice associated with higher perceived utility of clinical decision support. This practice may be particularly helpful at community-based clinics, which are typically less connected to VA medical center resources.
Collapse
Affiliation(s)
- Julian Brunner
- Department of Health Policy and Management, University of California, Los Angeles Fielding School of Public Health, 650 Charles Young Dr. S., Los Angeles, CA, 90095, USA; VA HSR&D Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles Healthcare System (Sepulveda Campus),16111 Plummer Street, Mailcode 152, Sepulveda, CA 91343, USA.
| | - Emmeline Chuang
- Department of Health Policy and Management, University of California, Los Angeles Fielding School of Public Health, 650 Charles Young Dr. S., Los Angeles, CA, 90095, USA
| | - Caroline Goldzweig
- Cedars-Sinai Medical Center,8700 Beverly Blvd., Suite 2211, Los Angeles, CA 90048, USA, USA
| | - Cindy L Cain
- Department of Health Policy and Management, University of California, Los Angeles Fielding School of Public Health, 650 Charles Young Dr. S., Los Angeles, CA, 90095, USA
| | - Catherine Sugar
- Department of Biostatistics, University of California, Los Angeles Fielding School of Public Health, 650 Charles Young Dr. S., Los Angeles, CA, 90095, USA
| | - Elizabeth M Yano
- VA HSR&D Center for the Study of Healthcare Innovation, Implementation, and Policy (CSHIIP), VA Greater Los Angeles Healthcare System (Sepulveda Campus),16111 Plummer Street, Mailcode 152, Sepulveda, CA 91343, USA
| |
Collapse
|
19
|
Morrow D, Hasegawa-Johnson M, Huang T, Schuh W, Azevedo RFL, Gu K, Zhang Y, Roy B, Garcia-Retamero R. A multidisciplinary approach to designing and evaluating Electronic Medical Record portal messages that support patient self-care. J Biomed Inform 2017; 69:63-74. [PMID: 28347856 PMCID: PMC5492515 DOI: 10.1016/j.jbi.2017.03.015] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2016] [Revised: 03/13/2017] [Accepted: 03/22/2017] [Indexed: 10/19/2022]
Abstract
We describe a project intended to improve the use of Electronic Medical Record (EMR) patient portal information by older adults with diverse numeracy and literacy abilities, so that portals can better support patient-centered care. Patient portals are intended to bridge patients and providers by ensuring patients have continuous access to their health information and services. However, they are underutilized, especially by older adults with low health literacy, because they often function more as information repositories than as tools to engage patients. We outline an interdisciplinary approach to designing and evaluating portal-based messages that convey clinical test results so as to support patient-centered care. We first describe a theory-based framework for designing effective messages for patients. This involves analyzing shortcomings of the standard portal message format (presenting numerical test results with little context to guide comprehension) and developing verbally, graphically, video- and computer agent-based formats that enhance context. The framework encompasses theories from cognitive and behavioral science (health literacy, fuzzy trace memory, behavior change) as well as computational/engineering approaches (e.g., image and speech processing models). We then describe an approach to evaluating whether the formats improve comprehension of and responses to the messages about test results, focusing on our methods. The approach combines quantitative (e.g., response accuracy, Likert scale responses) and qualitative (interview) measures, as well as experimental and individual difference methods in order to investigate which formats are more effective, and whether some formats benefit some types of patients more than others. We also report the results of two pilot studies conducted as part of developing the message formats.
Collapse
Affiliation(s)
- Daniel Morrow
- University of Illinois at Urbana-Champaign, Department of Educational Psychology, Champaign, IL, United States.
| | - Mark Hasegawa-Johnson
- University of Illinois at Urbana-Champaign, Department of Electrical and Computer Engineering, Urbana, IL, United States
| | - Thomas Huang
- University of Illinois at Urbana-Champaign, Department of Electrical and Computer Engineering, Urbana, IL, United States
| | - William Schuh
- Carle Foundation Hospital, Urbana, IL, United States
| | | | - Kuangxiao Gu
- University of Illinois at Urbana-Champaign, Department of Electrical and Computer Engineering, Urbana, IL, United States
| | - Yang Zhang
- University of Illinois at Urbana-Champaign, Department of Electrical and Computer Engineering, Urbana, IL, United States
| | - Bidisha Roy
- University of Illinois at Urbana-Champaign, Department of Educational Psychology, Champaign, IL, United States
| | | |
Collapse
|
20
|
Calvitti A, Hochheiser H, Ashfaq S, Bell K, Chen Y, El Kareh R, Gabuzda MT, Liu L, Mortensen S, Pandey B, Rick S, Street RL, Weibel N, Weir C, Agha Z. Physician activity during outpatient visits and subjective workload. J Biomed Inform 2017; 69:135-149. [PMID: 28323114 DOI: 10.1016/j.jbi.2017.03.011] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2016] [Revised: 03/14/2017] [Accepted: 03/16/2017] [Indexed: 12/25/2022]
Abstract
We describe methods for capturing and analyzing EHR use and clinical workflow of physicians during outpatient encounters and relating activity to physicians' self-reported workload. We collected temporally-resolved activity data including audio, video, EHR activity, and eye-gaze along with post-visit assessments of workload. These data are then analyzed through a combination of manual content analysis and computational techniques to temporally align streams, providing a range of process measures of EHR usage, clinical workflow, and physician-patient communication. Data was collected from primary care and specialty clinics at the Veterans Administration San Diego Healthcare System and UCSD Health, who use Electronic Health Record (EHR) platforms, CPRS and Epic, respectively. Grouping visit activity by physician, site, specialty, and patient status enables rank-ordering activity factors by their correlation to physicians' subjective work-load as captured by NASA Task Load Index survey. We developed a coding scheme that enabled us to compare timing studies between CPRS and Epic and extract patient and visit complexity profiles. We identified similar patterns of EHR use and navigation at the 2 sites despite differences in functions, user interfaces and consequent coded representations. Both sites displayed similar proportions of EHR function use and navigation, and distribution of visit length, proportion of time physicians attended to EHRs (gaze), and subjective work-load as measured by the task load survey. We found that visit activity was highly variable across individual physicians, and the observed activity metrics ranged widely as correlates to subjective workload. We discuss implications of our study for methodology, clinical workflow and EHR redesign.
Collapse
Affiliation(s)
- Alan Calvitti
- Veterans Medical Research Foundation, 3350 La Jolla Village Dr, San Diego, CA 92161, United States.
| | - Harry Hochheiser
- Department of Biomedical Informatics and Intelligent Systems Program, University of Pittsburgh, 5607 Baum Blvd, Pittsburgh, PA 15206, United States
| | - Shazia Ashfaq
- Veterans Medical Research Foundation, 3350 La Jolla Village Dr, San Diego, CA 92161, United States
| | - Kristin Bell
- VA San Diego Healthcare System, 3350 La Jolla Village Dr, San Diego, CA 92161, United States
| | - Yunan Chen
- Department of Medical Informatics UC Irvine, Irvine, CA 92697, United States
| | - Robert El Kareh
- Department of Hospital Medicine, University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093, United States
| | - Mark T Gabuzda
- VA San Diego Healthcare System, 3350 La Jolla Village Dr, San Diego, CA 92161, United States
| | - Lin Liu
- Department of Family Medicine and Public Health, University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093, United States
| | - Sara Mortensen
- Veterans Medical Research Foundation, 3350 La Jolla Village Dr, San Diego, CA 92161, United States
| | - Braj Pandey
- VA San Diego Healthcare System, 3350 La Jolla Village Dr, San Diego, CA 92161, United States
| | - Steven Rick
- Department of Computer Science and Engineering, University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093, United States
| | - Richard L Street
- Department of Communication, Texas A&M University, 400 Bizzell St, College Station, TX 77840, United States
| | - Nadir Weibel
- Department of Computer Science and Engineering, University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093, United States
| | - Charlene Weir
- Department of Biomedical Informatics, University of Utah, SLC, VA, Salt Lake City, UT, United States
| | - Zia Agha
- Veterans Medical Research Foundation, 3350 La Jolla Village Dr, San Diego, CA 92161, United States; Department of Medicine, University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093, United States; West Health Institute, 10350 N. Torrey Pines Road, La Jolla, CA 92037, United States
| |
Collapse
|
21
|
Hettinger AZ, Roth EM, Bisantz AM. Cognitive engineering and health informatics: Applications and intersections. J Biomed Inform 2017; 67:21-33. [PMID: 28126605 DOI: 10.1016/j.jbi.2017.01.010] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2016] [Revised: 01/13/2017] [Accepted: 01/17/2017] [Indexed: 10/20/2022]
Abstract
Cognitive engineering is an applied field with roots in both cognitive science and engineering that has been used to support design of information displays, decision support, human-automation interaction, and training in numerous high risk domains ranging from nuclear power plant control to transportation and defense systems. Cognitive engineering provides a set of structured, analytic methods for data collection and analysis that intersect with and complement methods of Cognitive Informatics. These methods support discovery of aspects of the work that make performance challenging, as well as the knowledge, skills, and strategies that experts use to meet those challenges. Importantly, cognitive engineering methods provide novel representations that highlight the inherent complexities of the work domain and traceable links between the results of cognitive analyses and actionable design requirements. This article provides an overview of relevant cognitive engineering methods, and illustrates how they have been applied to the design of health information technology (HIT) systems. Additionally, although cognitive engineering methods have been applied in the design of user-centered informatics systems, methods drawn from informatics are not typically incorporated into a cognitive engineering analysis. This article presents a discussion regarding ways in which data-rich methods can inform cognitive engineering.
Collapse
Affiliation(s)
- A Zachary Hettinger
- Department of Emergency Medicine, Georgetown University School of Medicine, Washington, DC, United States; National Center for Human Factors in Healthcare, MedStar Health, Washington, DC, United States.
| | - Emilie M Roth
- Roth Cognitive Engineering, Stanford, CA, United States
| | - Ann M Bisantz
- Department of Industrial and Systems Engineering, University at Buffalo, State University of New York, Buffalo, NY, United States
| |
Collapse
|
22
|
Luna DR, Rizzato Lede DA, Otero CM, Risk MR, González Bernaldo de Quirós F. User-centered design improves the usability of drug-drug interaction alerts: Experimental comparison of interfaces. J Biomed Inform 2017; 66:204-213. [PMID: 28108211 DOI: 10.1016/j.jbi.2017.01.009] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2016] [Revised: 01/04/2017] [Accepted: 01/15/2017] [Indexed: 01/16/2023]
Abstract
Clinical Decision Support Systems can alert health professionals about drug interactions when they prescribe medications. The Hospital Italiano de Buenos Aires in Argentina developed an electronic health record with drug-drug interaction alerts, using traditional software engineering techniques and requirements. Despite enhancing the drug-drug interaction knowledge database, the alert override rate of this system was very high. We redesigned the alert system using user-centered design (UCD) and participatory design techniques to enhance the drug-drug interaction alert interface. This paper describes the methodology of our UCD. We used crossover method with realistic, clinical vignettes to compare usability of the standard and new software versions in terms of efficiency, effectiveness, and user satisfaction. Our study showed that, compared to the traditional alert system, the UCD alert system was more efficient (alerts faster resolution), more effective (tasks completed with fewer errors), and more satisfying. These results indicate that UCD techniques that follow ISO 9241-210 can generate more usable alerts than traditional design.
Collapse
Affiliation(s)
- Daniel R Luna
- Health Informatics Department, Hospital Italiano de Buenos Aires, Argentina; Instituto Tecnológico de Buenos Aires (ITBA), Argentina.
| | | | - Carlos M Otero
- Health Informatics Department, Hospital Italiano de Buenos Aires, Argentina
| | - Marcelo R Risk
- Instituto Tecnológico de Buenos Aires (ITBA), Argentina; Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Argentina
| | | |
Collapse
|
23
|
Brunyé TT, Mercan E, Weaver DL, Elmore JG. Accuracy is in the eyes of the pathologist: The visual interpretive process and diagnostic accuracy with digital whole slide images. J Biomed Inform 2017; 66:171-179. [PMID: 28087402 DOI: 10.1016/j.jbi.2017.01.004] [Citation(s) in RCA: 38] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2016] [Revised: 01/06/2017] [Accepted: 01/09/2017] [Indexed: 12/30/2022]
Abstract
Digital whole slide imaging is an increasingly common medium in pathology, with application to education, telemedicine, and rendering second opinions. It has also made it possible to use eye tracking devices to explore the dynamic visual inspection and interpretation of histopathological features of tissue while pathologists review cases. Using whole slide images, the present study examined how a pathologist's diagnosis is influenced by fixed case-level factors, their prior clinical experience, and their patterns of visual inspection. Participating pathologists interpreted one of two test sets, each containing 12 digital whole slide images of breast biopsy specimens. Cases represented four diagnostic categories as determined via expert consensus: benign without atypia, atypia, ductal carcinoma in situ (DCIS), and invasive cancer. Each case included one or more regions of interest (ROIs) previously determined as of critical diagnostic importance. During pathologist interpretation we tracked eye movements, viewer tool behavior (zooming, panning), and interpretation time. Models were built using logistic and linear regression with generalized estimating equations, testing whether variables at the level of the pathologists, cases, and visual interpretive behavior would independently and/or interactively predict diagnostic accuracy and efficiency. Diagnostic accuracy varied as a function of case consensus diagnosis, replicating earlier research. As would be expected, benign cases tended to elicit false positives, and atypia, DCIS, and invasive cases tended to elicit false negatives. Pathologist experience levels, case consensus diagnosis, case difficulty, eye fixation durations, and the extent to which pathologists' eyes fixated within versus outside of diagnostic ROIs, all independently or interactively predicted diagnostic accuracy. Higher zooming behavior predicted a tendency to over-interpret benign and atypia cases, but not DCIS cases. Efficiency was not predicted by pathologist- or visual search-level variables. Results provide new insights into the medical interpretive process and demonstrate the complex interactions between pathologists and cases that guide diagnostic decision-making. Implications for training, clinical practice, and computer-aided decision aids are considered.
Collapse
Affiliation(s)
- Tad T Brunyé
- Center for Applied Brain & Cognitive Sciences, Tufts University, Medford, MA, United States.
| | - Ezgi Mercan
- Department of Computer Science and Engineering, University of Washington, Seattle, WA, United States
| | - Donald L Weaver
- Department of Pathology and UVM Cancer Center, University of Vermont, Burlington, VT, United States
| | - Joann G Elmore
- Department of Medicine, University of Washington School of Medicine, Seattle, WA, United States
| |
Collapse
|
24
|
Aselmaa A, van Herk M, Laprie A, Nestle U, Götz I, Wiedenmann N, Schimek-Jasch T, Picaud F, Syrykh C, Cagetti LV, Jolnerovski M, Song Y, Goossens RH. Using a contextualized sensemaking model for interaction design: A case study of tumor contouring. J Biomed Inform 2017; 65:145-158. [DOI: 10.1016/j.jbi.2016.12.001] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2016] [Revised: 11/02/2016] [Accepted: 12/04/2016] [Indexed: 12/28/2022]
|
25
|
Russ AL, Melton BL, Daggy JK, Saleem JJ. Pilot evaluation of a method to assess prescribers' information processing of medication alerts. J Biomed Inform 2016; 66:11-18. [PMID: 27908833 DOI: 10.1016/j.jbi.2016.11.011] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2016] [Revised: 10/12/2016] [Accepted: 11/27/2016] [Indexed: 12/01/2022]
Abstract
BACKGROUND Prescribers commonly receive alerts during medication ordering. Prescribers work in a complex, time-pressured environment; to enhance the effectiveness of safety alerts, the effort needed to cognitively process these alerts should be minimized. Methods to evaluate the extent to which computerized alerts support prescribers' information processing are lacking. OBJECTIVE To develop a methodological protocol to assess the extent to which alerts support prescribers' information processing at-a-glance; specifically, the incorporation of information into their working memory. We hypothesized that the method would be feasible and that we would be able to detect a significant difference in prescribers' information processing with a revised alert display that incorporates warning design guidelines compared to the original alert display. METHODS A counterbalanced, within-subject study was conducted with 20 prescribers in a human-computer interaction laboratory. We tested a single alert that was displayed in two different ways. Prescribers were informed that an alert would appear for 10s. After the alert was shown, a white screen was displayed, and prescribers were asked to verbally describe what they saw; indicate how many total warnings; and describe anything else they remembered about the alert. We measured information processing via the accuracy of prescribers' free recall and their ability to identify that three warning messages were present. Two analysts independently evaluated participants' responses against a comprehensive catalog of alert elements and then discussed discrepancies until reaching consensus. RESULTS This feasibility study demonstrated that the method seemed to be effective for evaluating prescribers' information processing of medication alert displays. With this method, we were able to detect significant differences in prescribers' recall of alert information. The proportion of total data elements that prescribers were able to accurately recall was significantly greater for the revised versus original alert display (p=0.006). With the revised display, more prescribers accurately reported that three warnings were shown (p=0.002). CONCLUSIONS The methodological protocol was feasible for evaluating the alert display and yielded important findings on prescribers' information processing. Study methods supplement traditional usability evaluation methods and may be useful for evaluating information processing of other healthcare technologies.
Collapse
Affiliation(s)
- Alissa L Russ
- Center for Health Information and Communication, Department of Veterans Affairs (VA), Veterans Health Administration, Health Services Research and Development Service, Richard L. Roudebush VA Medical Center, Indianapolis, IN, United States; Regenstrief Institute, Inc., Indianapolis, IN, United States; College of Pharmacy, Purdue University, West Lafayette, IN, United States.
| | - Brittany L Melton
- School of Pharmacy, University of Kansas, Lawrence, KS, United States
| | - Joanne K Daggy
- Department of Biostatistics, Indiana University School of Medicine, Indianapolis, IN, United States
| | - Jason J Saleem
- Department of Industrial Engineering, University of Louisville, Louisville, KY, United States
| |
Collapse
|
26
|
Kannampallil TG, Abraham J, Patel VL. Methodological framework for evaluating clinical processes: A cognitive informatics perspective. J Biomed Inform 2016; 64:342-351. [DOI: 10.1016/j.jbi.2016.11.002] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2016] [Revised: 11/10/2016] [Accepted: 11/11/2016] [Indexed: 01/10/2023]
|
27
|
Abstract
OBJECTIVE This paper presents the development of medical informatics education during the years from the establishment of the International Medical Informatics Association (IMIA) until today. METHOD A search in the literature was performed using search engines and appropriate keywords as well as a manual selection of papers. The search covered English language papers and was limited to search on papers title and abstract only. RESULTS The aggregated papers were analyzed on the basis of the subject area, origin, time span, and curriculum development, and conclusions were drawn. CONCLUSIONS From the results, it is evident that IMIA has played a major role in comparing and integrating the Biomedical and Health Informatics educational efforts across the different levels of education and the regional distribution of educators and institutions. A large selection of references is presented facilitating future work on the field of education in biomedical and health informatics.
Collapse
Affiliation(s)
- J Mantas
- John Mantas, Health Informatics Laboratory, Department of Nursing, School of Health Sciences, National and Kapodistrian University of Athens, Greece, E-mail:
| |
Collapse
|
28
|
Sacchi L, Holmes JH. Progress in Biomedical Knowledge Discovery: A 25-year Retrospective. Yearb Med Inform 2016; Suppl 1:S117-29. [PMID: 27488403 PMCID: PMC5171499 DOI: 10.15265/iys-2016-s033] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022] Open
Abstract
OBJECTIVES We sought to explore, via a systematic review of the literature, the state of the art of knowledge discovery in biomedical databases as it existed in 1992, and then now, 25 years later, mainly focused on supervised learning. METHODS We performed a rigorous systematic search of PubMed and latent Dirichlet allocation to identify themes in the literature and trends in the science of knowledge discovery in and between time periods and compare these trends. We restricted the result set using a bracket of five years previous, such that the 1992 result set was restricted to articles published between 1987 and 1992, and the 2015 set between 2011 and 2015. This was to reflect the current literature available at the time to researchers and others at the target dates of 1992 and 2015. The search term was framed as: Knowledge Discovery OR Data Mining OR Pattern Discovery OR Pattern Recognition, Automated. RESULTS A total 538 and 18,172 documents were retrieved for 1992 and 2015, respectively. The number and type of data sources increased dramatically over the observation period, primarily due to the advent of electronic clinical systems. The period 1992- 2015 saw the emergence of new areas of research in knowledge discovery, and the refinement and application of machine learning approaches that were nascent or unknown in 1992. CONCLUSIONS Over the 25 years of the observation period, we identified numerous developments that impacted the science of knowledge discovery, including the availability of new forms of data, new machine learning algorithms, and new application domains. Through a bibliometric analysis we examine the striking changes in the availability of highly heterogeneous data resources, the evolution of new algorithmic approaches to knowledge discovery, and we consider from legal, social, and political perspectives possible explanations of the growth of the field. Finally, we reflect on the achievements of the past 25 years to consider what the next 25 years will bring with regard to the availability of even more complex data and to the methods that could be, and are being now developed for the discovery of new knowledge in biomedical data.
Collapse
Affiliation(s)
| | - J H Holmes
- John H Holmes, Institute for Biomedical Informatics, University of Pennsylvania School of Medicine, 717 Blockley Hall, 423 Guardian Drive, Philadelphia, PA 19104, USA, Tel: 215-898-4833, Fax: 215-573-5325, E-Mail:
| |
Collapse
|
29
|
Cognitive informatics methods for interactive clinical systems. J Biomed Inform 2016. [DOI: 10.1016/j.jbi.2016.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|