1
|
Nagendran M, Festor P, Komorowski M, Gordon AC, Faisal AA. Eye tracking insights into physician behaviour with safe and unsafe explainable AI recommendations. NPJ Digit Med 2024; 7:202. [PMID: 39095449 PMCID: PMC11297294 DOI: 10.1038/s41746-024-01200-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 07/19/2024] [Indexed: 08/04/2024] Open
Abstract
We studied clinical AI-supported decision-making as an example of a high-stakes setting in which explainable AI (XAI) has been proposed as useful (by theoretically providing physicians with context for the AI suggestion and thereby helping them to reject unsafe AI recommendations). Here, we used objective neurobehavioural measures (eye-tracking) to see how physicians respond to XAI with N = 19 ICU physicians in a hospital's clinical simulation suite. Prescription decisions were made both pre- and post-reveal of either a safe or unsafe AI recommendation and four different types of simultaneously presented XAI. We used overt visual attention as a marker for where physician mental attention was directed during the simulations. Unsafe AI recommendations attracted significantly greater attention than safe AI recommendations. However, there was no appreciably higher level of attention placed onto any of the four types of explanation during unsafe AI scenarios (i.e. XAI did not appear to 'rescue' decision-makers). Furthermore, self-reported usefulness of explanations by physicians did not correlate with the level of attention they devoted to the explanations reinforcing the notion that using self-reports alone to evaluate XAI tools misses key aspects of the interaction behaviour between human and machine.
Collapse
Affiliation(s)
- Myura Nagendran
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK
- Division of Anaesthetics, Pain Medicine, and Intensive Care, Imperial College London, London, UK
- Brain and Behaviour Lab, Imperial College London, London, UK
| | - Paul Festor
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK
- Brain and Behaviour Lab, Imperial College London, London, UK
- Department of Computing, Imperial College London, London, UK
| | - Matthieu Komorowski
- Division of Anaesthetics, Pain Medicine, and Intensive Care, Imperial College London, London, UK
| | - Anthony C Gordon
- Division of Anaesthetics, Pain Medicine, and Intensive Care, Imperial College London, London, UK
| | - Aldo A Faisal
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK.
- Brain and Behaviour Lab, Imperial College London, London, UK.
- Department of Computing, Imperial College London, London, UK.
- Institute of Artificial & Human Intelligence, University of Bayreuth, Bayreuth, Germany.
| |
Collapse
|
2
|
Rabindranath M, Naghibzadeh M, Zhao X, Holdsworth S, Brudno M, Sidhu A, Bhat M. Clinical Deployment of Machine Learning Tools in Transplant Medicine: What Does the Future Hold? Transplantation 2024; 108:1700-1708. [PMID: 39042768 DOI: 10.1097/tp.0000000000004876] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/08/2023]
Abstract
Medical applications of machine learning (ML) have shown promise in analyzing patient data to support clinical decision-making and provide patient-specific outcomes. In transplantation, several applications of ML exist which include pretransplant: patient prioritization, donor-recipient matching, organ allocation, and posttransplant outcomes. Numerous studies have shown the development and utility of ML models, which have the potential to augment transplant medicine. Despite increasing efforts to develop robust ML models for clinical use, very few of these tools are deployed in the healthcare setting. Here, we summarize the current applications of ML in transplant and discuss a potential clinical deployment framework using examples in organ transplantation. We identified that creating an interdisciplinary team, curating a reliable dataset, addressing the barriers to implementation, and understanding current clinical evaluation models could help in deploying ML models into the transplant clinic setting.
Collapse
Affiliation(s)
- Madhumitha Rabindranath
- Transplant AI Initiative, Ajmera Transplant Program, University Health Network, Toronto, ON, Canada
- Institute of Medical Science, University of Toronto, Toronto, ON, Canada
- Toronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada
| | - Maryam Naghibzadeh
- Transplant AI Initiative, Ajmera Transplant Program, University Health Network, Toronto, ON, Canada
| | - Xun Zhao
- Transplant AI Initiative, Ajmera Transplant Program, University Health Network, Toronto, ON, Canada
| | - Sandra Holdsworth
- Transplant AI Initiative, Ajmera Transplant Program, University Health Network, Toronto, ON, Canada
| | - Michael Brudno
- Transplant AI Initiative, Ajmera Transplant Program, University Health Network, Toronto, ON, Canada
- Department of Computer Science, University of Toronto, Toronto, ON, Canada
| | - Aman Sidhu
- Transplant AI Initiative, Ajmera Transplant Program, University Health Network, Toronto, ON, Canada
- Department of Medicine, University of Toronto, Toronto, ON, Canada
| | - Mamatha Bhat
- Transplant AI Initiative, Ajmera Transplant Program, University Health Network, Toronto, ON, Canada
- Institute of Medical Science, University of Toronto, Toronto, ON, Canada
- Toronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada
- Department of Medicine, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
3
|
Soukup T, Dean Franklin B. Quality, safety and artificial intelligence. BMJ Qual Saf 2024; 33:406-410. [PMID: 38760073 DOI: 10.1136/bmjqs-2024-017382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Accepted: 04/01/2024] [Indexed: 05/19/2024]
Affiliation(s)
- Tayana Soukup
- Department of Surgery and Cancer, Faculty of Medicine, Imperial College London, London, UK
| | - Bryony Dean Franklin
- Imperial College Healthcare NHS Trust, London, UK
- UCL School of Pharmacy, London, UK
| |
Collapse
|
4
|
Catling FJR, Nagendran M, Festor P, Bien Z, Harris S, Faisal AA, Gordon AC, Komorowski M. Can Machine Learning Personalize Cardiovascular Therapy in Sepsis? Crit Care Explor 2024; 6:e1087. [PMID: 38709088 DOI: 10.1097/cce.0000000000001087] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/07/2024] Open
Abstract
Large randomized trials in sepsis have generally failed to find effective novel treatments. This is increasingly attributed to patient heterogeneity, including heterogeneous cardiovascular changes in septic shock. We discuss the potential for machine learning systems to personalize cardiovascular resuscitation in sepsis. While the literature is replete with proofs of concept, the technological readiness of current systems is low, with a paucity of clinical trials and proven patient benefit. Systems may be vulnerable to confounding and poor generalization to new patient populations or contemporary patterns of care. Typical electronic health records do not capture rich enough data, at sufficient temporal resolution, to produce systems that make actionable treatment suggestions. To resolve these issues, we recommend a simultaneous focus on technical challenges and removing barriers to translation. This will involve improving data quality, adopting causally grounded models, prioritizing safety assessment and integration into healthcare workflows, conducting randomized clinical trials and aligning with regulatory requirements.
Collapse
Affiliation(s)
- Finneas J R Catling
- Institute of Healthcare Engineering, University College London, London, United Kingdom
- Division of Anaesthetics, Pain Medicine, and Intensive Care, Imperial College London, London, United Kingdom
| | - Myura Nagendran
- Division of Anaesthetics, Pain Medicine, and Intensive Care, Imperial College London, London, United Kingdom
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, United Kingdom
| | - Paul Festor
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, United Kingdom
- Department of Computing, Imperial College London, London, United Kingdom
| | - Zuzanna Bien
- School of Life Course & Population Sciences, King's College London, United Kingdom
| | - Steve Harris
- Department of Critical Care, University College London Hospital, London, United Kingdom
- Institute of Health Informatics, University College London, London, United Kingdom
| | - A Aldo Faisal
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, United Kingdom
- Department of Computing, Imperial College London, London, United Kingdom
- Institute of Artificial and Human Intelligence, Universität Bayreuth, Bayreuth, Germany
- Department of Bioengineering, Imperial College London, London, United Kingdom
| | - Anthony C Gordon
- Division of Anaesthetics, Pain Medicine, and Intensive Care, Imperial College London, London, United Kingdom
| | - Matthieu Komorowski
- Division of Anaesthetics, Pain Medicine, and Intensive Care, Imperial College London, London, United Kingdom
| |
Collapse
|
5
|
Ferrara M, Bertozzi G, Di Fazio N, Aquila I, Di Fazio A, Maiese A, Volonnino G, Frati P, La Russa R. Risk Management and Patient Safety in the Artificial Intelligence Era: A Systematic Review. Healthcare (Basel) 2024; 12:549. [PMID: 38470660 PMCID: PMC10931321 DOI: 10.3390/healthcare12050549] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Revised: 02/19/2024] [Accepted: 02/23/2024] [Indexed: 03/14/2024] Open
Abstract
BACKGROUND Healthcare systems represent complex organizations within which multiple factors (physical environment, human factor, technological devices, quality of care) interconnect to form a dense network whose imbalance is potentially able to compromise patient safety. In this scenario, the need for hospitals to expand reactive and proactive clinical risk management programs is easily understood, and artificial intelligence fits well in this context. This systematic review aims to investigate the state of the art regarding the impact of AI on clinical risk management processes. To simplify the analysis of the review outcomes and to motivate future standardized comparisons with any subsequent studies, the findings of the present review will be grouped according to the possibility of applying AI in the prevention of the different incident type groups as defined by the ICPS. MATERIALS AND METHODS On 3 November 2023, a systematic review of the literature according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was carried out using the SCOPUS and Medline (via PubMed) databases. A total of 297 articles were identified. After the selection process, 36 articles were included in the present systematic review. RESULTS AND DISCUSSION The studies included in this review allowed for the identification of three main "incident type" domains: clinical process, healthcare-associated infection, and medication. Another relevant application of AI in clinical risk management concerns the topic of incident reporting. CONCLUSIONS This review highlighted that AI can be applied transversely in various clinical contexts to enhance patient safety and facilitate the identification of errors. It appears to be a promising tool to improve clinical risk management, although its use requires human supervision and cannot completely replace human skills. To facilitate the analysis of the present review outcome and to enable comparison with future systematic reviews, it was deemed useful to refer to a pre-existing taxonomy for the identification of adverse events. However, the results of the present study highlighted the usefulness of AI not only for risk prevention in clinical practice, but also in improving the use of an essential risk identification tool, which is incident reporting. For this reason, the taxonomy of the areas of application of AI to clinical risk processes should include an additional class relating to risk identification and analysis tools. For this purpose, it was considered convenient to use ICPS classification.
Collapse
Affiliation(s)
- Michela Ferrara
- Department of Anatomical, Histological, Forensic and Orthopaedic Sciences, Sapienza University of Rome, 00161 Rome, Italy; (M.F.); (N.D.F.); (P.F.)
| | - Giuseppe Bertozzi
- Complex Intercompany Structure of Forensic Medicine, 85100 Potenza, Italy;
| | - Nicola Di Fazio
- Department of Anatomical, Histological, Forensic and Orthopaedic Sciences, Sapienza University of Rome, 00161 Rome, Italy; (M.F.); (N.D.F.); (P.F.)
| | - Isabella Aquila
- Department of Medical and Surgical Sciences, University Magna Graecia of Catanzaro, 88100 Catanzaro, Italy;
| | - Aldo Di Fazio
- Regional Hospital “San Carlo”, 85100 Potenza, Italy;
| | - Aniello Maiese
- Department of Surgical Pathology, Medical, Molecular and Critical Area, Institute of Legal Medicine, University of Pisa, 56126 Pisa, Italy;
| | - Gianpietro Volonnino
- Department of Anatomical, Histological, Forensic and Orthopaedic Sciences, Sapienza University of Rome, 00161 Rome, Italy; (M.F.); (N.D.F.); (P.F.)
| | - Paola Frati
- Department of Anatomical, Histological, Forensic and Orthopaedic Sciences, Sapienza University of Rome, 00161 Rome, Italy; (M.F.); (N.D.F.); (P.F.)
| | - Raffaele La Russa
- Department of Clinical Medicine, Public Health, Life and Environment Science, University of L’Aquila, 67100 L’Aquila, Italy;
| |
Collapse
|
6
|
Otten M, Jagesar AR, Dam TA, Biesheuvel LA, den Hengst F, Ziesemer KA, Thoral PJ, de Grooth HJ, Girbes ARJ, François-Lavet V, Hoogendoorn M, Elbers PWG. Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness Assessment. Crit Care Med 2024; 52:e79-e88. [PMID: 37938042 DOI: 10.1097/ccm.0000000000006100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2023]
Abstract
OBJECTIVE Reinforcement learning (RL) is a machine learning technique uniquely effective at sequential decision-making, which makes it potentially relevant to ICU treatment challenges. We set out to systematically review, assess level-of-readiness and meta-analyze the effect of RL on outcomes for critically ill patients. DATA SOURCES A systematic search was performed in PubMed, Embase.com, Clarivate Analytics/Web of Science Core Collection, Elsevier/SCOPUS and the Institute of Electrical and Electronics Engineers Xplore Digital Library from inception to March 25, 2022, with subsequent citation tracking. DATA EXTRACTION Journal articles that used an RL technique in an ICU population and reported on patient health-related outcomes were included for full analysis. Conference papers were included for level-of-readiness assessment only. Descriptive statistics, characteristics of the models, outcome compared with clinician's policy and level-of-readiness were collected. RL-health risk of bias and applicability assessment was performed. DATA SYNTHESIS A total of 1,033 articles were screened, of which 18 journal articles and 18 conference papers, were included. Thirty of those were prototyping or modeling articles and six were validation articles. All articles reported RL algorithms to outperform clinical decision-making by ICU professionals, but only in retrospective data. The modeling techniques for the state-space, action-space, reward function, RL model training, and evaluation varied widely. The risk of bias was high in all articles, mainly due to the evaluation procedure. CONCLUSION In this first systematic review on the application of RL in intensive care medicine we found no studies that demonstrated improved patient outcomes from RL-based technologies. All studies reported that RL-agent policies outperformed clinician policies, but such assessments were all based on retrospective off-policy evaluation.
Collapse
Affiliation(s)
- Martijn Otten
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Ameet R Jagesar
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Tariq A Dam
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Laurens A Biesheuvel
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Floris den Hengst
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | | | - Patrick J Thoral
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Harm-Jan de Grooth
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Armand R J Girbes
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Vincent François-Lavet
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Mark Hoogendoorn
- Quantitative Data Analytics Group, Department of Computer Science, Faculty of Science, Vrije Universiteit, Amsterdam, The Netherlands
| | - Paul W G Elbers
- Department of Intensive Care Medicine, Center for Critical Care Computational Intelligence, Amsterdam Medical Data Science (AMDS), Amsterdam Cardiovascular Science (ACS), Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| |
Collapse
|
7
|
Wieneke H, Voigt I. Principles of artificial intelligence and its application in cardiovascular medicine. Clin Cardiol 2024; 47:e24148. [PMID: 37721424 PMCID: PMC10766001 DOI: 10.1002/clc.24148] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/24/2023] [Revised: 08/23/2023] [Accepted: 08/29/2023] [Indexed: 09/19/2023] Open
Abstract
Artificial intelligence (AI) represents a rapidly developing field. Its use can improve diagnosis and therapy in many areas of medicine. Despite this enormous progress, many physicians perceive it as a black box and are skeptical about it. This review will present the basics of machine learning. Different classifications of artificial intelligence, such as supervised versus unsupervised and discriminative versus generative AI, are given. Analogies to human intelligence are discussed as far as algorithms are oriented toward it. In the second step, the most common models like random forest, k-means clustering, convolutional neural network, and transformers will be presented in a way that the underlying idea can be understood. Corresponding medical applications in cardiovascular medicine will be named for all models, respectively. The overview is intended to show that the term artificial intelligence covers a wide range of different concepts. It should help physicians understand the principles of AI to make up one's minds about its application in cardiology. It should also enable them to evaluate results obtained with AI's help critically.
Collapse
Affiliation(s)
- Heinrich Wieneke
- Department of Cardiology and Angiology, Contilia Heart and Vascular CenterElisabeth‐Krankenhaus EssenEssenGermany
| | - Ingo Voigt
- Department of Cardiology and Angiology, Contilia Heart and Vascular CenterElisabeth‐Krankenhaus EssenEssenGermany
| |
Collapse
|
8
|
Schefzik R, Hahn B, Schneider-Lindner V. Dissecting contributions of individual systemic inflammatory response syndrome criteria from a prospective algorithm to the prediction and diagnosis of sepsis in a polytrauma cohort. Front Med (Lausanne) 2023; 10:1227031. [PMID: 37583420 PMCID: PMC10424878 DOI: 10.3389/fmed.2023.1227031] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Accepted: 07/17/2023] [Indexed: 08/17/2023] Open
Abstract
Background Sepsis is the leading cause of death in intensive care units (ICUs), and its timely detection and treatment improve clinical outcome and survival. Systemic inflammatory response syndrome (SIRS) refers to the concurrent fulfillment of at least two out of the following four clinical criteria: tachycardia, tachypnea, abnormal body temperature, and abnormal leukocyte count. While SIRS was controversially abandoned from the current sepsis definition, a dynamic SIRS representation still has potential for sepsis prediction and diagnosis. Objective We retrospectively elucidate the individual contributions of the SIRS criteria in a polytrauma cohort from the post-surgical ICU of University Medical Center Mannheim (Germany). Methods We used a dynamic and prospective SIRS algorithm tailored to the ICU setting by accounting for catecholamine therapy and mechanical ventilation. Two clinically relevant tasks are considered: (i) sepsis prediction using the first 24 h after admission to our ICU, and (ii) sepsis diagnosis using the last 24 h before sepsis onset and a time point of comparable ICU treatment duration for controls, respectively. We determine the importance of individual SIRS criteria by systematically varying criteria weights when summarizing the SIRS algorithm output with SIRS descriptors and assessing the classification performance of the resulting logistic regression models using a specifically developed ranking score. Results Our models perform better for the diagnosis than the prediction task (maximum AUROC 0.816 vs. 0.693). Risk models containing only the SIRS level average mostly show reasonable performance across criteria weights, with prediction and diagnosis AUROCs ranging from 0.455 (weight on leukocyte criterion only) to 0.693 and 0.619 to 0.800, respectively. For sepsis prediction, temperature and tachypnea are the most important SIRS criteria, whereas the leukocytes criterion is least important and potentially even counterproductive. For sepsis diagnosis, all SIRS criteria are relevant, with the temperature criterion being most influential. Conclusion SIRS is relevant for sepsis prediction and diagnosis in polytrauma, and no criterion should a priori be omitted. Hence, the original expert-defined SIRS criteria are valid, capturing important sepsis risk determinants. Our prospective SIRS algorithm provides dynamic determination of SIRS criteria and descriptors, allowing their integration in sepsis risk models also in other settings.
Collapse
Affiliation(s)
- Roman Schefzik
- Department of Anesthesiology and Surgical Intensive Care Medicine, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | | | | |
Collapse
|