1
|
Miller M, McCann L, Lewis L, Miaskowski C, Ream E, Darley A, Harris J, Kotronoulas G, V Berg G, Lubowitzki S, Armes J, Patiraki E, Furlong E, Fox P, Gaiger A, Cardone A, Orr D, Flowerday A, Katsaragakis S, Skene S, Moore M, McCrone P, De Souza N, Donnan PT, Maguire R. Patients' and Clinicians' Perceptions of the Clinical Utility of Predictive Risk Models for Chemotherapy-Related Symptom Management: Qualitative Exploration Using Focus Groups and Interviews. J Med Internet Res 2024; 26:e49309. [PMID: 38901021 DOI: 10.2196/49309] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 11/22/2023] [Accepted: 03/06/2024] [Indexed: 06/22/2024] Open
Abstract
BACKGROUND Interest in the application of predictive risk models (PRMs) in health care to identify people most likely to experience disease and treatment-related complications is increasing. In cancer care, these techniques are focused primarily on the prediction of survival or life-threatening toxicities (eg, febrile neutropenia). Fewer studies focus on the use of PRMs for symptoms or supportive care needs. The application of PRMs to chemotherapy-related symptoms (CRS) would enable earlier identification and initiation of prompt, personalized, and tailored interventions. While some PRMs exist for CRS, few were translated into clinical practice, and human factors associated with their use were not reported. OBJECTIVE We aim to explore patients' and clinicians' perspectives of the utility and real-world application of PRMs to improve the management of CRS. METHODS Focus groups (N=10) and interviews (N=5) were conducted with patients (N=28) and clinicians (N=26) across 5 European countries. Interactions were audio-recorded, transcribed verbatim, and analyzed thematically. RESULTS Both clinicians and patients recognized the value of having individualized risk predictions for CRS and appreciated how this type of information would facilitate the provision of tailored preventative treatments or supportive care interactions. However, cautious and skeptical attitudes toward the use of PRMs in clinical care were noted by both groups, particularly in relationship to the uncertainty regarding how the information would be generated. Visualization and presentation of PRM information in a usable and useful format for both patients and clinicians was identified as a challenge to their successful implementation in clinical care. CONCLUSIONS Findings from this study provide information on clinicians' and patients' perspectives on the clinical use of PRMs for the management of CRS. These international perspectives are important because they provide insight into the risks and benefits of using PRMs to evaluate CRS. In addition, they highlight the need to find ways to more effectively present and use this information in clinical practice. Further research that explores the best ways to incorporate this type of information while maintaining the human side of care is warranted. TRIAL REGISTRATION ClinicalTrials.gov NCT02356081; https://clinicaltrials.gov/study/NCT02356081.
Collapse
Affiliation(s)
- Morven Miller
- Computer & Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| | - Lisa McCann
- Computer & Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| | - Liane Lewis
- Johnson and Johnson Medical, Norderstedt, Germany
| | | | - Emma Ream
- School of Health Sciences, University of Surrey, Guildford, United Kingdom
| | - Andrew Darley
- School of Medicine, University College Dublin, Dublin, Ireland
| | - Jenny Harris
- School of Health Sciences, University of Surrey, Guildford, United Kingdom
| | - Grigorios Kotronoulas
- School of Medicine, Dentistry and Nursing, University of Glasgow, Glasgow, United Kingdom
| | - Geir V Berg
- Department of Health Sciences, Norwegian University of Science and Technology, Gjøvik, Norway
| | - Simone Lubowitzki
- Department of Internal Medicine 1, Division of Hematology and Hemostaseology, Medical University of Vienna, Vienna, Austria
| | - Jo Armes
- School of Health Sciences, University of Surrey, Guildford, United Kingdom
| | - Elizabeth Patiraki
- School of Health Sciences, National and Kapodistrian University of Athens, Athens, Greece
| | - Eileen Furlong
- School of Nursing, Midwifery and Health Systems, University College Dublin, Dublin, Ireland
| | - Patricia Fox
- School of Nursing, Midwifery and Health Systems, University College Dublin, Dublin, Ireland
| | - Alexander Gaiger
- Department of Internal Medicine 1, Division of Hematology and Hemostaseology, Medical University of Vienna, Vienna, Austria
| | | | | | | | - Stylianos Katsaragakis
- School of Health Sciences, National and Kapodistrian University of Athens, Athens, Greece
| | - Simon Skene
- Surrey Clinical Trials Unit, University of Surrey, Guildford, United Kingdom
| | - Margaret Moore
- Computer & Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| | - Paul McCrone
- Department of Health Services and Population Research, Institute of Psychiatry Psychology and Neuroscience, King's College London, London, United Kingdom
| | - Nicosha De Souza
- Population Health and Genomics, Medical School, University of Dundee, Dundee, United Kingdom
| | - Peter T Donnan
- Population Health and Genomics, Medical School, University of Dundee, Dundee, United Kingdom
| | - Roma Maguire
- Computer & Information Sciences, University of Strathclyde, Glasgow, United Kingdom
| |
Collapse
|
2
|
Chen S, Guevara M, Moningi S, Hoebers F, Elhalawani H, Kann BH, Chipidza FE, Leeman J, Aerts HJWL, Miller T, Savova GK, Gallifant J, Celi LA, Mak RH, Lustberg M, Afshar M, Bitterman DS. The effect of using a large language model to respond to patient messages. Lancet Digit Health 2024; 6:e379-e381. [PMID: 38664108 DOI: 10.1016/s2589-7500(24)00060-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Revised: 03/04/2024] [Accepted: 03/12/2024] [Indexed: 05/26/2024]
Affiliation(s)
- Shan Chen
- Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Marco Guevara
- Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA
| | - Shalini Moningi
- Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA
| | - Frank Hoebers
- Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Department of Radiation Oncology, GROW School for Oncology and Reproduction, Maastricht University, Maastricht, Netherlands
| | - Hesham Elhalawani
- Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA
| | - Benjamin H Kann
- Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA
| | - Fallon E Chipidza
- Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA
| | - Jonathan Leeman
- Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA
| | - Hugo J W L Aerts
- Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Radiology and Nuclear Medicine, GROW and Cardiovascular Research Institute Maastricht, Maastricht University, Maastricht, Netherlands
| | - Timothy Miller
- Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Guergana K Savova
- Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Jack Gallifant
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Leo A Celi
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA; Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA; Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA
| | - Raymond H Mak
- Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA
| | - Maryam Lustberg
- Department of Medical Oncology, Yale School of Medicine, New Haven, CT, USA
| | - Majid Afshar
- Department of Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA
| | - Danielle S Bitterman
- Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Boston, MA 02115, USA; Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA.
| |
Collapse
|
3
|
Weissman GE, Greer JA, Temel JS. Use of Machine Learning to Optimize Referral for Early Palliative Care: Are Prognostic Predictions Enough? J Clin Oncol 2024; 42:1603-1606. [PMID: 38489555 DOI: 10.1200/jco.24.00024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Revised: 01/20/2024] [Accepted: 01/24/2024] [Indexed: 03/17/2024] Open
Affiliation(s)
- Gary E Weissman
- Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA
- Departments of Medicine and Biostatistics, Epidemiology & Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA
| | - Joseph A Greer
- Massachusetts General Hospital, Boston, MA
- Harvard Medical School, Boston, MA
| | - Jennifer S Temel
- Massachusetts General Hospital, Boston, MA
- Harvard Medical School, Boston, MA
| |
Collapse
|
4
|
Montomoli J, Bitondo MM, Cascella M, Rezoagli E, Romeo L, Bellini V, Semeraro F, Gamberini E, Frontoni E, Agnoletti V, Altini M, Benanti P, Bignami EG. Algor-ethics: charting the ethical path for AI in critical care. J Clin Monit Comput 2024:10.1007/s10877-024-01157-y. [PMID: 38573370 DOI: 10.1007/s10877-024-01157-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2023] [Accepted: 03/22/2024] [Indexed: 04/05/2024]
Abstract
The integration of Clinical Decision Support Systems (CDSS) based on artificial intelligence (AI) in healthcare is groundbreaking evolution with enormous potential, but its development and ethical implementation, presents unique challenges, particularly in critical care, where physicians often deal with life-threating conditions requiring rapid actions and patients unable to participate in the decisional process. Moreover, development of AI-based CDSS is complex and should address different sources of bias, including data acquisition, health disparities, domain shifts during clinical use, and cognitive biases in decision-making. In this scenario algor-ethics is mandatory and emphasizes the integration of 'Human-in-the-Loop' and 'Algorithmic Stewardship' principles, and the benefits of advanced data engineering. The establishment of Clinical AI Departments (CAID) is necessary to lead AI innovation in healthcare, ensuring ethical integrity and human-centered development in this rapidly evolving field.
Collapse
Affiliation(s)
- Jonathan Montomoli
- Department of Anesthesia and Intensive Care, Infermi Hospital, Romagna Local Health Authority, Viale Settembrini 2, Rimini, 47923, Italy.
- Health Services Research, Evaluation and Policy Unit, Romagna Local Health Authority, Viale Settembrini 2, Rimini, 47923, Italy.
| | - Maria Maddalena Bitondo
- Department of Anesthesia and Intensive Care, Infermi Hospital, Romagna Local Health Authority, Viale Settembrini 2, Rimini, 47923, Italy
| | - Marco Cascella
- Unit of Anesthesia and Pain Medicine, Department of Medicine, Surgery and Dentistry "Scuola Medica Salernitana, " University of Salerno, Baronissi, Salerno, Italy
| | - Emanuele Rezoagli
- School of Medicine and Surgery, University of Milano-Bicocca, Via Cadore, 48, Monza, 20900, Italy
- Dipartimento di Emergenza e Urgenza, Terapia intensiva e Semintensiva adulti e pediatrica, Fondazione IRCCS San Gerardo dei Tintori, Via Pergolesi, 33, Monza, 20900, Italy
| | - Luca Romeo
- Department of Economics and Law, University of Macerata, Macerata, 62100, Italy
| | - Valentina Bellini
- Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Via Gramsci 14, Parma, 43125, Italy
| | - Federico Semeraro
- Department of Anesthesia, Intensive Care and Prehospital Emergency, Ospedale Maggiore Carlo Alberto Pizzardi, Largo Bartolo Nigrisoli, 2, Bologna, 40133, Italy
| | - Emiliano Gamberini
- Department of Anesthesia and Intensive Care, Infermi Hospital, Romagna Local Health Authority, Viale Settembrini 2, Rimini, 47923, Italy
| | - Emanuele Frontoni
- Department of Political Sciences, Communication and International Relations, University of Macerata, Macerata, 62100, Italy
| | - Vanni Agnoletti
- Department of Surgery and Trauma, Anesthesia and Intensive Care Unit, Maurizio Bufalini Hospital, Romagna Local Health Authority, Viale Giovanni Ghirotti, 286, Cesena, 47521, Italy
| | - Mattia Altini
- Hospital Care Sector, Emilia-Romagna Region, Via Aldo Moro, 21, Bologna, 40127, Italy
| | - Paolo Benanti
- Pontifical Gregorian University, Piazza della Pilotta 4, Roma, 00187, Italy
| | - Elena Giovanna Bignami
- Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Via Gramsci 14, Parma, 43125, Italy
| |
Collapse
|
5
|
Campion JR, O'Connor DB, Lahiff C. Human-artificial intelligence interaction in gastrointestinal endoscopy. World J Gastrointest Endosc 2024; 16:126-135. [PMID: 38577646 PMCID: PMC10989254 DOI: 10.4253/wjge.v16.i3.126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/31/2023] [Revised: 01/18/2024] [Accepted: 02/23/2024] [Indexed: 03/14/2024] Open
Abstract
The number and variety of applications of artificial intelligence (AI) in gastrointestinal (GI) endoscopy is growing rapidly. New technologies based on machine learning (ML) and convolutional neural networks (CNNs) are at various stages of development and deployment to assist patients and endoscopists in preparing for endoscopic procedures, in detection, diagnosis and classification of pathology during endoscopy and in confirmation of key performance indicators. Platforms based on ML and CNNs require regulatory approval as medical devices. Interactions between humans and the technologies we use are complex and are influenced by design, behavioural and psychological elements. Due to the substantial differences between AI and prior technologies, important differences may be expected in how we interact with advice from AI technologies. Human–AI interaction (HAII) may be optimised by developing AI algorithms to minimise false positives and designing platform interfaces to maximise usability. Human factors influencing HAII may include automation bias, alarm fatigue, algorithm aversion, learning effect and deskilling. Each of these areas merits further study in the specific setting of AI applications in GI endoscopy and professional societies should engage to ensure that sufficient emphasis is placed on human-centred design in development of new AI technologies.
Collapse
Affiliation(s)
- John R Campion
- Department of Gastroenterology, Mater Misericordiae University Hospital, Dublin D07 AX57, Ireland
- School of Medicine, University College Dublin, Dublin D04 C7X2, Ireland
| | - Donal B O'Connor
- Department of Surgery, Trinity College Dublin, Dublin D02 R590, Ireland
| | - Conor Lahiff
- Department of Gastroenterology, Mater Misericordiae University Hospital, Dublin D07 AX57, Ireland
- School of Medicine, University College Dublin, Dublin D04 C7X2, Ireland
| |
Collapse
|
6
|
Sibbald M, Zwaan L, Yilmaz Y, Lal S. Incorporating artificial intelligence in medical diagnosis: A case for an invisible and (un)disruptive approach. J Eval Clin Pract 2024; 30:3-8. [PMID: 35761764 DOI: 10.1111/jep.13730] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Accepted: 06/13/2022] [Indexed: 12/30/2022]
Abstract
As big data becomes more publicly accessible, artificial intelligence (AI) is increasingly available and applicable to problems around clinical decision-making. Yet the adoption of AI technology in healthcare lags well behind other industries. The gap between what technology could do, and what technology is actually being used for is rapidly widening. While many solutions are proposed to address this gap, clinician resistance to the adoption of AI remains high. To aid with change, we propose facilitating clinician decisions through technology by seamlessly weaving what we call 'invisible AI' into existing clinician workflows, rather than sequencing new steps into clinical processes. We explore evidence from the change management and human factors literature to conceptualize a new approach to AI implementation in health organizations. We discuss challenges and provide recommendations for organizations to employ this strategy.
Collapse
Affiliation(s)
- Matt Sibbald
- Department of Medicine, McMaster Education Research Innovation and Theory (MERIT) Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Laura Zwaan
- Erasmus Medical Center, Institute of Medical Education Research Rotterdam (iMERR), Rotterdam, The Netherlands
| | - Yusuf Yilmaz
- McMaster Education Research Innovation and Theory (MERIT) Program, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Continuing Professional Development Office, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- Department of Medical Education, Faculty of Medicine, Ege University, Izmir, Turkey
| | - Sarrah Lal
- Department of Medicine, Division of Innovation and Education, McMaster University, Hamilton, ON, Canada
| |
Collapse
|
7
|
Macrae C. Managing risk and resilience in autonomous and intelligent systems: Exploring safety in the development, deployment, and use of artificial intelligence in healthcare. RISK ANALYSIS : AN OFFICIAL PUBLICATION OF THE SOCIETY FOR RISK ANALYSIS 2024. [PMID: 38246857 DOI: 10.1111/risa.14273] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/12/2022] [Revised: 01/07/2024] [Accepted: 01/08/2024] [Indexed: 01/23/2024]
Abstract
Autonomous and intelligent systems (AIS) are being developed and deployed across a wide range of sectors and encompass a variety of technologies designed to engage in different forms of independent reasoning and self-directed behavior. These technologies may bring considerable benefits to society but also pose a range of risk management challenges, particularly when deployed in safety-critical sectors where complex interactions between human, social, and technical processes underpin safety and resilience. Healthcare is one safety-critical sector at the forefront of efforts to develop and deploy intelligent technologies, such as through artificial intelligence (AI) systems intended to automate key aspects of healthcare tasks such as reading medical images to identify signs of pathology. This article develops a qualitative analysis of the sociotechnical sources of risk and resilience associated with the development, deployment, and use of AI in healthcare, drawing on 40 in-depth interviews with participants involved in the development, management, and regulation of AI. Qualitative template analysis is used to examine sociotechnical sources of risk and resilience, drawing on and elaborating Macrae's (2022, Risk Analysis, 42(9), 1999-2025) SOTEC framework that integrates structural, organizational, technological, epistemic, and cultural sources of risk in AIS. This analysis explores an array of sociotechnical sources of risk associated with the development, deployment, and use of AI in healthcare and identifies an array of sociotechnical patterns of resilience that may counter those risks. In doing so, the SOTEC framework is elaborated and translated to define key sources of both risk and resilience in AIS.
Collapse
Affiliation(s)
- Carl Macrae
- Nottingham University Business School, University of Nottingham, Nottingham, UK
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|
8
|
Evans H, Snead D. Why do errors arise in artificial intelligence diagnostic tools in histopathology and how can we minimize them? Histopathology 2024; 84:279-287. [PMID: 37921030 DOI: 10.1111/his.15071] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 09/22/2023] [Accepted: 09/27/2023] [Indexed: 11/04/2023]
Abstract
Artificial intelligence (AI)-based diagnostic tools can offer numerous benefits to the field of histopathology, including improved diagnostic accuracy, efficiency and productivity. As a result, such tools are likely to have an increasing role in routine practice. However, all AI tools are prone to errors, and these AI-associated errors have been identified as a major risk in the introduction of AI into healthcare. The errors made by AI tools are different, in terms of both cause and nature, to the errors made by human pathologists. As highlighted by the National Institute for Health and Care Excellence, it is imperative that practising pathologists understand the potential limitations of AI tools, including the errors made. Pathologists are in a unique position to be gatekeepers of AI tool use, maximizing patient benefit while minimizing harm. Furthermore, their pathological knowledge is essential to understanding when, and why, errors have occurred and so to developing safer future algorithms. This paper summarises the literature on errors made by AI diagnostic tools in histopathology. These include erroneous errors, data concerns (data bias, hidden stratification, data imbalances, distributional shift, and lack of generalisability), reinforcement of outdated practices, unsafe failure mode, automation bias, and insensitivity to impact. Methods to reduce errors in both tool design and clinical use are discussed, and the practical roles for pathologists in error minimisation are highlighted. This aims to inform and empower pathologists to move safely through this seismic change in practice and help ensure that novel AI tools are adopted safely.
Collapse
Affiliation(s)
- Harriet Evans
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
- Warwick Medical School, University of Warwick, Coventry, UK
| | - David Snead
- Histopathology Department, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
- Warwick Medical School, University of Warwick, Coventry, UK
| |
Collapse
|
9
|
Staes CJ, Beck AC, Chalkidis G, Scheese CH, Taft T, Guo JW, Newman MG, Kawamoto K, Sloss EA, McPherson JP. Design of an interface to communicate artificial intelligence-based prognosis for patients with advanced solid tumors: a user-centered approach. J Am Med Inform Assoc 2023; 31:174-187. [PMID: 37847666 PMCID: PMC10746322 DOI: 10.1093/jamia/ocad201] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Revised: 09/18/2023] [Accepted: 10/02/2023] [Indexed: 10/19/2023] Open
Abstract
OBJECTIVES To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design. MATERIALS AND METHODS Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements. RESULTS Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist. CONCLUSION User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.
Collapse
Affiliation(s)
- Catherine J Staes
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Anna C Beck
- Department of Internal Medicine, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT 84112, United States
| | - George Chalkidis
- Healthcare IT Research Department, Center for Digital Services, Hitachi Ltd., Tokyo, Japan
| | - Carolyn H Scheese
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Teresa Taft
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Jia-Wen Guo
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Michael G Newman
- Department of Population Sciences, Huntsman Cancer Institute, Salt Lake City, UT 84112, United States
| | - Kensaku Kawamoto
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Elizabeth A Sloss
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
| | - Jordan P McPherson
- Department of Pharmacotherapy, College of Pharmacy, University of Utah, Salt Lake City, UT 84108, United States
- Department of Pharmacy, Huntsman Cancer Institute, Salt Lake City, UT 84112, United States
| |
Collapse
|
10
|
Li LT, Haley LC, Boyd AK, Bernstam EV. Technical/Algorithm, Stakeholder, and Society (TASS) barriers to the application of artificial intelligence in medicine: A systematic review. J Biomed Inform 2023; 147:104531. [PMID: 37884177 DOI: 10.1016/j.jbi.2023.104531] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 09/14/2023] [Accepted: 10/22/2023] [Indexed: 10/28/2023]
Abstract
INTRODUCTION The use of artificial intelligence (AI), particularly machine learning and predictive analytics, has shown great promise in health care. Despite its strong potential, there has been limited use in health care settings. In this systematic review, we aim to determine the main barriers to successful implementation of AI in healthcare and discuss potential ways to overcome these challenges. METHODS We conducted a literature search in PubMed (1/1/2001-1/1/2023). The search was restricted to publications in the English language, and human study subjects. We excluded articles that did not discuss AI, machine learning, predictive analytics, and barriers to the use of these techniques in health care. Using grounded theory methodology, we abstracted concepts to identify major barriers to AI use in medicine. RESULTS We identified a total of 2,382 articles. After reviewing the 306 included papers, we developed 19 major themes, which we categorized into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). These themes included: Lack of Explainability, Need for Validation Protocols, Need for Standards for Interoperability, Need for Reporting Guidelines, Need for Standardization of Performance Metrics, Lack of Plan for Updating Algorithm, Job Loss, Skills Loss, Workflow Challenges, Loss of Patient Autonomy and Consent, Disturbing the Patient-Clinician Relationship, Lack of Trust in AI, Logistical Challenges, Lack of strategic plan, Lack of Cost-effectiveness Analysis and Proof of Efficacy, Privacy, Liability, Bias and Social Justice, and Education. CONCLUSION We identified 19 major barriers to the use of AI in healthcare and categorized them into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). Future studies should expand on barriers in pediatric care and focus on developing clearly defined protocols to overcome these barriers.
Collapse
Affiliation(s)
- Linda T Li
- Department of Surgery, Division of Pediatric Surgery, Icahn School of Medicine at Mount Sinai, 1 Gustave L. Levy Pl, New York, NY 10029, United States; McWilliams School of Biomedical Informatics at UT Health Houston, 7000 Fannin St, Suite 600, Houston, TX 77030, United States.
| | - Lauren C Haley
- McGovern Medical School at the University of Texas Health Science Center at Houston, 6431 Fannin St, Houston, TX 77030, United States.
| | - Alexandra K Boyd
- McGovern Medical School at the University of Texas Health Science Center at Houston, 6431 Fannin St, Houston, TX 77030, United States.
| | - Elmer V Bernstam
- McWilliams School of Biomedical Informatics at UT Health Houston, 7000 Fannin St, Suite 600, Houston, TX 77030, United States; McGovern Medical School at the University of Texas Health Science Center at Houston, 6431 Fannin St, Houston, TX 77030, United States.
| |
Collapse
|
11
|
Monteith S, Glenn T, Geddes JR, Achtyes ED, Whybrow PC, Bauer M. Challenges and Ethical Considerations to Successfully Implement Artificial Intelligence in Clinical Medicine and Neuroscience: a Narrative Review. PHARMACOPSYCHIATRY 2023; 56:209-213. [PMID: 37643732 DOI: 10.1055/a-2142-9325] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/31/2023]
Abstract
This narrative review discusses how the safe and effective use of clinical artificial intelligence (AI) prediction tools requires recognition of the importance of human intelligence. Human intelligence, creativity, situational awareness, and professional knowledge, are required for successful implementation. The implementation of clinical AI prediction tools may change the workflow in medical practice resulting in new challenges and safety implications. Human understanding of how a clinical AI prediction tool performs in routine and exceptional situations is fundamental to successful implementation. Physicians must be involved in all aspects of the selection, implementation, and ongoing product monitoring of clinical AI prediction tools.
Collapse
Affiliation(s)
- Scott Monteith
- Department of Psychiatry, Michigan State University College of Human Medicine, Traverse City Campus, Traverse City, MI, USA
| | - Tasha Glenn
- ChronoRecord Association, Fullerton, CA, USA
| | - John R Geddes
- Department of Psychiatry, University of Oxford, Warneford Hospital, Oxford, UK
| | - Eric D Achtyes
- Department of Psychiatry, Western Michigan University Homer Stryker M.D. School of Medicine, Kalamazoo, MI, USA
| | - Peter C Whybrow
- Department of Psychiatry and Biobehavioral Sciences, Semel Institute for Neuroscience and Human Behavior, University of California Los Angeles (UCLA), Los Angeles, CA, USA
| | - Michael Bauer
- Department of Psychiatry and Psychotherapy, University Hospital Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
12
|
Suppadungsuk S, Thongprayoon C, Miao J, Krisanapan P, Qureshi F, Kashani K, Cheungpasitporn W. Exploring the Potential of Chatbots in Critical Care Nephrology. MEDICINES (BASEL, SWITZERLAND) 2023; 10:58. [PMID: 37887265 PMCID: PMC10608511 DOI: 10.3390/medicines10100058] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/05/2023] [Revised: 10/17/2023] [Accepted: 10/18/2023] [Indexed: 10/28/2023]
Abstract
The exponential growth of artificial intelligence (AI) has allowed for its integration into multiple sectors, including, notably, healthcare. Chatbots have emerged as a pivotal resource for improving patient outcomes and assisting healthcare practitioners through various AI-based technologies. In critical care, kidney-related conditions play a significant role in determining patient outcomes. This article examines the potential for integrating chatbots into the workflows of critical care nephrology to optimize patient care. We detail their specific applications in critical care nephrology, such as managing acute kidney injury, alert systems, and continuous renal replacement therapy (CRRT); facilitating discussions around palliative care; and bolstering collaboration within a multidisciplinary team. Chatbots have the potential to augment real-time data availability, evaluate renal health, identify potential risk factors, build predictive models, and monitor patient progress. Moreover, they provide a platform for enhancing communication and education for both patients and healthcare providers, paving the way for enriched knowledge and honed professional skills. However, it is vital to recognize the inherent challenges and limitations when using chatbots in this domain. Here, we provide an in-depth exploration of the concerns tied to chatbots' accuracy, dependability, data protection and security, transparency, potential algorithmic biases, and ethical implications in critical care nephrology. While human discernment and intervention are indispensable, especially in complex medical scenarios or intricate situations, the sustained advancements in AI signal that the integration of precision-engineered chatbot algorithms within critical care nephrology has considerable potential to elevate patient care and pivotal outcome metrics in the future.
Collapse
Affiliation(s)
- Supawadee Suppadungsuk
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
- Chakri Naruebodindra Medical Institute, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Samut Prakan 10540, Thailand
| | - Charat Thongprayoon
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
| | - Jing Miao
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
| | - Pajaree Krisanapan
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
- Division of Nephrology and Hypertension, Thammasat University Hospital, Pathum Thani 12120, Thailand
| | - Fawad Qureshi
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
| | - Kianoush Kashani
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
| | - Wisit Cheungpasitporn
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
| |
Collapse
|
13
|
Sujan M. Integrating digital health technologies into complex clinical systems. BMJ Health Care Inform 2023; 30:e100885. [PMID: 37832968 PMCID: PMC10583035 DOI: 10.1136/bmjhci-2023-100885] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Accepted: 09/26/2023] [Indexed: 10/15/2023] Open
Affiliation(s)
- Mark Sujan
- Investigation Education, Health Services Safety Investigation Body, Poole, UK
- Human Factors Everywhere, Woking, UK
| |
Collapse
|
14
|
Peek N, Sujan M, Scott P. Digital health and care: emerging from pandemic times. BMJ Health Care Inform 2023; 30:e100861. [PMID: 37832967 PMCID: PMC10583078 DOI: 10.1136/bmjhci-2023-100861] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2023] [Accepted: 09/20/2023] [Indexed: 10/15/2023] Open
Abstract
In 2020, we published an editorial about the massive disruption of health and care services caused by the COVID-19 pandemic and the rapid changes in digital service delivery, artificial intelligence and data sharing that were taking place at the time. Now, 3 years later, we describe how these developments have progressed since, reflect on lessons learnt and consider key challenges and opportunities ahead by reviewing significant developments reported in the literature. As before, the three key areas we consider are digital transformation of services, realising the potential of artificial intelligence and wise data sharing to facilitate learning health systems. We conclude that the field of digital health has rapidly matured during the pandemic, but there are still major sociotechnical, evaluation and trust challenges in the development and deployment of new digital services.
Collapse
Affiliation(s)
- Niels Peek
- Centre for Health Informatics, The University of Manchester, Manchester, UK
- NIHR Applied Research Collaboration Greater Manchester, The University of Manchester, Manchester, UK
| | - Mark Sujan
- Human Factors Everywhere Ltd, Woking, UK
| | - Philip Scott
- Institute of Management and Health, University of Wales Trinity Saint David, Swansea, UK
| |
Collapse
|
15
|
Wang DY, Ding J, Sun AL, Liu SG, Jiang D, Li N, Yu JK. Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias. J Am Med Inform Assoc 2023; 30:1684-1692. [PMID: 37561535 PMCID: PMC10531198 DOI: 10.1093/jamia/ocad118] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2022] [Revised: 05/30/2023] [Accepted: 06/19/2023] [Indexed: 08/11/2023] Open
Abstract
BACKGROUND Incorporating artificial intelligence (AI) into clinics brings the risk of automation bias, which potentially misleads the clinician's decision-making. The purpose of this study was to propose a potential strategy to mitigate automation bias. METHODS This was a laboratory study with a randomized cross-over design. The diagnosis of anterior cruciate ligament (ACL) rupture, a common injury, on magnetic resonance imaging (MRI) was used as an example. Forty clinicians were invited to diagnose 200 ACLs with and without AI assistance. The AI's correcting and misleading (automation bias) effects on the clinicians' decision-making processes were analyzed. An ordinal logistic regression model was employed to predict the correcting and misleading probabilities of the AI. We further proposed an AI suppression strategy that retracted AI diagnoses with a higher misleading probability and provided AI diagnoses with a higher correcting probability. RESULTS The AI significantly increased clinicians' accuracy from 87.2%±13.1% to 96.4%±1.9% (P < .001). However, the clinicians' errors in the AI-assisted round were associated with automation bias, accounting for 45.5% of the total mistakes. The automation bias was found to affect clinicians of all levels of expertise. Using a logistic regression model, we identified an AI output zone with higher probability to generate misleading diagnoses. The proposed AI suppression strategy was estimated to decrease clinicians' automation bias by 41.7%. CONCLUSION Although AI improved clinicians' diagnostic performance, automation bias was a serious problem that should be addressed in clinical practice. The proposed AI suppression strategy is a practical method for decreasing automation bias.
Collapse
Affiliation(s)
- Ding-Yu Wang
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| | - Jia Ding
- Beijing Yizhun Medical AI Co., Ltd, Beijing, China
| | - An-Lan Sun
- Beijing Yizhun Medical AI Co., Ltd, Beijing, China
| | - Shang-Gui Liu
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| | - Dong Jiang
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| | - Nan Li
- Research Center of Clinical Epidemiology, Peking University Third Hospital, Beijing, China
| | - Jia-Kuo Yu
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| |
Collapse
|
16
|
Garcia Valencia OA, Suppadungsuk S, Thongprayoon C, Miao J, Tangpanithandee S, Craici IM, Cheungpasitporn W. Ethical Implications of Chatbot Utilization in Nephrology. J Pers Med 2023; 13:1363. [PMID: 37763131 PMCID: PMC10532744 DOI: 10.3390/jpm13091363] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Revised: 09/01/2023] [Accepted: 09/06/2023] [Indexed: 09/29/2023] Open
Abstract
This comprehensive critical review critically examines the ethical implications associated with integrating chatbots into nephrology, aiming to identify concerns, propose policies, and offer potential solutions. Acknowledging the transformative potential of chatbots in healthcare, responsible implementation guided by ethical considerations is of the utmost importance. The review underscores the significance of establishing robust guidelines for data collection, storage, and sharing to safeguard privacy and ensure data security. Future research should prioritize defining appropriate levels of data access, exploring anonymization techniques, and implementing encryption methods. Transparent data usage practices and obtaining informed consent are fundamental ethical considerations. Effective security measures, including encryption technologies and secure data transmission protocols, are indispensable for maintaining the confidentiality and integrity of patient data. To address potential biases and discrimination, the review suggests regular algorithm reviews, diversity strategies, and ongoing monitoring. Enhancing the clarity of chatbot capabilities, developing user-friendly interfaces, and establishing explicit consent procedures are essential for informed consent. Striking a balance between automation and human intervention is vital to preserve the doctor-patient relationship. Cultural sensitivity and multilingual support should be considered through chatbot training. To ensure ethical chatbot utilization in nephrology, it is imperative to prioritize the development of comprehensive ethical frameworks encompassing data handling, security, bias mitigation, informed consent, and collaboration. Continuous research and innovation in this field are crucial for maximizing the potential of chatbot technology and ultimately improving patient outcomes.
Collapse
Affiliation(s)
- Oscar A. Garcia Valencia
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (O.A.G.V.); (S.S.); (C.T.); (J.M.); (S.T.); (I.M.C.)
| | - Supawadee Suppadungsuk
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (O.A.G.V.); (S.S.); (C.T.); (J.M.); (S.T.); (I.M.C.)
- Chakri Naruebodindra Medical Institute, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Samut Prakan 10540, Thailand
| | - Charat Thongprayoon
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (O.A.G.V.); (S.S.); (C.T.); (J.M.); (S.T.); (I.M.C.)
| | - Jing Miao
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (O.A.G.V.); (S.S.); (C.T.); (J.M.); (S.T.); (I.M.C.)
| | - Supawit Tangpanithandee
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (O.A.G.V.); (S.S.); (C.T.); (J.M.); (S.T.); (I.M.C.)
- Chakri Naruebodindra Medical Institute, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Samut Prakan 10540, Thailand
| | - Iasmina M. Craici
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (O.A.G.V.); (S.S.); (C.T.); (J.M.); (S.T.); (I.M.C.)
| | - Wisit Cheungpasitporn
- Division of Nephrology and Hypertension, Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA; (O.A.G.V.); (S.S.); (C.T.); (J.M.); (S.T.); (I.M.C.)
| |
Collapse
|
17
|
Magrabi F, Lyell D, Coiera E. Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings. Yearb Med Inform 2023; 32:115-126. [PMID: 38147855 PMCID: PMC10751141 DOI: 10.1055/s-0043-1768733] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2023] Open
Abstract
AIMS AND OBJECTIVES To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings. METHOD PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised. RESULTS AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making. CONCLUSION AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.
Collapse
Affiliation(s)
- Farah Magrabi
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - David Lyell
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Enrico Coiera
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| |
Collapse
|
18
|
Lyell D, Wang Y, Coiera E, Magrabi F. More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA. J Am Med Inform Assoc 2023; 30:1227-1236. [PMID: 37071804 PMCID: PMC10280342 DOI: 10.1093/jamia/ocad065] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2022] [Revised: 03/13/2023] [Accepted: 03/30/2023] [Indexed: 04/20/2023] Open
Abstract
OBJECTIVE To examine the real-world safety problems involving machine learning (ML)-enabled medical devices. MATERIALS AND METHODS We analyzed 266 safety events involving approved ML medical devices reported to the US FDA's MAUDE program between 2015 and October 2021. Events were reviewed against an existing framework for safety problems with Health IT to identify whether a reported problem was due to the ML device (device problem) or its use, and key contributors to the problem. Consequences of events were also classified. RESULTS Events described hazards with potential to harm (66%), actual harm (16%), consequences for healthcare delivery (9%), near misses that would have led to harm if not for intervention (4%), no harm or consequences (3%), and complaints (2%). While most events involved device problems (93%), use problems (7%) were 4 times more likely to harm (relative risk 4.2; 95% CI 2.5-7). Problems with data input to ML devices were the top contributor to events (82%). DISCUSSION Much of what is known about ML safety comes from case studies and the theoretical limitations of ML. We contribute a systematic analysis of ML safety problems captured as part of the FDA's routine post-market surveillance. Most problems involved devices and concerned the acquisition of data for processing by algorithms. However, problems with the use of devices were more likely to harm. CONCLUSIONS Safety problems with ML devices involve more than algorithms, highlighting the need for a whole-of-system approach to safe implementation with a special focus on how users interact with devices.
Collapse
Affiliation(s)
- David Lyell
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia
| | - Ying Wang
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia
| | - Enrico Coiera
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia
| | - Farah Magrabi
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia
| |
Collapse
|
19
|
Ibba S, Tancredi C, Fantesini A, Cellina M, Presta R, Montanari R, Papa S, Alì M. How do patients perceive the AI-radiologists interaction? Results of a survey on 2119 responders. Eur J Radiol 2023; 165:110917. [PMID: 37327548 DOI: 10.1016/j.ejrad.2023.110917] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Revised: 05/16/2023] [Accepted: 05/31/2023] [Indexed: 06/18/2023]
Abstract
PURPOSE In this study we investigate how patients perceive the interaction between artificial intelligence (AI) and radiologists by designing a survey. METHOD We created a survey focused on the application of Artificial Intelligence in radiology which consisted of 20 questions distributed in three sections:Only completed questionnaires were considered for analysis. RESULTS 2119 subjects completed the survey. Among them, 1216 respondents were over 60 years old, showing interest in AI even though they were not digital natives. Although >45% of the respondents reported a high level of education, only 3% said they were AI experts. 87% of respondents favored using AI to support diagnosis but would like to be informed. Only 10% would consult another specialist if their doctor used AI support. Most respondents (76%) said they would not feel comfortable if the diagnosis was made by the AI alone, highlighting the importance of the physician's role in the emotional management of the patient. Finally, 36% of respondents were willing to discuss the topic further in a focus group. CONCLUSION Patients' perception of the use of AI in radiology was positive, although still strictly linked to the supervision of the radiologist. Respondents showed interest and willingness to learn more about AI in the medical field, confirming how patients' confidence in AI technology and its acceptance is central to its widespread use in clinical practice.
Collapse
Affiliation(s)
- Simona Ibba
- Unit of Diagnostic Imaging and Stereotactic Radiosurgery, CDI Centro Diagnostico Italiano S.p.A., Via Simone Saint Bon 20, 20147 Milan, Italy.
| | - Chiara Tancredi
- Suor Orsola Benincasa University, Corso Vittorio Emanuele 292, 80135 Naples, Italy.
| | - Arianna Fantesini
- Suor Orsola Benincasa University, Corso Vittorio Emanuele 292, 80135 Naples, Italy; RE:LAB s.r.l., Via Tamburini, 5, 42122 Reggio Emilia, Italy.
| | - Michaela Cellina
- Radiology Department, ASST Fatebenefratelli Sacco, Piazza Principessa Clotilde 3, 20121 Milan, Italy.
| | - Roberta Presta
- Suor Orsola Benincasa University, Corso Vittorio Emanuele 292, 80135 Naples, Italy.
| | - Roberto Montanari
- Suor Orsola Benincasa University, Corso Vittorio Emanuele 292, 80135 Naples, Italy; RE:LAB s.r.l., Via Tamburini, 5, 42122 Reggio Emilia, Italy.
| | - Sergio Papa
- Unit of Diagnostic Imaging and Stereotactic Radiosurgery, CDI Centro Diagnostico Italiano S.p.A., Via Simone Saint Bon 20, 20147 Milan, Italy.
| | - Marco Alì
- Unit of Diagnostic Imaging and Stereotactic Radiosurgery, CDI Centro Diagnostico Italiano S.p.A., Via Simone Saint Bon 20, 20147 Milan, Italy; Bracco Imaging S.p.A., Via Egidio Folli, 50, 20134 Milan, Italy.
| |
Collapse
|
20
|
Sujan M, Smith-Frazer C, Malamateniou C, Connor J, Gardner A, Unsworth H, Husain H. Validation framework for the use of AI in healthcare: overview of the new British standard BS30440. BMJ Health Care Inform 2023; 30:e100749. [PMID: 37364922 PMCID: PMC10410839 DOI: 10.1136/bmjhci-2023-100749] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Accepted: 05/09/2023] [Indexed: 06/28/2023] Open
Affiliation(s)
- Mark Sujan
- Human Factors Everywhere, Woking, UK
- Education, Healthcare Safety Investigation Branch, Reading, UK
| | | | | | | | - Allison Gardner
- National Institute for Health and Care Excellence, London, UK
| | | | | |
Collapse
|
21
|
Rees N, Holding K, Sujan M. Information governance as a socio-technical process in the development of trustworthy healthcare AI. FRONTIERS IN COMPUTER SCIENCE 2023. [DOI: 10.3389/fcomp.2023.1134818] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/19/2023] Open
Abstract
In this paper we describe our experiences of managing information governance (IG) processes for the assurance of healthcare AI, using the example of an out-of-hospital-cardiac-arrest recognition software within the context of the Welsh Ambulance Service. We frame IG as a socio-technical process. IG processes for the development of trustworthy healthcare AI rely on information governance work, which entails dialogue, negotiation, and trade-offs around the legal basis for data sharing, data requirements and data control. Information governance work should start early in the design life cycle and will likely continue throughout. This includes a focus on establishing and building relationships, as well as a focus on organizational readiness and deeper understanding of both AI technologies as well as their safety assurance requirements.
Collapse
|
22
|
Petersson L, Vincent K, Svedberg P, Nygren JM, Larsson I. Ethical considerations in implementing AI for mortality prediction in the emergency department: Linking theory and practice. Digit Health 2023; 9:20552076231206588. [PMID: 37829612 PMCID: PMC10566278 DOI: 10.1177/20552076231206588] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/21/2023] [Indexed: 10/14/2023] Open
Abstract
Background Artificial intelligence (AI) is predicted to be a solution for improving healthcare, increasing efficiency, and saving time and recourses. A lack of ethical principles for the use of AI in practice has been highlighted by several stakeholders due to the recent attention given to it. Research has shown an urgent need for more knowledge regarding the ethical implications of AI applications in healthcare. However, fundamental ethical principles may not be sufficient to describe ethical concerns associated with implementing AI applications. Objective The aim of this study is twofold, (1) to use the implementation of AI applications to predict patient mortality in emergency departments as a setting to explore healthcare professionals' perspectives on ethical issues in relation to ethical principles and (2) to develop a model to guide ethical considerations in AI implementation in healthcare based on ethical theory. Methods Semi-structured interviews were conducted with 18 participants. The abductive approach used to analyze the empirical data consisted of four steps alternating between inductive and deductive analyses. Results Our findings provide an ethical model demonstrating the need to address six ethical principles (autonomy, beneficence, non-maleficence, justice, explicability, and professional governance) in relation to ethical theories defined as virtue, deontology, and consequentialism when AI applications are to be implemented in clinical practice. Conclusions Ethical aspects of AI applications are broader than the prima facie principles of medical ethics and the principle of explicability. Ethical aspects thus need to be viewed from a broader perspective to cover different situations that healthcare professionals, in general, and physicians, in particular, may face when using AI applications in clinical practice.
Collapse
Affiliation(s)
- Lena Petersson
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Kalista Vincent
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Petra Svedberg
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Jens M Nygren
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| | - Ingrid Larsson
- School of Health and Welfare, Halmstad University, Halmstad, Sweden
| |
Collapse
|
23
|
Kostick-Quenet KM, Gerke S. AI in the hands of imperfect users. NPJ Digit Med 2022; 5:197. [PMID: 36577851 PMCID: PMC9795935 DOI: 10.1038/s41746-022-00737-z] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Accepted: 11/29/2022] [Indexed: 12/29/2022] Open
Abstract
As the use of artificial intelligence and machine learning (AI/ML) continues to expand in healthcare, much attention has been given to mitigating bias in algorithms to ensure they are employed fairly and transparently. Less attention has fallen to addressing potential bias among AI/ML's human users or factors that influence user reliance. We argue for a systematic approach to identifying the existence and impacts of user biases while using AI/ML tools and call for the development of embedded interface design features, drawing on insights from decision science and behavioral economics, to nudge users towards more critical and reflective decision making using AI/ML.
Collapse
Affiliation(s)
| | - Sara Gerke
- Penn State Dickinson Law, Carlisle, PA, USA
| |
Collapse
|
24
|
Coiera E, Liu S. Evidence synthesis, digital scribes, and translational challenges for artificial intelligence in healthcare. Cell Rep Med 2022; 3:100860. [PMID: 36513071 PMCID: PMC9798027 DOI: 10.1016/j.xcrm.2022.100860] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 10/15/2022] [Accepted: 11/18/2022] [Indexed: 12/14/2022]
Abstract
Healthcare has well-known challenges with safety, quality, and effectiveness, and many see artificial intelligence (AI) as essential to any solution. Emerging applications include the automated synthesis of best-practice research evidence including systematic reviews, which would ultimately see all clinical trial data published in a computational form for immediate synthesis. Digital scribes embed themselves in the process of care to detect, record, and summarize events and conversations for the electronic record. However, three persistent translational challenges must be addressed before AI is widely deployed. First, little effort is spent replicating AI trials, exposing patients to risks of methodological error and biases. Next, there is little reporting of patient harms from trials. Finally, AI built using machine learning may perform less effectively in different clinical settings.
Collapse
Affiliation(s)
- Enrico Coiera
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Level 6, 75 Talavera Road, North Ryde, Sydney, NSW 2109, Australia.
| | - Sidong Liu
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Level 6, 75 Talavera Road, North Ryde, Sydney, NSW 2109, Australia
| |
Collapse
|
25
|
Salwei ME, Carayon P. A Sociotechnical Systems Framework for the Application of Artificial Intelligence in Health Care Delivery. JOURNAL OF COGNITIVE ENGINEERING AND DECISION MAKING 2022; 16:194-206. [PMID: 36704421 PMCID: PMC9873227 DOI: 10.1177/15553434221097357] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
In the coming years, artificial intelligence (AI) will pervade almost every aspect of the health care delivery system. AI has the potential to improve patient safety (e.g. diagnostic accuracy) as well as reduce the burden on clinicians (e.g. documentation-related workload); however, these benefits are yet to be realized. AI is only one element of a larger sociotechnical system that needs to be considered for effective AI application. In this paper, we describe the current challenges of integrating AI into clinical care and propose a sociotechnical systems (STS) approach for AI design and implementation. We demonstrate the importance of an STS approach through a case study on the design and implementation of a clinical decision support (CDS). In order for AI to reach its potential, the entire work system as well as clinical workflow must be systematically considered throughout the design of AI technology.
Collapse
Affiliation(s)
- Megan E. Salwei
- Center for Research and Innovation in Systems Safety, Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, TN
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN
| | - Pascale Carayon
- Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI
- Wisconsin Institute for Healthcare Systems Engineering, University of Wisconsin-Madison, Madison, WI
| |
Collapse
|
26
|
|
27
|
Festor P, Jia Y, Gordon AC, Faisal AA, Habli I, Komorowski M. Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment. BMJ Health Care Inform 2022; 29:bmjhci-2022-100549. [PMID: 35851286 PMCID: PMC9289024 DOI: 10.1136/bmjhci-2022-100549] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2022] [Accepted: 06/04/2022] [Indexed: 11/25/2022] Open
Abstract
Objectives Establishing confidence in the safety of Artificial Intelligence (AI)-based clinical decision support systems is important prior to clinical deployment and regulatory approval for systems with increasing autonomy. Here, we undertook safety assurance of the AI Clinician, a previously published reinforcement learning-based treatment recommendation system for sepsis. Methods As part of the safety assurance, we defined four clinical hazards in sepsis resuscitation based on clinical expert opinion and the existing literature. We then identified a set of unsafe scenarios, intended to limit the action space of the AI agent with the goal of reducing the likelihood of hazardous decisions. Results Using a subset of the Medical Information Mart for Intensive Care (MIMIC-III) database, we demonstrated that our previously published ‘AI clinician’ recommended fewer hazardous decisions than human clinicians in three out of our four predefined clinical scenarios, while the difference was not statistically significant in the fourth scenario. Then, we modified the reward function to satisfy our safety constraints and trained a new AI Clinician agent. The retrained model shows enhanced safety, without negatively impacting model performance. Discussion While some contextual patient information absent from the data may have pushed human clinicians to take hazardous actions, the data were curated to limit the impact of this confounder. Conclusion These advances provide a use case for the systematic safety assurance of AI-based clinical systems towards the generation of explicit safety evidence, which could be replicated for other AI applications or other clinical contexts, and inform medical device regulatory bodies.
Collapse
Affiliation(s)
- Paul Festor
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK
- Brain & Behvaiour Lab: Departments of Bioengineering and Computing, Imperial College London, London, UK
| | - Yan Jia
- Assuring Autonomy International Programme, University of York, York, UK
- Department of Computing, University of York, York, UK
| | - Anthony C Gordon
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - A Aldo Faisal
- Brain & Behvaiour Lab: Departments of Bioengineering and Computing, Imperial College London, London, UK
- Institute of artificial and human intellgience, Universität Bayreuth, Bayreuth, Bayern, Germany
| | - Ibrahim Habli
- Assuring Autonomy International Programme, University of York, York, UK
- Department of Computer Science, University of York, York, UK
| | - Matthieu Komorowski
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK
- Department of Surgery and Cancer, Imperial College London, London, UK
| |
Collapse
|
28
|
Held LA, Wewetzer L, Steinhäuser J. Determinants of the implementation of an artificial intelligence-supported device for the screening of diabetic retinopathy in primary care - a qualitative study. Health Informatics J 2022; 28:14604582221112816. [PMID: 35921547 DOI: 10.1177/14604582221112816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Diabetic retinopathy is a microvascular complication of diabetes mellitus that is usually asymptomatic in the early stages. Therefore, its timely detection and treatment are essential. First pilot projects exist to establish a smartphone-based and AI-supported screening of DR in primary care. This study explored health professionals' perceptions of potential barriers and enablers of using a screening such as this in primary care to understand the mechanisms that could influence implementation into routine clinical practice. Semi-structured telephone interviews were conducted and analysed with the help of qualitative analysis of Mayring. The following main influencing factors to implementation have been identified: personal attitude, organisation, time, financial factors, education, support, technical requirement, influence on profession and patient welfare. Most determinants could be relocated in the behaviour change wheel, a validated implementation model. Further research on the patients' perspective and a ranking of the determinants found is needed.
Collapse
Affiliation(s)
- Linda A Held
- Institute of Family Medicine, 54360University Medical Center Schleswig-Holstein, Campus Lübeck, Germany
| | - Larisa Wewetzer
- Institute of Family Medicine, 54360University Medical Center Schleswig-Holstein, Campus Lübeck, Germany
| | - Jost Steinhäuser
- Institute of Family Medicine, 54360University Medical Center Schleswig-Holstein, Campus Lübeck, Germany
| |
Collapse
|
29
|
Sujan M, Thimbleby H, Habli I, Cleve A, Maaløe L, Rees N. Assuring safe artificial intelligence in critical ambulance service response: study protocol. Br Paramed J 2022; 7:36-42. [DOI: 10.29045/14784726.2022.06.7.1.36] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022] Open
Abstract
Introduction: Early recognition of out-of-hospital cardiac arrest (OHCA) by ambulance service call centre operators is important so that cardiopulmonary resuscitation can be delivered immediately, but around 25% of OHCAs are not picked up by call centre operators. An artificial
intelligence (AI) system has been developed to support call centre operators in the detection of OHCA. The study aims to (1) explore ambulance service stakeholder perceptions on the safety of OHCA AI decision support in call centres, and (2) develop a clinical safety case for the OHCA AI decision-support
system.Methods and analysis: The study will be undertaken within the Welsh Ambulance Service. The study is part research and part service evaluation. The research utilises a qualitative study design based on thematic analysis of interview data. The service evaluation consists of
the development of a clinical safety case based on document analysis, analysis of the AI model and its development process and informal interviews with the technology developer.Conclusions: AI presents many opportunities for ambulance services, but safety assurance requirements
need to be understood. The ASSIST project will continue to explore and build the body of knowledge in this area.
Collapse
Affiliation(s)
- Mark Sujan
- Human Factors Everywhere Ltd. ORCID iD:, URL: https://orcid.org/0000-0001-6895-946X
| | | | | | | | | | - Nigel Rees
- Welsh Ambulance Service NHS Trust ORCID iD:, URL: https://orcid.org/0000-0001-8799-5335
| |
Collapse
|
30
|
Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, Denniston AK, Faes L, Geerts B, Ibrahim M, Liu X, Mateen BA, Mathur P, McCradden MD, Morgan L, Ordish J, Rogers C, Saria S, Ting DSW, Watkinson P, Weber W, Wheatstone P, McCulloch P. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ 2022; 377:e070904. [PMID: 35584845 PMCID: PMC9116198 DOI: 10.1136/bmj-2022-070904] [Citation(s) in RCA: 52] [Impact Index Per Article: 26.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/26/2022] [Indexed: 01/04/2023]
Affiliation(s)
- Baptiste Vasey
- Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
- Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Myura Nagendran
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK
| | - Bruce Campbell
- University of Exeter Medical School, Exeter, UK
- Royal Devon and Exeter Hospital, Exeter, UK
| | - David A Clifton
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Spiros Denaxas
- Institute of Health Informatics, University College London, London, UK
- British Heart Foundation Data Science Centre, London, UK
- Health Data Research UK, London, UK
- UCL Hospitals Biomedical Research Centre, London, UK
| | - Alastair K Denniston
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- Moorfields Eye Hospital NHS Foundation Trust, London, UK
| | - Livia Faes
- Moorfields Eye Hospital NHS Foundation Trust, London, UK
| | | | - Mudathir Ibrahim
- Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
- Department of Surgery, Maimonides Medical Center, New York, NY, USA
| | - Xiaoxuan Liu
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Bilal A Mateen
- Institute of Health Informatics, University College London, London, UK
- Wellcome Trust, London, UK
- Alan Turing Institute, London, UK
| | - Piyush Mathur
- Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA
| | - Melissa D McCradden
- Hospital for Sick Children, Toronto, ON, Canada
- Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
| | | | - Johan Ordish
- The Medicines and Healthcare products Regulatory Agency, London, UK
| | | | - Suchi Saria
- Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD, USA
- Bayesian Health, New York, NY, USA
| | - Daniel S W Ting
- Singapore National Eye Center, Singapore Eye Research Institute, Singapore
- Duke-NUS Medical School, National University of Singapore, Singapore
| | - Peter Watkinson
- Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
- NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK
| | | | | | - Peter McCulloch
- Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
| |
Collapse
|
31
|
Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, Denniston AK, Faes L, Geerts B, Ibrahim M, Liu X, Mateen BA, Mathur P, McCradden MD, Morgan L, Ordish J, Rogers C, Saria S, Ting DSW, Watkinson P, Weber W, Wheatstone P, McCulloch P. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med 2022; 28:924-933. [PMID: 35585198 DOI: 10.1038/s41591-022-01772-9] [Citation(s) in RCA: 116] [Impact Index Per Article: 58.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2021] [Accepted: 03/03/2022] [Indexed: 12/31/2022]
Abstract
A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico evaluation, but few have yet demonstrated real benefit to patient care. Early-stage clinical evaluation is important to assess an AI system's actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use and pave the way to further large-scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multi-stakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two-round, modified Delphi process to collect and analyze expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 pre-defined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. In total, 123 experts participated in the first round of Delphi, 138 in the second round, 16 in the consensus meeting and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI-specific reporting items (made of 28 subitems) and ten generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we developed a guideline comprising key items that should be reported in early-stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.
Collapse
Affiliation(s)
- Baptiste Vasey
- Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK.
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK.
- Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
| | - Myura Nagendran
- UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK
| | - Bruce Campbell
- University of Exeter Medical School, Exeter, UK
- Royal Devon and Exeter Hospital, Exeter, UK
| | - David A Clifton
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Spiros Denaxas
- Institute of Health Informatics, University College London, London, UK
- British Heart Foundation Data Science Centre, London, UK
- Health Data Research UK, London, UK
- UCL Hospitals Biomedical Research Centre, London, UK
| | - Alastair K Denniston
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- Moorfields Eye Hospital NHS Foundation Trust, London, UK
| | - Livia Faes
- Moorfields Eye Hospital NHS Foundation Trust, London, UK
| | - Bart Geerts
- Healthplus.ai-R&D BV, Amsterdam, The Netherlands
| | - Mudathir Ibrahim
- Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
- Department of Surgery, Maimonides Medical Center, Brooklyn, NY, USA
| | - Xiaoxuan Liu
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Bilal A Mateen
- Institute of Health Informatics, University College London, London, UK
- The Wellcome Trust, London, UK
- The Alan Turing Institute, London, UK
| | - Piyush Mathur
- Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA
| | - Melissa D McCradden
- The Hospital for Sick Children, Toronto ON, Canada
- Dalla Lana School of Public Health, University of Toronto, Toronto ON, Canada
| | | | - Johan Ordish
- Medicines and Healthcare products Regulatory Agency, London, UK
| | | | - Suchi Saria
- Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD, USA
- Bayesian Health, New York, NY, USA
| | - Daniel S W Ting
- Singapore National Eye Center, Singapore Eye Research Institute, Singapore, Singapore
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
| | - Peter Watkinson
- Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
- NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK
| | | | | | - Peter McCulloch
- Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
| |
Collapse
|
32
|
Tulk Jesso S, Kelliher A, Sanghavi H, Martin T, Henrickson Parker S. Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review. Front Psychol 2022; 13:830345. [PMID: 35465567 PMCID: PMC9022040 DOI: 10.3389/fpsyg.2022.830345] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Accepted: 02/09/2022] [Indexed: 12/11/2022] Open
Abstract
The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.
Collapse
Affiliation(s)
- Stephanie Tulk Jesso
- Fralin Biomedical Research Institute, Virginia Tech, Roanoke, VA, United States.,Institute for Creativity, Arts, and Technology, Blacksburg, VA, United States
| | - Aisling Kelliher
- Department of Computer Science, College of Engineering, Virginia Tech, Blacksburg, VA, United States
| | | | - Thomas Martin
- Institute for Creativity, Arts, and Technology, Blacksburg, VA, United States.,Department of Electrical and Computer Engineering, College of Engineering, Virginia Tech, Blacksburg, VA, United States
| | - Sarah Henrickson Parker
- Fralin Biomedical Research Institute, Virginia Tech, Roanoke, VA, United States.,Department of Health Systems and Implementation Science, Virginia Tech Carilion School of Medicine, Roanoke, VA, United States
| |
Collapse
|
33
|
Sujan M, Pool R, Salmon P. Eight human factors and ergonomics principles for healthcare artificial intelligence. BMJ Health Care Inform 2022; 29:bmjhci-2021-100516. [PMID: 35121617 PMCID: PMC8819549 DOI: 10.1136/bmjhci-2021-100516] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2021] [Accepted: 01/26/2022] [Indexed: 01/21/2023] Open
Affiliation(s)
- Mark Sujan
- Human Factors Everywhere, Woking, UK .,Chartered Institute of Ergonomics and Human Factors, Birmingham, UK
| | | | - Paul Salmon
- Centre for Human Factors and Sociotechnical Systems, University of the Sunshine Coast, Maroochydore DC, Queensland, Australia
| |
Collapse
|
34
|
Sujan M, Pickup L, Bowie P, Hignett S, Ives F, Vosper H, Rashid N. The contribution of human factors and ergonomics to the design and delivery of safe future healthcare. Future Healthc J 2021; 8:e574-e579. [PMID: 34888444 DOI: 10.7861/fhj.2021-0112] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
Abstract
Human factors and ergonomics (HF/E) is concerned with the design of work and work systems. There is an increasing appreciation of the value that HF/E can bring to enhancing the quality and safety of care, but the professionalisation of HF/E in healthcare is still in its infancy. In this paper, we set out a vision for HF/E in healthcare based on the work of the Chartered Institute of Ergonomics and Human Factors (CIEHF), which is the professional body for HF/E in the UK. We consider the contribution of HF/E in design, in digital transformation, in organisational learning and during COVID-19.
Collapse
Affiliation(s)
| | | | | | | | - Fran Ives
- West Midlands Academic Health Science Network, Edgbaston, UK
| | | | - Noorzaman Rashid
- Chartered Institute of Ergonomics and Human Factors, Wootton Wawen, UK
| |
Collapse
|
35
|
Thomasian NM, Eickhoff C, Adashi EY. Advancing health equity with artificial intelligence. J Public Health Policy 2021; 42:602-611. [PMID: 34811466 PMCID: PMC8607970 DOI: 10.1057/s41271-021-00319-5] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/14/2021] [Indexed: 12/17/2022]
Abstract
Population and public health are in the midst of an artificial intelligence revolution capable of radically altering existing models of care delivery and practice. Just as AI seeks to mirror human cognition through its data-driven analytics, it can also reflect the biases present in our collective conscience. In this Viewpoint, we use past and counterfactual examples to illustrate the sequelae of unmitigated bias in healthcare artificial intelligence. Past examples indicate that if the benefits of emerging AI technologies are to be realized, consensus around the regulation of algorithmic bias at the policy level is needed to ensure their ethical integration into the health system. This paper puts forth regulatory strategies for uprooting bias in healthcare AI that can inform ongoing efforts to establish a framework for federal oversight. We highlight three overarching oversight principles in bias mitigation that maps to each phase of the algorithm life cycle.
Collapse
Affiliation(s)
- Nicole M Thomasian
- Warren Alpert Medical School of Brown University, Brown University, 222 Richmond Street, Providence, RI, 02906, USA.
- The Harvard Kennedy School of Government, Harvard University, Cambridge, MA, USA.
| | - Carsten Eickhoff
- Center for Biomedical Informatics, Brown University, Providence, RI, USA
- Department of Computer Science, Brown University, Providence, RI, USA
| | - Eli Y Adashi
- Warren Alpert Medical School of Brown University, Brown University, 222 Richmond Street, Providence, RI, 02906, USA
| |
Collapse
|
36
|
Asan O, Choudhury A. Research Trends in Artificial Intelligence Applications in Human Factors Health Care: Mapping Review. JMIR Hum Factors 2021; 8:e28236. [PMID: 34142968 PMCID: PMC8277302 DOI: 10.2196/28236] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Revised: 04/14/2021] [Accepted: 05/03/2021] [Indexed: 01/17/2023] Open
Abstract
BACKGROUND Despite advancements in artificial intelligence (AI) to develop prediction and classification models, little research has been devoted to real-world translations with a user-centered design approach. AI development studies in the health care context have often ignored two critical factors of ecological validity and human cognition, creating challenges at the interface with clinicians and the clinical environment. OBJECTIVE The aim of this literature review was to investigate the contributions made by major human factors communities in health care AI applications. This review also discusses emerging research gaps, and provides future research directions to facilitate a safer and user-centered integration of AI into the clinical workflow. METHODS We performed an extensive mapping review to capture all relevant articles published within the last 10 years in the major human factors journals and conference proceedings listed in the "Human Factors and Ergonomics" category of the Scopus Master List. In each published volume, we searched for studies reporting qualitative or quantitative findings in the context of AI in health care. Studies are discussed based on the key principles such as evaluating workload, usability, trust in technology, perception, and user-centered design. RESULTS Forty-eight articles were included in the final review. Most of the studies emphasized user perception, the usability of AI-based devices or technologies, cognitive workload, and user's trust in AI. The review revealed a nascent but growing body of literature focusing on augmenting health care AI; however, little effort has been made to ensure ecological validity with user-centered design approaches. Moreover, few studies (n=5 against clinical/baseline standards, n=5 against clinicians) compared their AI models against a standard measure. CONCLUSIONS Human factors researchers should actively be part of efforts in AI design and implementation, as well as dynamic assessments of AI systems' effects on interaction, workflow, and patient outcomes. An AI system is part of a greater sociotechnical system. Investigators with human factors and ergonomics expertise are essential when defining the dynamic interaction of AI within each element, process, and result of the work system.
Collapse
Affiliation(s)
- Onur Asan
- School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, United States
| | - Avishek Choudhury
- School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, United States
| |
Collapse
|
37
|
Lyell D, Coiera E, Chen J, Shah P, Magrabi F. How machine learning is embedded to support clinician decision making: an analysis of FDA-approved medical devices. BMJ Health Care Inform 2021; 28:bmjhci-2020-100301. [PMID: 33853863 PMCID: PMC8054073 DOI: 10.1136/bmjhci-2020-100301] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2020] [Revised: 02/10/2021] [Accepted: 03/02/2021] [Indexed: 12/20/2022] Open
Abstract
Objective To examine how and to what extent medical devices using machine learning (ML) support clinician decision making. Methods We searched for medical devices that were (1) approved by the US Food and Drug Administration (FDA) up till February 2020; (2) intended for use by clinicians; (3) in clinical tasks or decisions and (4) used ML. Descriptive information about the clinical task, device task, device input and output, and ML method were extracted. The stage of human information processing automated by ML-based devices and level of autonomy were assessed. Results Of 137 candidates, 59 FDA approvals for 49 unique devices were included. Most approvals (n=51) were since 2018. Devices commonly assisted with diagnostic (n=35) and triage (n=10) tasks. Twenty-three devices were assistive, providing decision support but left clinicians to make important decisions including diagnosis. Twelve automated the provision of information (autonomous information), such as quantification of heart ejection fraction, while 14 automatically provided task decisions like triaging the reading of scans according to suspected findings of stroke (autonomous decisions). Stages of human information processing most automated by devices were information analysis, (n=14) providing information as an input into clinician decision making, and decision selection (n=29), where devices provide a decision. Conclusion Leveraging the benefits of ML algorithms to support clinicians while mitigating risks, requires a solid relationship between clinician and ML-based devices. Such relationships must be carefully designed, considering how algorithms are embedded in devices, the tasks supported, information provided and clinicians’ interactions with them.
Collapse
Affiliation(s)
- David Lyell
- Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, NSW, Australia
| | - Enrico Coiera
- Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, NSW, Australia
| | - Jessica Chen
- Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, NSW, Australia
| | - Parina Shah
- Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, NSW, Australia
| | - Farah Magrabi
- Australian Institute of Health Innovation, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, NSW, Australia
| |
Collapse
|
38
|
Safety-driven design of machine learning for sepsis treatment. J Biomed Inform 2021; 117:103762. [PMID: 33798716 DOI: 10.1016/j.jbi.2021.103762] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2020] [Revised: 01/28/2021] [Accepted: 03/22/2021] [Indexed: 11/20/2022]
Abstract
Machine learning (ML) has the potential to bring significant clinical benefits. However, there are patient safety challenges in introducing ML in complex healthcare settings and in assuring the technology to the satisfaction of the different regulators. The work presented in this paper tackles the urgent problem of proactively assuring ML in its clinical context as a step towards enabling the safe introduction of ML into clinical practice. In particular, the paper considers the use of deep Reinforcement Learning, a type of ML, for sepsis treatment. The methodology starts with the modelling of a clinical workflow that integrates the ML model for sepsis treatment recommendations. Then safety analysis is carried out based on the clinical workflow, identifying hazards and safety requirements for the ML model. In this paper the design of the ML model is enhanced to satisfy the safety requirements for mitigating a major clinical hazard: sudden change of vasopressor dose. A rigorous evaluation is conducted to show how these requirements are met. A safety case is presented, providing a basis for regulators to make a judgement on the acceptability of introducing the ML model into sepsis treatment in a healthcare setting. The overall argument is broad in considering the wider patient safety considerations, but the detailed rationale and supporting evidence presented relate to this specific hazard. Whilst there are no agreed regulatory approaches to introducing ML into healthcare, the work presented in this paper has shown a possible direction for overcoming this barrier and exploit the benefits of ML without compromising safety.
Collapse
|
39
|
Felmingham CM, Adler NR, Ge Z, Morton RL, Janda M, Mar VJ. The Importance of Incorporating Human Factors in the Design and Implementation of Artificial Intelligence for Skin Cancer Diagnosis in the Real World. Am J Clin Dermatol 2021; 22:233-242. [PMID: 33354741 DOI: 10.1007/s40257-020-00574-4] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
Artificial intelligence (AI) algorithms have been shown to diagnose skin lesions with impressive accuracy in experimental settings. The majority of the literature to date has compared AI and dermatologists as opponents in skin cancer diagnosis. However, in the real-world clinical setting, the clinician will work in collaboration with AI. Existing evidence regarding the integration of such AI diagnostic tools into clinical practice is limited. Human factors, such as cognitive style, personality, experience, preferences, and attitudes may influence clinicians' use of AI. In this review, we consider these human factors and the potential cognitive errors, biases, and unintended consequences that could arise when using an AI skin cancer diagnostic tool in the real world. Integrating this knowledge in the design and implementation of AI technology will assist in ensuring that the end product can be used effectively. Dermatologist leadership in the development of these tools will further improve their clinical relevance and safety.
Collapse
Affiliation(s)
- Claire M Felmingham
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC, Australia.
- Victorian Melanoma Service, Alfred Hospital, 55 Commercial Road, Melbourne, VIC, 3004, Australia.
| | - Nikki R Adler
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC, Australia
| | - Zongyuan Ge
- Monash eResearch Centre, Monash University, Clayton, Australia
- Department of Electrical and Computer Systems Engineering, Faculty of Engineering, Monash University, Melbourne, VIC, Australia
- Monash-Airdoc Research Centre, Monash University, Melbourne, VIC, Australia
| | - Rachael L Morton
- NHMRC Clinical Trials Centre, Faculty of Medicine and Health, University of Sydney, Camperdown, NSW, Australia
| | - Monika Janda
- Centre for Health Services Research, Faculty of Medicine, The University of Queensland, Brisbane, QLD, Australia
| | - Victoria J Mar
- School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC, Australia
- Victorian Melanoma Service, Alfred Hospital, 55 Commercial Road, Melbourne, VIC, 3004, Australia
| |
Collapse
|
40
|
Efficacy of Artificial-Intelligence-Driven Differential-Diagnosis List on the Diagnostic Accuracy of Physicians: An Open-Label Randomized Controlled Study. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18042086. [PMID: 33669930 PMCID: PMC7924871 DOI: 10.3390/ijerph18042086] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 02/12/2021] [Accepted: 02/17/2021] [Indexed: 12/27/2022]
Abstract
Background: The efficacy of artificial intelligence (AI)-driven automated medical-history-taking systems with AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy was shown. However, considering the negative effects of AI-driven differential-diagnosis lists such as omission (physicians reject a correct diagnosis suggested by AI) and commission (physicians accept an incorrect diagnosis suggested by AI) errors, the efficacy of AI-driven automated medical-history-taking systems without AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy should be evaluated. Objective: The present study was conducted to evaluate the efficacy of AI-driven automated medical-history-taking systems with or without AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy. Methods: This randomized controlled study was conducted in January 2021 and included 22 physicians working at a university hospital. Participants were required to read 16 clinical vignettes in which the AI-driven medical history of real patients generated up to three differential diagnoses per case. Participants were divided into two groups: with and without an AI-driven differential-diagnosis list. Results: There was no significant difference in diagnostic accuracy between the two groups (57.4% vs. 56.3%, respectively; p = 0.91). Vignettes that included a correct diagnosis in the AI-generated list showed the greatest positive effect on physicians’ diagnostic accuracy (adjusted odds ratio 7.68; 95% CI 4.68–12.58; p < 0.001). In the group with AI-driven differential-diagnosis lists, 15.9% of diagnoses were omission errors and 14.8% were commission errors. Conclusions: Physicians’ diagnostic accuracy using AI-driven automated medical history did not differ between the groups with and without AI-driven differential-diagnosis lists.
Collapse
|
41
|
Cabitza F, Campagner A, Sconfienza LM. Studying human-AI collaboration protocols: the case of the Kasparov's law in radiological double reading. Health Inf Sci Syst 2021; 9:8. [PMID: 33585029 PMCID: PMC7864624 DOI: 10.1007/s13755-021-00138-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2020] [Accepted: 01/13/2021] [Indexed: 12/17/2022] Open
Abstract
Purpose The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in collaborative tasks, even in currently more viable settings, like independent double-reading screening tasks. Methods To this aim, we report about a retrospective case–control study, involving 12 board-certified radiologists, in the detection of knee lesions by means of Magnetic Resonance Imaging, in which we simulated the serial combination of two Deep Learning models with humans in eight double-reading protocols. Inspired by the so-called Kasparov’s Laws, we investigate whether the combination of humans and AI models could achieve better performance than AI models alone, and whether weak reader, when supported by fit-for-use interaction protocols, could out-perform stronger readers. Results We discuss two main findings: groups of humans who perform significantly worse than a state-of-the-art AI can significantly outperform it if their judgements are aggregated by majority voting (in concordance with the first part of the Kasparov’s law); small ensembles of significantly weaker readers can significantly outperform teams of stronger readers, supported by the same computational tool, when the judgments of the former ones are combined within “fit-for-use” protocols (in concordance with the second part of the Kasparov’s law). Conclusion Our study shows that good interaction protocols can guarantee improved decision performance that easily surpasses the performance of individual agents, even of realistic super-human AI systems. This finding highlights the importance of focusing on how to guarantee better co-operation within human-AI teams, so to enable safer and more human sustainable care practices.
Collapse
Affiliation(s)
- Federico Cabitza
- Università degli Studi di Milano-Bicocca, Viale Sarca 336, 20126 Milan, Italy
| | - Andrea Campagner
- Università degli Studi di Milano-Bicocca, Viale Sarca 336, 20126 Milan, Italy
| | - Luca Maria Sconfienza
- Department of Biomedical Sciences for Health, University of Milan, Milan, Italy.,IRCCS Istituto Ortopedico Galeazzi, Milan, Italy
| |
Collapse
|
42
|
York T, Jenney H, Jones G. Clinician and computer: a study on patient perceptions of artificial intelligence in skeletal radiography. BMJ Health Care Inform 2020; 27:e100233. [PMID: 33187956 PMCID: PMC7668302 DOI: 10.1136/bmjhci-2020-100233] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2020] [Revised: 10/05/2020] [Accepted: 10/15/2020] [Indexed: 11/09/2022] Open
Abstract
BACKGROUND Up to half of all musculoskeletal injuries are investigated with plain radiographs. However, high rates of image interpretation error mean that novel solutions such as artificial intelligence (AI) are being explored. OBJECTIVES To determine patient confidence in clinician-led radiograph interpretation, the perception of AI-assisted interpretation and management, and to identify factors which might influence these views. METHODS A novel questionnaire was distributed to patients attending fracture clinic in a large inner-city teaching hospital. Categorical and Likert scale questions were used to assess participant demographics, daily electronics use, pain score and perceptions towards AI used to assist in interpretation of their radiographs, and guide management. RESULTS 216 questionnaires were included (M=126, F=90). Significantly higher confidence in clinician rather than AI-assisted interpretation was observed (clinician=9.20, SD=1.27 vs AI=7.06, SD=2.13), 95.4% reported favouring clinician over AI-performed interpretation in the event of disagreement.Small positive correlations were observed between younger age/educational achievement and confidence in AI-assistance. Students demonstrated similarly increased confidence (8.43, SD 1.80), and were over-represented in the minority who indicated a preference for AI-assessment over their clinicians (50%). CONCLUSIONS Participant's held the clinician's assessment in the highest regard and expressed a clear preference for it over the hypothetical AI assessment. However, robust confidence scores for the role of AI-assistance in interpreting skeletal imaging suggest patients view the technology favourably.Findings indicate that younger, more educated patients are potentially more comfortable with a role for AI-assistance however further research is needed to overcome the small number of responses on which these observations are based.
Collapse
Affiliation(s)
- Thomas York
- Trauma and Orthopaedics, Imperial College Healthcare NHS Trust, London, UK
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Heloise Jenney
- Trauma and Orthopaedics, Imperial College Healthcare NHS Trust, London, UK
| | - Gareth Jones
- Clinical Senior Lecturer, Trauma and Orthopaedics, Imperial College London, London, UK
| |
Collapse
|
43
|
Ross P, Spates K. Considering the Safety and Quality of Artificial Intelligence in Health Care. Jt Comm J Qual Patient Saf 2020; 46:596-599. [PMID: 32878718 PMCID: PMC7415213 DOI: 10.1016/j.jcjq.2020.08.002] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2020] [Revised: 07/23/2020] [Accepted: 08/04/2020] [Indexed: 12/13/2022]
|
44
|
Moreau JT, Baillet S, Dudley RW. Biased intelligence: on the subjectivity of digital objectivity. BMJ Health Care Inform 2020; 27:e100146. [PMID: 32830107 PMCID: PMC7445351 DOI: 10.1136/bmjhci-2020-100146] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2020] [Accepted: 07/07/2020] [Indexed: 11/03/2022] Open
Affiliation(s)
- Jeremy T Moreau
- Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, Montreal, Québec, Canada
| | - Sylvain Baillet
- Neurology and Neurosurgery, Montreal Neurological Institute and Hospital, Montreal, Québec, Canada
| | - Roy Wr Dudley
- Paediatric Surgery, Division of Neurosurgery, Montreal Children's Hospital, Montreal, Québec, Canada
| |
Collapse
|
45
|
Habli I, Lawton T, Porter Z. Artificial intelligence in health care: accountability and safety. Bull World Health Organ 2020; 98:251-256. [PMID: 32284648 PMCID: PMC7133468 DOI: 10.2471/blt.19.237487] [Citation(s) in RCA: 53] [Impact Index Per Article: 13.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2019] [Revised: 01/07/2020] [Accepted: 01/09/2020] [Indexed: 01/13/2023] Open
Abstract
The prospect of patient harm caused by the decisions made by an artificial intelligence-based clinical tool is something to which current practices of accountability and safety worldwide have not yet adjusted. We focus on two aspects of clinical artificial intelligence used for decision-making: moral accountability for harm to patients; and safety assurance to protect patients against such harm. Artificial intelligence-based tools are challenging the standard clinical practices of assigning blame and assuring safety. Human clinicians and safety engineers have weaker control over the decisions reached by artificial intelligence systems and less knowledge and understanding of precisely how the artificial intelligence systems reach their decisions. We illustrate this analysis by applying it to an example of an artificial intelligence-based system developed for use in the treatment of sepsis. The paper ends with practical suggestions for ways forward to mitigate these concerns. We argue for a need to include artificial intelligence developers and systems safety engineers in our assessments of moral accountability for patient harm. Meanwhile, none of the actors in the model robustly fulfil the traditional conditions of moral accountability for the decisions of an artificial intelligence system. We should therefore update our conceptions of moral accountability in this context. We also need to move from a static to a dynamic model of assurance, accepting that considerations of safety are not fully resolvable during the design of the artificial intelligence system before the system has been deployed.
Collapse
Affiliation(s)
- Ibrahim Habli
- Department of Computer Science, University of York, Deramore Lane, Heslington, York YO10 5GH, England
| | - Tom Lawton
- Bradford Teaching Hospitals NHS Foundation Trust, Bradford, England
| | - Zoe Porter
- Department of Philosophy, University of York, York, England
| |
Collapse
|