1
|
Weber S, Wyszynski M, Godefroid M, Plattfaut R, Niehaves B. How do medical professionals make sense (or not) of AI? A social-media-based computational grounded theory study and an online survey. Comput Struct Biotechnol J 2024; 24:146-159. [PMID: 38434249 PMCID: PMC10904922 DOI: 10.1016/j.csbj.2024.02.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Revised: 02/14/2024] [Accepted: 02/14/2024] [Indexed: 03/05/2024] Open
Abstract
To investigate opinions and attitudes of medical professionals towards adopting AI-enabled healthcare technologies in their daily business, we used a mixed-methods approach. Study 1 employed a qualitative computational grounded theory approach analyzing 181 Reddit threads in the several subreddits of r/medicine. By utilizing an unsupervised machine learning clustering method, we identified three key themes: (1) consequences of AI, (2) physician-AI relationship, and (3) a proposed way forward. In particular Reddit posts related to the first two themes indicated that the medical professionals' fear of being replaced by AI and skepticism toward AI played a major role in the argumentations. Moreover, the results suggest that this fear is driven by little or moderate knowledge about AI. Posts related to the third theme focused on factual discussions about how AI and medicine have to be designed to become broadly adopted in health care. Study 2 quantitatively examined the relationship between the fear of AI, knowledge about AI, and medical professionals' intention to use AI-enabled technologies in more detail. Results based on a sample of 223 medical professionals who participated in the online survey revealed that the intention to use AI technologies increases with increasing knowledge about AI and that this effect is moderated by the fear of being replaced by AI.
Collapse
Affiliation(s)
- Sebastian Weber
- University of Bremen, Digital Public, Bibliothekstr. 1, 28359 Bremen, Germany
| | - Marc Wyszynski
- University of Bremen, Digital Public, Bibliothekstr. 1, 28359 Bremen, Germany
| | - Marie Godefroid
- University of Siegen, Information Systems, Kohlbettstr. 15, 57072 Siegen, Germany
| | - Ralf Plattfaut
- University of Duisburg-Essen, Information Systems and Transformation Management, Universitätsstr. 9, 45141 Essen, Germany
| | - Bjoern Niehaves
- University of Bremen, Digital Public, Bibliothekstr. 1, 28359 Bremen, Germany
| |
Collapse
|
2
|
Yi RC, Gantz HY, Feldman SR. Utilizing artificial intelligence technology with emotional intelligence in clinical office visits. J DERMATOL TREAT 2024; 35:2374500. [PMID: 39042947 DOI: 10.1080/09546634.2024.2374500] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2024] [Accepted: 06/04/2024] [Indexed: 07/25/2024]
Affiliation(s)
- Robin C Yi
- Center for Dermatology Research, Department of Dermatology, Wake Forest University School of Medicine, Winston-Salem, North Carolina
| | - Hannah Y Gantz
- Center for Dermatology Research, Department of Dermatology, Wake Forest University School of Medicine, Winston-Salem, North Carolina
| | - Steven R Feldman
- Center for Dermatology Research, Department of Dermatology, Wake Forest University School of Medicine, Winston-Salem, North Carolina
- Department of Pathology, Wake Forest University School of Medicine, Winston-Salem, North Carolina
- Department of Social Sciences & Health Policy, Wake Forest University School of Medicine, Winston-Salem, North Carolina
| |
Collapse
|
3
|
Wang Y, Han X, Li C, Luo L, Yin Q, Zhang J, Peng G, Shi D, He M. Impact of Gold-Standard Label Errors on Evaluating Performance of Deep Learning Models in Diabetic Retinopathy Screening: Nationwide Real-World Validation Study. J Med Internet Res 2024; 26:e52506. [PMID: 39141915 DOI: 10.2196/52506] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 12/30/2023] [Accepted: 03/22/2024] [Indexed: 08/16/2024] Open
Abstract
BACKGROUND For medical artificial intelligence (AI) training and validation, human expert labels are considered the gold standard that represents the correct answers or desired outputs for a given data set. These labels serve as a reference or benchmark against which the model's predictions are compared. OBJECTIVE This study aimed to assess the accuracy of a custom deep learning (DL) algorithm on classifying diabetic retinopathy (DR) and further demonstrate how label errors may contribute to this assessment in a nationwide DR-screening program. METHODS Fundus photographs from the Lifeline Express, a nationwide DR-screening program, were analyzed to identify the presence of referable DR using both (1) manual grading by National Health Service England-certificated graders and (2) a DL-based DR-screening algorithm with validated good lab performance. To assess the accuracy of labels, a random sample of images with disagreement between the DL algorithm and the labels was adjudicated by ophthalmologists who were masked to the previous grading results. The error rates of labels in this sample were then used to correct the number of negative and positive cases in the entire data set, serving as postcorrection labels. The DL algorithm's performance was evaluated against both pre- and postcorrection labels. RESULTS The analysis included 736,083 images from 237,824 participants. The DL algorithm exhibited a gap between the real-world performance and the lab-reported performance in this nationwide data set, with a sensitivity increase of 12.5% (from 79.6% to 92.5%, P<.001) and a specificity increase of 6.9% (from 91.6% to 98.5%, P<.001). In the random sample, 63.6% (560/880) of negative images and 5.2% (140/2710) of positive images were misclassified in the precorrection human labels. High myopia was the primary reason for misclassifying non-DR images as referable DR images, while laser spots were predominantly responsible for misclassified referable cases. The estimated label error rate for the entire data set was 1.2%. The label correction was estimated to bring about a 12.5% enhancement in the estimated sensitivity of the DL algorithm (P<.001). CONCLUSIONS Label errors based on human image grading, although in a small percentage, can significantly affect the performance evaluation of DL algorithms in real-world DR screening.
Collapse
Affiliation(s)
- Yueye Wang
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China
- School of Optometry, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong)
| | - Xiaotong Han
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China
| | - Cong Li
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China
| | - Lixia Luo
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China
| | - Qiuxia Yin
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China
| | - Jian Zhang
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Visual Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China
| | - Guankai Peng
- Guangzhou Vision Tech Medical Technology Co, Ltd, Guangzhou, China
| | - Danli Shi
- School of Optometry, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong)
- Research Centre for SHARP Vision, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong)
| | - Mingguang He
- School of Optometry, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong)
- Research Centre for SHARP Vision, The Hong Kong Polytechnic University, Kowloon, China (Hong Kong)
- Centre for Eye and Vision Research, Hong Kong, China (Hong Kong)
| |
Collapse
|
4
|
Spoladore D, Stella F, Tosi M, Lorenzini EC, Bettini C. A knowledge-based decision support system to support family doctors in personalizing type-2 diabetes mellitus medical nutrition therapy. Comput Biol Med 2024; 180:109001. [PMID: 39126791 DOI: 10.1016/j.compbiomed.2024.109001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2024] [Revised: 07/12/2024] [Accepted: 08/05/2024] [Indexed: 08/12/2024]
Abstract
BACKGROUND Type-2 Diabetes Mellitus (T2D) is a growing concern worldwide, and family doctors are called to help diabetic patients manage this chronic disease, also with Medical Nutrition Therapy (MNT). However, MNT for Diabetes is usually standardized, while it would be much more effective if tailored to the patient. There is a gap in patient-tailored MNT which, if addressed, could support family doctors in delivering effective recommendations. In this context, decision support systems (DSSs) are valuable tools for physicians to support MNT for T2D patients - as long as DSSs are transparent to humans in their decision-making process. Indeed, the lack of transparency in data-driven DSS might hinder their adoption in clinical practice, thus leaving family physicians to adopt general nutrition guidelines provided by the national healthcare systems. METHOD This work presents a prototypical ontology-based clinical Decision Support System (OnT2D- DSS) aimed at assisting general practice doctors in managing T2D patients, specifically in creating a tailored dietary plan, leveraging clinical expert knowledge. OnT2D-DSS exploits clinical expert knowledge formalized as a domain ontology to identify a patient's phenotype and potential comorbidities, providing personalized MNT recommendations for macro- and micro-nutrient intake. The system can be accessed via a prototypical interface. RESULTS Two preliminary experiments are conducted to assess both the quality and correctness of the inferences provided by the system and the usability and acceptance of the OnT2D-DSS (conducted with nutrition experts and family doctors, respectively). CONCLUSIONS Overall, the system is deemed accurate by the nutrition experts and valuable by the family doctors, with minor suggestions for future improvements collected during the experiments.
Collapse
Affiliation(s)
- Daniele Spoladore
- Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing (STIIMA), National Research Council (Cnr), Lecco, Italy.
| | - Francesco Stella
- Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing (STIIMA), National Research Council (Cnr), Lecco, Italy; Department of Computer Science, University of Milan, Milan, Italy.
| | - Martina Tosi
- Department of Health Sciences, University of Milan, Milan, Italy.
| | | | - Claudio Bettini
- Department of Computer Science, University of Milan, Milan, Italy.
| |
Collapse
|
5
|
Kamel Rahimi A, Pienaar O, Ghadimi M, Canfell OJ, Pole JD, Shrapnel S, van der Vegt AH, Sullivan C. Implementing AI in Hospitals to Achieve a Learning Health System: Systematic Review of Current Enablers and Barriers. J Med Internet Res 2024; 26:e49655. [PMID: 39094106 PMCID: PMC11329852 DOI: 10.2196/49655] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 02/08/2024] [Accepted: 05/22/2024] [Indexed: 08/04/2024] Open
Abstract
BACKGROUND Efforts are underway to capitalize on the computational power of the data collected in electronic medical records (EMRs) to achieve a learning health system (LHS). Artificial intelligence (AI) in health care has promised to improve clinical outcomes, and many researchers are developing AI algorithms on retrospective data sets. Integrating these algorithms with real-time EMR data is rare. There is a poor understanding of the current enablers and barriers to empower this shift from data set-based use to real-time implementation of AI in health systems. Exploring these factors holds promise for uncovering actionable insights toward the successful integration of AI into clinical workflows. OBJECTIVE The first objective was to conduct a systematic literature review to identify the evidence of enablers and barriers regarding the real-world implementation of AI in hospital settings. The second objective was to map the identified enablers and barriers to a 3-horizon framework to enable the successful digital health transformation of hospitals to achieve an LHS. METHODS The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines were adhered to. PubMed, Scopus, Web of Science, and IEEE Xplore were searched for studies published between January 2010 and January 2022. Articles with case studies and guidelines on the implementation of AI analytics in hospital settings using EMR data were included. We excluded studies conducted in primary and community care settings. Quality assessment of the identified papers was conducted using the Mixed Methods Appraisal Tool and ADAPTE frameworks. We coded evidence from the included studies that related to enablers of and barriers to AI implementation. The findings were mapped to the 3-horizon framework to provide a road map for hospitals to integrate AI analytics. RESULTS Of the 1247 studies screened, 26 (2.09%) met the inclusion criteria. In total, 65% (17/26) of the studies implemented AI analytics for enhancing the care of hospitalized patients, whereas the remaining 35% (9/26) provided implementation guidelines. Of the final 26 papers, the quality of 21 (81%) was assessed as poor. A total of 28 enablers was identified; 8 (29%) were new in this study. A total of 18 barriers was identified; 5 (28%) were newly found. Most of these newly identified factors were related to information and technology. Actionable recommendations for the implementation of AI toward achieving an LHS were provided by mapping the findings to a 3-horizon framework. CONCLUSIONS Significant issues exist in implementing AI in health care. Shifting from validating data sets to working with live data is challenging. This review incorporated the identified enablers and barriers into a 3-horizon framework, offering actionable recommendations for implementing AI analytics to achieve an LHS. The findings of this study can assist hospitals in steering their strategic planning toward successful adoption of AI.
Collapse
Affiliation(s)
- Amir Kamel Rahimi
- Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Brisbane, Australia
- Digital Health Cooperative Research Centre, Australian Government, Sydney, Australia
| | - Oliver Pienaar
- The School of Mathematics and Physics, The University of Queensland, Brisbane, Australia
| | - Moji Ghadimi
- The School of Mathematics and Physics, The University of Queensland, Brisbane, Australia
| | - Oliver J Canfell
- Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Brisbane, Australia
- Digital Health Cooperative Research Centre, Australian Government, Sydney, Australia
- Business School, The University of Queensland, Brisbane, Australia
- Department of Nutritional Sciences, Faculty of Life Sciences and Medicine, King's College London, London, United Kingdom
| | - Jason D Pole
- Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Brisbane, Australia
- Dalla Lana School of Public Health, The University of Toronto, Toronto, ON, Canada
- ICES, Toronto, ON, Canada
| | - Sally Shrapnel
- Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Brisbane, Australia
- The School of Mathematics and Physics, The University of Queensland, Brisbane, Australia
| | - Anton H van der Vegt
- Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Brisbane, Australia
| | - Clair Sullivan
- Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Brisbane, Australia
- Metro North Hospital and Health Service, Department of Health, Queensland Government, Brisbane, Australia
| |
Collapse
|
6
|
Petrella RJ. The AI Future of Emergency Medicine. Ann Emerg Med 2024; 84:139-153. [PMID: 38795081 DOI: 10.1016/j.annemergmed.2024.01.031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Revised: 01/23/2024] [Accepted: 01/24/2024] [Indexed: 05/27/2024]
Abstract
In the coming years, artificial intelligence (AI) and machine learning will likely give rise to profound changes in the field of emergency medicine, and medicine more broadly. This article discusses these anticipated changes in terms of 3 overlapping yet distinct stages of AI development. It reviews some fundamental concepts in AI and explores their relation to clinical practice, with a focus on emergency medicine. In addition, it describes some of the applications of AI in disease diagnosis, prognosis, and treatment, as well as some of the practical issues that they raise, the barriers to their implementation, and some of the legal and regulatory challenges they create.
Collapse
Affiliation(s)
- Robert J Petrella
- Emergency Departments, CharterCARE Health Partners, Providence and North Providence, RI; Emergency Department, Boston VA Medical Center, Boston, MA; Emergency Departments, Steward Health Care System, Boston and Methuen, MA; Harvard Medical School, Boston, MA; Department of Chemistry and Chemical Biology, Harvard University, Cambridge, MA; Department of Medicine, Brigham and Women's Hospital, Boston, MA.
| |
Collapse
|
7
|
Maita KC, Avila FR, Torres-Guzman RA, Garcia JP, De Sario Velasquez GD, Borna S, Brown SA, Haider CR, Ho OS, Forte AJ. The usefulness of artificial intelligence in breast reconstruction: a systematic review. Breast Cancer 2024; 31:562-571. [PMID: 38619786 DOI: 10.1007/s12282-024-01582-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 03/30/2024] [Indexed: 04/16/2024]
Abstract
BACKGROUND Artificial Intelligence (AI) offers an approach to predictive modeling. The model learns to determine specific patterns of undesirable outcomes in a dataset. Therefore, a decision-making algorithm can be built based on these patterns to prevent negative results. This systematic review aimed to evaluate the usefulness of AI in breast reconstruction. METHODS A systematic review was conducted in August 2022 following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. MEDLINE, EMBASE, SCOPUS, and Google Scholar online databases were queried to capture all publications studying the use of artificial intelligence in breast reconstruction. RESULTS A total of 23 studies were full text-screened after removing duplicates, and twelve articles fulfilled our inclusion criteria. The Machine Learning algorithms applied for neuropathic pain, lymphedema diagnosis, microvascular abdominal flap failure, donor site complications associated to muscle sparing Transverse Rectus Abdominis flap, surgical complications, financial toxicity, and patient-reported outcomes after breast surgery demonstrated that AI is a helpful tool to accurately predict patient results. In addition, one study used Computer Vision technology to assist in Deep Inferior Epigastric Perforator Artery detection for flap design, considerably reducing the preoperative time compared to manual identification. CONCLUSIONS In breast reconstruction, AI can help the surgeon by optimizing the perioperative patients' counseling to predict negative outcomes, allowing execution of timely interventions and reducing the postoperative burden, which leads to obtaining the most successful results and improving patient satisfaction.
Collapse
Affiliation(s)
- Karla C Maita
- Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL, 32224, USA
| | - Francisco R Avila
- Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL, 32224, USA
| | | | - John P Garcia
- Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL, 32224, USA
| | | | - Sahar Borna
- Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL, 32224, USA
| | - Sally A Brown
- Department of Administration, Mayo Clinic, Jacksonville, FL, USA
| | - Clifton R Haider
- Department of Physiology and Biomedical Engineering, Mayo Clinic, Rochester, MN, USA
| | - Olivia S Ho
- Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL, 32224, USA
| | - Antonio Jorge Forte
- Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL, 32224, USA.
| |
Collapse
|
8
|
Wang Y, Fu W, Zhang Y, Wang D, Gu Y, Wang W, Xu H, Ge X, Ye C, Fang J, Su L, Wang J, He W, Zhang X, Feng R. Constructing and implementing a performance evaluation indicator set for artificial intelligence decision support systems in pediatric outpatient clinics: an observational study. Sci Rep 2024; 14:14482. [PMID: 38914707 PMCID: PMC11196575 DOI: 10.1038/s41598-024-64893-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Accepted: 06/13/2024] [Indexed: 06/26/2024] Open
Abstract
Artificial intelligence (AI) decision support systems in pediatric healthcare have a complex application background. As an AI decision support system (AI-DSS) can be costly, once applied, it is crucial to focus on its performance, interpret its success, and then monitor and update it to ensure ongoing success consistently. Therefore, a set of evaluation indicators was explicitly developed for AI-DSS in pediatric healthcare, enabling continuous and systematic performance monitoring. The study unfolded in two stages. The first stage encompassed establishing the evaluation indicator set through a literature review, a focus group interview, and expert consultation using the Delphi method. In the second stage, weight analysis was conducted. Subjective weights were calculated based on expert opinions through analytic hierarchy process, while objective weights were determined using the entropy weight method. Subsequently, subject and object weights were synthesized to form the combined weight. In the two rounds of expert consultation, the authority coefficients were 0.834 and 0.846, Kendall's coordination coefficient was 0.135 in Round 1 and 0.312 in Round 2. The final evaluation indicator set has three first-class indicators, fifteen second-class indicators, and forty-seven third-class indicators. Indicator I-1(Organizational performance) carries the highest weight, followed by Indicator I-2(Societal performance) and Indicator I-3(User experience performance) in the objective and combined weights. Conversely, 'Societal performance' holds the most weight among the subjective weights, followed by 'Organizational performance' and 'User experience performance'. In this study, a comprehensive and specialized set of evaluation indicators for the AI-DSS in the pediatric outpatient clinic was established, and then implemented. Continuous evaluation still requires long-term data collection to optimize the weight proportions of the established indicators.
Collapse
Affiliation(s)
- Yingwen Wang
- Nursing Department, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Weijia Fu
- Medical Information Center, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Yuejie Zhang
- School of Computer Science, Fudan University, Shanghai, 200438, China
| | - Daoyang Wang
- School of Public, Health Fudan University, Shanghai, 200032, China
| | - Ying Gu
- Nursing Department, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Weibing Wang
- School of Public, Health Fudan University, Shanghai, 200032, China
| | - Hong Xu
- Nephrology Department, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Xiaoling Ge
- Statistical and Data Management Center, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Chengjie Ye
- Medical Information Center, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Jinwu Fang
- School of Public, Health Fudan University, Shanghai, 200032, China
| | - Ling Su
- Statistical and Data Management Center, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Jiayu Wang
- National Health Commission Key Laboratory of Neonatal Diseases (Fudan University), Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Wen He
- Respiratory Department, Children's Hospital of Fudan University, Shanghai, 201102, China
| | - Xiaobo Zhang
- Respiratory Department, Children's Hospital of Fudan University, Shanghai, 201102, China.
| | - Rui Feng
- School of Computer Science, Fudan University, Shanghai, 200438, China.
- School of Computer Science, Fudan University, 2005 Songhu Road, Shanghai, 200438, China.
| |
Collapse
|
9
|
Scott IA, van der Vegt A, Lane P, McPhail S, Magrabi F. Achieving large-scale clinician adoption of AI-enabled decision support. BMJ Health Care Inform 2024; 31:e100971. [PMID: 38816209 PMCID: PMC11141172 DOI: 10.1136/bmjhci-2023-100971] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Accepted: 05/15/2024] [Indexed: 06/01/2024] Open
Abstract
Computerised decision support (CDS) tools enabled by artificial intelligence (AI) seek to enhance accuracy and efficiency of clinician decision-making at the point of care. Statistical models developed using machine learning (ML) underpin most current tools. However, despite thousands of models and hundreds of regulator-approved tools internationally, large-scale uptake into routine clinical practice has proved elusive. While underdeveloped system readiness and investment in AI/ML within Australia and perhaps other countries are impediments, clinician ambivalence towards adopting these tools at scale could be a major inhibitor. We propose a set of principles and several strategic enablers for obtaining broad clinician acceptance of AI/ML-enabled CDS tools.
Collapse
Affiliation(s)
- Ian A Scott
- Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Brisbane, Queensland, Australia
- Centre for Health Services Research, The University of Queensland Faculty of Medicine and Biomedical Sciences, Brisbane, Queensland, Australia
| | - Anton van der Vegt
- Digital Health Centre, The University of Queensland Faculty of Medicine and Biomedical Sciences, Herston, Queensland, Australia
| | - Paul Lane
- Safety, Quality and Innovation, The Prince Charles Hospital, Brisbane, Queensland, Australia
| | - Steven McPhail
- Australian Centre for Health Services Innovation, Queensland University of Technology Faculty of Health, Brisbane, Queensland, Australia
| | - Farah Magrabi
- Macquarie University, Sydney, New South Wales, Australia
| |
Collapse
|
10
|
Kuziemsky CE, Chrimes D, Minshall S, Mannerow M, Lau F. AI Quality Standards in Health Care: Rapid Umbrella Review. J Med Internet Res 2024; 26:e54705. [PMID: 38776538 PMCID: PMC11153979 DOI: 10.2196/54705] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Revised: 04/03/2024] [Accepted: 04/04/2024] [Indexed: 05/25/2024] Open
Abstract
BACKGROUND In recent years, there has been an upwelling of artificial intelligence (AI) studies in the health care literature. During this period, there has been an increasing number of proposed standards to evaluate the quality of health care AI studies. OBJECTIVE This rapid umbrella review examines the use of AI quality standards in a sample of health care AI systematic review articles published over a 36-month period. METHODS We used a modified version of the Joanna Briggs Institute umbrella review method. Our rapid approach was informed by the practical guide by Tricco and colleagues for conducting rapid reviews. Our search was focused on the MEDLINE database supplemented with Google Scholar. The inclusion criteria were English-language systematic reviews regardless of review type, with mention of AI and health in the abstract, published during a 36-month period. For the synthesis, we summarized the AI quality standards used and issues noted in these reviews drawing on a set of published health care AI standards, harmonized the terms used, and offered guidance to improve the quality of future health care AI studies. RESULTS We selected 33 review articles published between 2020 and 2022 in our synthesis. The reviews covered a wide range of objectives, topics, settings, designs, and results. Over 60 AI approaches across different domains were identified with varying levels of detail spanning different AI life cycle stages, making comparisons difficult. Health care AI quality standards were applied in only 39% (13/33) of the reviews and in 14% (25/178) of the original studies from the reviews examined, mostly to appraise their methodological or reporting quality. Only a handful mentioned the transparency, explainability, trustworthiness, ethics, and privacy aspects. A total of 23 AI quality standard-related issues were identified in the reviews. There was a recognized need to standardize the planning, conduct, and reporting of health care AI studies and address their broader societal, ethical, and regulatory implications. CONCLUSIONS Despite the growing number of AI standards to assess the quality of health care AI studies, they are seldom applied in practice. With increasing desire to adopt AI in different health topics, domains, and settings, practitioners and researchers must stay abreast of and adapt to the evolving landscape of health care AI quality standards and apply these standards to improve the quality of their AI studies.
Collapse
Affiliation(s)
| | - Dillon Chrimes
- School of Health Information Science, University of Victoria, Victoria, BC, Canada
| | - Simon Minshall
- School of Health Information Science, University of Victoria, Victoria, BC, Canada
| | | | - Francis Lau
- School of Health Information Science, University of Victoria, Victoria, BC, Canada
| |
Collapse
|
11
|
Pressman SM, Borna S, Gomez-Cabello CA, Haider SA, Haider CR, Forte AJ. Clinical and Surgical Applications of Large Language Models: A Systematic Review. J Clin Med 2024; 13:3041. [PMID: 38892752 PMCID: PMC11172607 DOI: 10.3390/jcm13113041] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2024] [Revised: 05/15/2024] [Accepted: 05/19/2024] [Indexed: 06/21/2024] Open
Abstract
Background: Large language models (LLMs) represent a recent advancement in artificial intelligence with medical applications across various healthcare domains. The objective of this review is to highlight how LLMs can be utilized by clinicians and surgeons in their everyday practice. Methods: A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Six databases were searched to identify relevant articles. Eligibility criteria emphasized articles focused primarily on clinical and surgical applications of LLMs. Results: The literature search yielded 333 results, with 34 meeting eligibility criteria. All articles were from 2023. There were 14 original research articles, four letters, one interview, and 15 review articles. These articles covered a wide variety of medical specialties, including various surgical subspecialties. Conclusions: LLMs have the potential to enhance healthcare delivery. In clinical settings, LLMs can assist in diagnosis, treatment guidance, patient triage, physician knowledge augmentation, and administrative tasks. In surgical settings, LLMs can assist surgeons with documentation, surgical planning, and intraoperative guidance. However, addressing their limitations and concerns, particularly those related to accuracy and biases, is crucial. LLMs should be viewed as tools to complement, not replace, the expertise of healthcare professionals.
Collapse
Affiliation(s)
| | - Sahar Borna
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | | | - Syed Ali Haider
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Clifton R. Haider
- Department of Physiology and Biomedical Engineering, Mayo Clinic, Rochester, MN 55905, USA
| | - Antonio Jorge Forte
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
- Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA
| |
Collapse
|
12
|
McMahon GT. The Risks and Challenges of Artificial Intelligence in Endocrinology. J Clin Endocrinol Metab 2024; 109:e1468-e1471. [PMID: 38471009 DOI: 10.1210/clinem/dgae017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Indexed: 03/14/2024]
Abstract
Artificial intelligence (AI) holds the promise of addressing many of the numerous challenges healthcare faces, which include a growing burden of illness, an increase in chronic health conditions and disabilities due to aging and epidemiological changes, higher demand for health services, overworked and burned-out clinicians, greater societal expectations, and rising health expenditures. While technological advancements in processing power, memory, storage, and the abundance of data have empowered computers to handle increasingly complex tasks with remarkable success, AI introduces a variety of meaningful risks and challenges. Among these are issues related to accuracy and reliability, bias and equity, errors and accountability, transparency, misuse, and privacy of data. As AI systems continue to rapidly integrate into healthcare settings, it is crucial to recognize the inherent risks they bring. These risks demand careful consideration to ensure the responsible and safe deployment of AI in healthcare.
Collapse
Affiliation(s)
- Graham T McMahon
- Accreditation Council for Continuing Medical Education, Chicago, IL 60611, USA
- Department of Medical Education and Division of Endocrinology, Metabolism and Molecular Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL 60611, USA
| |
Collapse
|
13
|
Fangerau H. Artifical intelligence in surgery: ethical considerations in the light of social trends in the perception of health and medicine. EFORT Open Rev 2024; 9:323-328. [PMID: 38726973 PMCID: PMC11099585 DOI: 10.1530/eor-24-0029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/12/2024] Open
Abstract
The use of artificial intelligence (AI) in medicine and surgery is currently predicted to be very promising. However, AI has the potential to change the doctor's role and the doctor-patient relationship. It has the potential to support people's desires for health, along with the potential to nudge or push people to behave in a certain way. To understand these potentials, we must see AI in the light of social developments that have brought about changes in how medicine's role, in a given society, is understood. The trends of 'privatisation of medicine' and 'public-healthisation of the private' are proposed as a contextual backdrop to explain why AI raises ethical concerns different from those previously caused by new medical technologies, and which therefore need to be addressed specifically for AI.
Collapse
Affiliation(s)
- Heiner Fangerau
- Department for the History, Philosophy and Ethics of Medicine, Medical Faculty, Heinrich-Heine University Duesseldorf Centre Health & Society, Moorenstraße 5, Düsseldorf, Germany
| |
Collapse
|
14
|
Newlands R, Bruhn H, Díaz MR, Lip G, Anderson LA, Ramsay C. A stakeholder analysis to prepare for real-world evaluation of integrating artificial intelligent algorithms into breast screening (PREP-AIR study): a qualitative study using the WHO guide. BMC Health Serv Res 2024; 24:569. [PMID: 38698386 PMCID: PMC11067265 DOI: 10.1186/s12913-024-10926-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2024] [Accepted: 03/28/2024] [Indexed: 05/05/2024] Open
Abstract
BACKGROUND The national breast screening programme in the United Kingdom is under pressure due to workforce shortages and having been paused during the COVID-19 pandemic. Artificial intelligence has the potential to transform how healthcare is delivered by improving care processes and patient outcomes. Research on the clinical and organisational benefits of artificial intelligence is still at an early stage, and numerous concerns have been raised around its implications, including patient safety, acceptance, and accountability for decisions. Reforming the breast screening programme to include artificial intelligence is a complex endeavour because numerous stakeholders influence it. Therefore, a stakeholder analysis was conducted to identify relevant stakeholders, explore their views on the proposed reform (i.e., integrating artificial intelligence algorithms into the Scottish National Breast Screening Service for breast cancer detection) and develop strategies for managing 'important' stakeholders. METHODS A qualitative study (i.e., focus groups and interviews, March-November 2021) was conducted using the stakeholder analysis guide provided by the World Health Organisation and involving three Scottish health boards: NHS Greater Glasgow & Clyde, NHS Grampian and NHS Lothian. The objectives included: (A) Identify possible stakeholders (B) Explore stakeholders' perspectives and describe their characteristics (C) Prioritise stakeholders in terms of importance and (D) Develop strategies to manage 'important' stakeholders. Seven stakeholder characteristics were assessed: their knowledge of the targeted reform, position, interest, alliances, resources, power and leadership. RESULTS Thirty-two participants took part from 14 (out of 17 identified) sub-groups of stakeholders. While they were generally supportive of using artificial intelligence in breast screening programmes, some concerns were raised. Stakeholder knowledge, influence and interests in the reform varied. Key advantages mentioned include service efficiency, quicker results and reduced work pressure. Disadvantages included overdiagnosis or misdiagnosis of cancer, inequalities in detection and the self-learning capacity of the algorithms. Five strategies (with considerations suggested by stakeholders) were developed to maintain and improve the support of 'important' stakeholders. CONCLUSIONS Health services worldwide face similar challenges of workforce issues to provide patient care. The findings of this study will help others to learn from Scottish experiences and provide guidance to conduct similar studies targeting healthcare reform. STUDY REGISTRATION researchregistry6579, date of registration: 16/02/2021.
Collapse
Affiliation(s)
- Rumana Newlands
- Health Services Research Unit, University of Aberdeen, Aberdeen, UK.
| | - Hanne Bruhn
- Health Services Research Unit, University of Aberdeen, Aberdeen, UK
| | | | - Gerald Lip
- North East Scotland Breast Screening Programme, NHS Grampian, Aberdeen, UK
| | - Lesley A Anderson
- Centre for Health Data Science, Institute of Applied Health Sciences, University of Aberdeen, Aberdeen, UK
| | - Craig Ramsay
- Health Services Research Unit, University of Aberdeen, Aberdeen, UK
| |
Collapse
|
15
|
Dixon D, Sattar H, Moros N, Kesireddy SR, Ahsan H, Lakkimsetti M, Fatima M, Doshi D, Sadhu K, Junaid Hassan M. Unveiling the Influence of AI Predictive Analytics on Patient Outcomes: A Comprehensive Narrative Review. Cureus 2024; 16:e59954. [PMID: 38854327 PMCID: PMC11161909 DOI: 10.7759/cureus.59954] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/08/2024] [Indexed: 06/11/2024] Open
Abstract
This comprehensive literature review explores the transformative impact of artificial intelligence (AI) predictive analytics on healthcare, particularly in improving patient outcomes regarding disease progression, treatment response, and recovery rates. AI, encompassing capabilities such as learning, problem-solving, and decision-making, is leveraged to predict disease progression, optimize treatment plans, and enhance recovery rates through the analysis of vast datasets, including electronic health records (EHRs), imaging, and genetic data. The utilization of machine learning (ML) and deep learning (DL) techniques in predictive analytics enables personalized medicine by facilitating the early detection of conditions, precision in drug discovery, and the tailoring of treatment to individual patient profiles. Ethical considerations, including data privacy, bias, and accountability, emerge as vital in the responsible implementation of AI in healthcare. The findings underscore the potential of AI predictive analytics in revolutionizing clinical decision-making and healthcare delivery, emphasizing the necessity of ethical guidelines and continuous model validation to ensure its safe and effective use in augmenting human judgment in medical practice.
Collapse
Affiliation(s)
- Diny Dixon
- Medicine, Jubilee Mission Medical College and Research Institute, Thrissur, IND
| | - Hina Sattar
- Medicine, Dow University of Health Sciences, Karachi, PAK
| | - Natalia Moros
- Medicine, Pontifical Javeriana University Medical School, Bogotá, COL
| | | | - Huma Ahsan
- Medicine, Jinnah Postgraduate Medical Centre, Karachi, PAK
| | | | - Madiha Fatima
- Medicine, Fatima Jinnah Medical University, Lahore, PAK
| | - Dhruvi Doshi
- Medicine, Gujarat Cancer Society Medical College, Hospital & Research Centre, Ahmedabad, IND
| | | | | |
Collapse
|
16
|
Esmaeilzadeh P. Challenges and strategies for wide-scale artificial intelligence (AI) deployment in healthcare practices: A perspective for healthcare organizations. Artif Intell Med 2024; 151:102861. [PMID: 38555850 DOI: 10.1016/j.artmed.2024.102861] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2023] [Revised: 03/19/2024] [Accepted: 03/25/2024] [Indexed: 04/02/2024]
Abstract
Healthcare organizations have realized that Artificial intelligence (AI) can provide a competitive edge through personalized patient experiences, improved patient outcomes, early diagnosis, augmented clinician capabilities, enhanced operational efficiencies, or improved medical service accessibility. However, deploying AI-driven tools in the healthcare ecosystem could be challenging. This paper categorizes AI applications in healthcare and comprehensively examines the challenges associated with deploying AI in medical practices at scale. As AI continues to make strides in healthcare, its integration presents various challenges, including production timelines, trust generation, privacy concerns, algorithmic biases, and data scarcity. The paper highlights that flawed business models and wrong workflows in healthcare practices cannot be rectified merely by deploying AI-driven tools. Healthcare organizations should re-evaluate root problems such as misaligned financial incentives (e.g., fee-for-service models), dysfunctional medical workflows (e.g., high rates of patient readmissions), poor care coordination between different providers, fragmented electronic health records systems, and inadequate patient education and engagement models in tandem with AI adoption. This study also explores the need for a cultural shift in viewing AI not as a threat but as an enabler that can enhance healthcare delivery and create new employment opportunities while emphasizing the importance of addressing underlying operational issues. The necessity of investments beyond finance is discussed, emphasizing the importance of human capital, continuous learning, and a supportive environment for AI integration. The paper also highlights the crucial role of clear regulations in building trust, ensuring safety, and guiding the ethical use of AI, calling for coherent frameworks addressing transparency, model accuracy, data quality control, liability, and ethics. Furthermore, this paper underscores the importance of advancing AI literacy within academia to prepare future healthcare professionals for an AI-driven landscape. Through careful navigation and proactive measures addressing these challenges, the healthcare community can harness AI's transformative power responsibly and effectively, revolutionizing healthcare delivery and patient care. The paper concludes with a vision and strategic suggestions for the future of healthcare with AI, emphasizing thoughtful, responsible, and innovative engagement as the pathway to realizing its full potential to unlock immense benefits for healthcare organizations, physicians, nurses, and patients while proactively mitigating risks.
Collapse
Affiliation(s)
- Pouyan Esmaeilzadeh
- Department of Information Systems and Business Analytics, College of Business, Florida International University (FIU), Modesto A. Maidique Campus, 11200 S.W. 8th St, RB 261B, Miami, FL 33199, United States.
| |
Collapse
|
17
|
Wenderott K, Krups J, Luetkens JA, Weigl M. Radiologists' perspectives on the workflow integration of an artificial intelligence-based computer-aided detection system: A qualitative study. APPLIED ERGONOMICS 2024; 117:104243. [PMID: 38306741 DOI: 10.1016/j.apergo.2024.104243] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 12/18/2023] [Accepted: 01/23/2024] [Indexed: 02/04/2024]
Abstract
In healthcare, artificial intelligence (AI) is expected to improve work processes, yet most research focuses on the technical features of AI rather than its real-world clinical implementation. To evaluate the implementation process of an AI-based computer-aided detection system (AI-CAD) for prostate MRI readings, we interviewed German radiologists in a pre-post design. We embedded our findings in the Model of Workflow Integration and the Technology Acceptance Model to analyze workflow effects, facilitators, and barriers. The most prominent barriers were: (i) a time delay in the work process, (ii) additional work steps to be taken, and (iii) an unstable performance of the AI-CAD. Most frequently named facilitators were (i) good self-organization, and (ii) good usability of the software. Our results underline the importance of a holistic approach to AI implementation considering the sociotechnical work system and provide valuable insights into key factors of the successful adoption of AI technologies in work systems.
Collapse
Affiliation(s)
- Katharina Wenderott
- Institute for Patient Safety, University Hospital Bonn, Venusberg-Campus 1, 53127, Bonn, Germany.
| | - Jim Krups
- Institute for Patient Safety, University Hospital Bonn, Venusberg-Campus 1, 53127, Bonn, Germany
| | - Julian A Luetkens
- Department of Diagnostic and Interventional Radiology, University Hospital Bonn, Germany; Quantitative Imaging Lab Bonn (QILaB), University Hospital Bonn, Germany
| | - Matthias Weigl
- Institute for Patient Safety, University Hospital Bonn, Venusberg-Campus 1, 53127, Bonn, Germany
| |
Collapse
|
18
|
Triberti S, Di Fuccio R, Scuotto C, Marsico E, Limone P. "Better than my professor?" How to develop artificial intelligence tools for higher education. Front Artif Intell 2024; 7:1329605. [PMID: 38665370 PMCID: PMC11044698 DOI: 10.3389/frai.2024.1329605] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Accepted: 04/02/2024] [Indexed: 04/28/2024] Open
Abstract
Artificial Intelligence (AI) tools are currently designed and tested in many fields to improve humans' ability to make decisions. One of these fields is higher education. For example, AI-based chatbots ("conversational pedagogical agents") could engage in conversations with students in order to provide timely feedback and responses to questions while the learning process is taking place and to collect data to personalize the delivery of course materials. However, many existent tools are able to perform tasks that human professionals (educators, tutors, professors) could perform, just in a timelier manner. While discussing the possible implementation of AI-based tools in our university's educational programs, we reviewed the current literature and identified a number of capabilities that future AI solutions may feature, in order to improve higher education processes, with a focus on distance higher education. Specifically, we suggest that innovative tools could influence the methodologies by which students approach learning; facilitate connections and information attainment beyond course materials; support the communication with the professor; and, draw from motivation theories to foster learning engagement, in a personalized manner. Future research should explore high-level opportunities represented by AI for higher education, including their effects on learning outcomes and the quality of the learning experience as a whole.
Collapse
Affiliation(s)
- Stefano Triberti
- Department of Psychology and Education, Università Telematica Pegaso, Naples, Italy
| | - Raffaele Di Fuccio
- Department of Psychology and Education, Università Telematica Pegaso, Naples, Italy
| | - Chiara Scuotto
- Department of Psychology and Education, Università Telematica Pegaso, Naples, Italy
- Department of Humanistic Studies, University of Foggia, Foggia, Italy
| | - Emanuele Marsico
- Department of Psychology and Education, Università Telematica Pegaso, Naples, Italy
| | - Pierpaolo Limone
- Department of Psychology and Education, Università Telematica Pegaso, Naples, Italy
| |
Collapse
|
19
|
Sideris K, Weir CR, Schmalfuss C, Hanson H, Pipke M, Tseng PH, Lewis N, Sallam K, Bozkurt B, Hanff T, Schofield R, Larimer K, Kyriakopoulos CP, Taleb I, Brinker L, Curry T, Knecht C, Butler JM, Stehlik J. Artificial intelligence predictive analytics in heart failure: results of the pilot phase of a pragmatic randomized clinical trial. J Am Med Inform Assoc 2024; 31:919-928. [PMID: 38341800 PMCID: PMC10990545 DOI: 10.1093/jamia/ocae017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 12/20/2023] [Accepted: 01/17/2024] [Indexed: 02/13/2024] Open
Abstract
OBJECTIVES We conducted an implementation planning process during the pilot phase of a pragmatic trial, which tests an intervention guided by artificial intelligence (AI) analytics sourced from noninvasive monitoring data in heart failure patients (LINK-HF2). MATERIALS AND METHODS A mixed-method analysis was conducted at 2 pilot sites. Interviews were conducted with 12 of 27 enrolled patients and with 13 participating clinicians. iPARIHS constructs were used for interview construction to identify workflow, communication patterns, and clinician's beliefs. Interviews were transcribed and analyzed using inductive coding protocols to identify key themes. Behavioral response data from the AI-generated notifications were collected. RESULTS Clinicians responded to notifications within 24 hours in 95% of instances, with 26.7% resulting in clinical action. Four implementation themes emerged: (1) High anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring. (2) The AI notifications required a differential and tailored balance of trust and action advice related to role. (3) Clinic experience with other home-based programs influenced utilization. (4) Responding to notifications involved significant effort, including electronic health record (EHR) review, patient contact, and consultation with other clinicians. DISCUSSION Clinician's use of AI data is a function of beliefs regarding the trustworthiness and usefulness of the data, the degree of autonomy in professional roles, and the cognitive effort involved. CONCLUSION The implementation planning analysis guided development of strategies that addressed communication technology, patient education, and EHR integration to reduce clinician and patient burden in the subsequent main randomized phase of the trial. Our results provide important insights into the unique implications of implementing AI analytics into clinical workflow.
Collapse
Affiliation(s)
- Konstantinos Sideris
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Charlene R Weir
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Carsten Schmalfuss
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | - Heather Hanson
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Matt Pipke
- PhysIQ, Inc., Chicago, IL 60563, United States
| | - Po-He Tseng
- PhysIQ, Inc., Chicago, IL 60563, United States
| | - Neil Lewis
- Cardiology Section, Medical Service, Hunter Holmes McGuire Veterans Medical Center, Richmond, VA 23249, United States
- Department of Internal Medicine, Division of Cardiovascular Disease, Virginia Commonwealth University, Richmond, VA 23249, United States
| | - Karim Sallam
- Cardiology Section, Medical Service, VA Palo Alto Health Care System, Palo Alto, CA 94304, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, Stanford University School of Medicine, Stanford, CA 94305, United States
| | - Biykem Bozkurt
- Cardiology Section, Medical Service, Michael E. DeBakey VA Medical Center, Houston, TX 77030, United States
- Section of Cardiology, Department of Medicine, Baylor College of Medicine, Houston, TX 77030, United States
| | - Thomas Hanff
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Richard Schofield
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | | | - Christos P Kyriakopoulos
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Iosif Taleb
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Lina Brinker
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Tempa Curry
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | - Cheri Knecht
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | - Jorie M Butler
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Josef Stehlik
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| |
Collapse
|
20
|
Moy S, Irannejad M, Manning SJ, Farahani M, Ahmed Y, Gao E, Prabhune R, Lorenz S, Mirza R, Klinger C. Patient Perspectives on the Use of Artificial Intelligence in Health Care: A Scoping Review. J Patient Cent Res Rev 2024; 11:51-62. [PMID: 38596349 PMCID: PMC11000703 DOI: 10.17294/2330-0698.2029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/11/2024] Open
Abstract
Purpose Artificial intelligence (AI) technology is being rapidly adopted into many different branches of medicine. Although research has started to highlight the impact of AI on health care, the focus on patient perspectives of AI is scarce. This scoping review aimed to explore the literature on adult patients' perspectives on the use of an array of AI technologies in the health care setting for design and deployment. Methods This scoping review followed Arksey and O'Malley's framework and Preferred Reporting Items for Systematic Reviews and Meta-Analysis for Scoping Reviews (PRISMA-ScR). To evaluate patient perspectives, we conducted a comprehensive literature search using eight interdisciplinary electronic databases, including grey literature. Articles published from 2015 to 2022 that focused on patient views regarding AI technology in health care were included. Thematic analysis was performed on the extracted articles. Results Of the 10,571 imported studies, 37 articles were included and extracted. From the 33 peer-reviewed and 4 grey literature articles, the following themes on AI emerged: (i) Patient attitudes, (ii) Influences on patient attitudes, (iii) Considerations for design, and (iv) Considerations for use. Conclusions Patients are key stakeholders essential to the uptake of AI in health care. The findings indicate that patients' needs and expectations are not fully considered in the application of AI in health care. Therefore, there is a need for patient voices in the development of AI in health care.
Collapse
Affiliation(s)
- Sally Moy
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Mona Irannejad
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | | | - Mehrdad Farahani
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Yomna Ahmed
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Ellis Gao
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Radhika Prabhune
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Suzan Lorenz
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Raza Mirza
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Christopher Klinger
- Translational Research Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada
- National Initiative for the Care of the Elderly, Toronto, Canada
| |
Collapse
|
21
|
Stewart J, Freeman S, Eroglu E, Dumitrascu N, Lu J, Goudie A, Sprivulis P, Akhlaghi H, Tran V, Sanfilippo F, Celenza A, Than M, Fatovich D, Walker K, Dwivedi G. Attitudes towards artificial intelligence in emergency medicine. Emerg Med Australas 2024; 36:252-265. [PMID: 38044755 DOI: 10.1111/1742-6723.14345] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 10/24/2023] [Accepted: 10/30/2023] [Indexed: 12/05/2023]
Abstract
OBJECTIVE To assess Australian and New Zealand emergency clinicians' attitudes towards the use of artificial intelligence (AI) in emergency medicine. METHODS We undertook a qualitative interview-based study based on grounded theory. Participants were recruited through ED internal mailing lists, the Australasian College for Emergency Medicine Bulletin, and the research teams' personal networks. Interviews were transcribed, coded and themes presented. RESULTS Twenty-five interviews were conducted between July 2021 and May 2022. Thematic saturation was achieved after 22 interviews. Most participants were from either Western Australia (52%) or Victoria (16%) and were consultants (96%). More participants reported feeling optimistic (10/25) than neutral (6/25), pessimistic (2/25) or mixed (7/25) towards the use of AI in the ED. A minority expressed scepticism regarding the feasibility or value of implementing AI into the ED. Multiple potential risks and ethical issues were discussed by participants including skill loss from overreliance on AI, algorithmic bias, patient privacy and concerns over liability. Participants also discussed perceived inadequacies in existing information technology systems. Participants felt that AI technologies would be used as decision support tools and not replace the roles of emergency clinicians. Participants were not concerned about the impact of AI on their job security. Most (17/25) participants thought that AI would impact emergency medicine within the next 10 years. CONCLUSIONS Emergency clinicians interviewed were generally optimistic about the use of AI in emergency medicine, so long as it is used as a decision support tool and they maintain the ability to override its recommendations.
Collapse
Affiliation(s)
- Jonathon Stewart
- School of Medicine, The University of Western Australia, Perth, Western Australia, Australia
- Department of Advanced Clinical and Translational Cardiovascular Imaging, Harry Perkins Institute of Medical Research, Perth, Western Australia, Australia
| | - Samuel Freeman
- SensiLab, Monash University, Melbourne, Victoria, Australia
- Department of Emergency Medicine, St Vincent's Hospital Melbourne, Melbourne, Victoria, Australia
| | - Ege Eroglu
- School of Medicine, The University of Notre Dame Australia, Fremantle, Western Australia, Australia
| | - Nicole Dumitrascu
- School of Medicine, The University of Notre Dame Australia, Fremantle, Western Australia, Australia
| | - Juan Lu
- Department of Advanced Clinical and Translational Cardiovascular Imaging, Harry Perkins Institute of Medical Research, Perth, Western Australia, Australia
- Department of Computer Science and Software Engineering, The University of Western Australia, Perth, Western Australia, Australia
| | - Adrian Goudie
- Department of Emergency Medicine, Fiona Stanley Hospital, Perth, Western Australia, Australia
| | - Peter Sprivulis
- Strategy and Governance Division, Western Australia Department of Health, Perth, Western Australia, Australia
| | - Hamed Akhlaghi
- Department of Emergency Medicine, St Vincent's Hospital Melbourne, Melbourne, Victoria, Australia
| | - Viet Tran
- School of Medicine, University of Tasmania, Hobart, Tasmania, Australia
- Department of Emergency Medicine, Royal Hobart Hospital, Hobart, Tasmania, Australia
| | - Frank Sanfilippo
- School of Population and Global Health, The University of Western Australia, Perth, Western Australia, Australia
| | - Antonio Celenza
- School of Medicine, The University of Western Australia, Perth, Western Australia, Australia
- Department of Emergency Medicine, Sir Charles Gairdner Hospital, Perth, Western Australia, Australia
| | - Martin Than
- Department of Emergency Medicine, Christchurch Hospital, Christchurch, New Zealand
| | - Daniel Fatovich
- Emergency Medicine, Royal Perth Hospital, The University of Western Australia, Perth, Western Australia, Australia
- Centre for Clinical Research in Emergency Medicine, Harry Perkins Institute of Medical Research, Perth, Western Australia, Australia
| | - Katie Walker
- School of Clinical Sciences at Monash Health, Monash University, Melbourne, Victoria, Australia
| | - Girish Dwivedi
- School of Medicine, The University of Western Australia, Perth, Western Australia, Australia
- Department of Advanced Clinical and Translational Cardiovascular Imaging, Harry Perkins Institute of Medical Research, Perth, Western Australia, Australia
- Department of Cardiology, Fiona Stanley Hospital, Perth, Western Australia, Australia
| |
Collapse
|
22
|
Alshehri S, Alahmari KA, Alasiry A. A Comprehensive Evaluation of AI-Assisted Diagnostic Tools in ENT Medicine: Insights and Perspectives from Healthcare Professionals. J Pers Med 2024; 14:354. [PMID: 38672981 PMCID: PMC11051468 DOI: 10.3390/jpm14040354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2024] [Revised: 03/18/2024] [Accepted: 03/22/2024] [Indexed: 04/28/2024] Open
Abstract
The integration of Artificial Intelligence (AI) into healthcare has the potential to revolutionize medical diagnostics, particularly in specialized fields such as Ear, Nose, and Throat (ENT) medicine. However, the successful adoption of AI-assisted diagnostic tools in ENT practice depends on the understanding of various factors; these include influences on their effectiveness and acceptance among healthcare professionals. This cross-sectional study aimed to assess the usability and integration of AI tools in ENT practice, determine the clinical impact and accuracy of AI-assisted diagnostics in ENT, measure the trust and confidence of ENT professionals in AI tools, gauge the overall satisfaction and outlook on the future of AI in ENT diagnostics, and identify challenges, limitations, and areas for improvement in AI-assisted ENT diagnostics. A structured online questionnaire was distributed to 600 certified ENT professionals with at least one year of experience in the field. The questionnaire assessed participants' familiarity with AI tools, usability, clinical impact, trust, satisfaction, and identified challenges. A total of 458 respondents completed the questionnaire, resulting in a response rate of 91.7%. The majority of respondents reported familiarity with AI tools (60.7%) and perceived them as generally usable and clinically impactful. However, challenges such as integration with existing systems, user-friendliness, accuracy, and cost were identified. Trust and satisfaction levels varied among participants, with concerns regarding data privacy and support. Geographic and practice setting differences influenced perceptions and experiences. The study highlights the diverse perceptions and experiences of ENT professionals regarding AI-assisted diagnostics. While there is general enthusiasm for these tools, challenges related to integration, usability, trust, and cost need to be addressed for their widespread adoption. These findings provide valuable insights for developers, policymakers, and healthcare providers aiming to enhance the role of AI in ENT practice.
Collapse
Affiliation(s)
- Sarah Alshehri
- Otology and Neurotology, Department of Surgery, College of Medicine, King Khalid University, Abha 61423, Saudi Arabia
| | - Khalid A. Alahmari
- Medical Rehabilitation Sciences, College of Applied Medical Sciences, King Khalid University, Abha 61423, Saudi Arabia;
| | - Areej Alasiry
- Department of Informatics and Computer Systems, College of Computer Science, King Khalid University, Abha 61423, Saudi Arabia;
| |
Collapse
|
23
|
Dupuis M, Delbos L, Rouquette A, Adamsbaum C, Veil R. External validation of an artificial intelligence solution for the detection of elbow fractures and joint effusions in children. Diagn Interv Imaging 2024; 105:104-109. [PMID: 37813759 DOI: 10.1016/j.diii.2023.09.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 09/19/2023] [Accepted: 09/25/2023] [Indexed: 10/11/2023]
Abstract
PURPOSE The purpose of this study was to conduct an external validation of an artificial intelligence (AI) solution for the detection of elbow fractures and joint effusions using radiographs from a real-life cohort of children. MATERIALS AND METHODS This single-center retrospective study was conducted on 758 radiographic sets (1637 images) obtained from consecutive emergency room visits of 712 children (mean age, 7.27 ± 3.97 [standard deviation] years; age range, 7 months and 10 days to 15 years and 10 months), referred for a trauma of the elbow. For each set, fracture and/or effusion detection by eleven senior radiologists (reference standard) and AI solution was recorded. Diagnostic performance of the AI solution was measured via four different approaches: fracture detection (presence/absence of fracture as binary variable), fracture enumeration, fracture localization and lesion detection (fracture and/or a joint effusion used as constructed binary variable). RESULTS The sensitivity of the AI solution for each of the four approaches was >89%. Greatest sensitivity of the AI solution was obtained for lesion detection (95.0%; 95% confidence interval: 92.1-96.9). The specificity of the AI solution ranged between 63% (for lesion detection) and 77% (for fracture detection). For all four approaches, the negative predictive values were >92% and the positive predictive values ranged between 54% (for fracture enumeration and localization) and 73% (for lesion detection). Specificity was lower for plastered children for all approaches (P < 0.001). CONCLUSION The AI solution demonstrates high performances for detecting elbow's fracture and/or joint effusion in children. However, in our context of use, 8% of the radiographic sets ruled-out by the algorithm concerned children with a genuine traumatic elbow lesion.
Collapse
Affiliation(s)
- Michel Dupuis
- AP-HP, Bicêtre Hospital, Pediatric Imaging Department, 94270 Le Kremlin Bicêtre, France
| | - Léo Delbos
- AP-HP, Bicêtre Hospital, Epidemiology and Public Health Department, 94270 Le Kremlin Bicêtre, France
| | - Alexandra Rouquette
- AP-HP, Bicêtre Hospital, Epidemiology and Public Health Department, 94270 Le Kremlin Bicêtre, France
| | - Catherine Adamsbaum
- AP-HP, Bicêtre Hospital, Pediatric Imaging Department, 94270 Le Kremlin Bicêtre, France; Paris Saclay University, Faculté de Médicine, 94270 Le Kremlin Bicêtre, France.
| | - Raphaël Veil
- AP-HP, Bicêtre Hospital, Epidemiology and Public Health Department, 94270 Le Kremlin Bicêtre, France
| |
Collapse
|
24
|
Wang K, Cui H, Zhu Y, Hu X, Hong C, Guo Y, An L, Zhang Q, Liu L. Evaluation of an artificial intelligence-based clinical trial matching system in Chinese patients with hepatocellular carcinoma: a retrospective study. BMC Cancer 2024; 24:246. [PMID: 38388861 PMCID: PMC10885498 DOI: 10.1186/s12885-024-11959-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Accepted: 02/05/2024] [Indexed: 02/24/2024] Open
Abstract
BACKGROUND Artificial intelligence (AI)-assisted clinical trial screening is a promising prospect, although previous matching systems were developed in English, and relevant studies have only been conducted in Western countries. Therefore, we evaluated an AI-based clinical trial matching system (CTMS) that extracts medical data from the electronic health record system and matches them to clinical trials automatically. METHODS This study included 1,053 consecutive inpatients primarily diagnosed with hepatocellular carcinoma who were referred to the liver tumor center of an academic medical center in China between January and December 2019. The eligibility criteria extracted from two clinical trials, patient attributes, and gold standard were decided manually. We evaluated the performance of the CTMS against the established gold standard by measuring the accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and run time required. RESULTS The manual reviewers demonstrated acceptable interrater reliability (Cohen's kappa 0.65-0.88). The performance results for the CTMS were as follows: accuracy, 92.9-98.0%; sensitivity, 51.9-83.5%; specificity, 99.0-99.1%; PPV, 75.7-85.1%; and NPV, 97.4-98.9%. The time required for eligibility determination by the CTMS and manual reviewers was 2 and 150 h, respectively. CONCLUSIONS We found that the CTMS is particularly reliable in excluding ineligible patients in a significantly reduced amount of time. The CTMS excluded ineligible patients for clinical trials with good performance, reducing 98.7% of the work time. Thus, such AI-based systems with natural language processing and machine learning have potential utility in Chinese clinical trials.
Collapse
Affiliation(s)
- Kunyuan Wang
- State Key Laboratory of Organ Failure Research, Guangdong Provincial Key Laboratory of Viral Hepatitis Research, Department of Infectious Diseases and Hepatology Unit, Nanfang Hospital, Southern Medical University, No. 1838, North Guangzhou Avenue, Baiyun District, Guangzhou, China
| | - Hao Cui
- State Key Laboratory of Organ Failure Research, Guangdong Provincial Key Laboratory of Viral Hepatitis Research, Department of Infectious Diseases and Hepatology Unit, Nanfang Hospital, Southern Medical University, No. 1838, North Guangzhou Avenue, Baiyun District, Guangzhou, China
| | - Yun Zhu
- State Key Laboratory of Organ Failure Research, Guangdong Provincial Key Laboratory of Viral Hepatitis Research, Department of Infectious Diseases and Hepatology Unit, Nanfang Hospital, Southern Medical University, No. 1838, North Guangzhou Avenue, Baiyun District, Guangzhou, China
| | - Xiaoyun Hu
- State Key Laboratory of Organ Failure Research, Guangdong Provincial Key Laboratory of Viral Hepatitis Research, Department of Infectious Diseases and Hepatology Unit, Nanfang Hospital, Southern Medical University, No. 1838, North Guangzhou Avenue, Baiyun District, Guangzhou, China
| | - Chang Hong
- State Key Laboratory of Organ Failure Research, Guangdong Provincial Key Laboratory of Viral Hepatitis Research, Department of Infectious Diseases and Hepatology Unit, Nanfang Hospital, Southern Medical University, No. 1838, North Guangzhou Avenue, Baiyun District, Guangzhou, China
| | - Yabing Guo
- State Key Laboratory of Organ Failure Research, Guangdong Provincial Key Laboratory of Viral Hepatitis Research, Department of Infectious Diseases and Hepatology Unit, Nanfang Hospital, Southern Medical University, No. 1838, North Guangzhou Avenue, Baiyun District, Guangzhou, China
| | - Lingyao An
- Research and Development Department, Huimei Technology Co., Ltd, Beijing, China
| | - Qi Zhang
- Research and Development Department, Huimei Technology Co., Ltd, Beijing, China
| | - Li Liu
- State Key Laboratory of Organ Failure Research, Guangdong Provincial Key Laboratory of Viral Hepatitis Research, Department of Infectious Diseases and Hepatology Unit, Nanfang Hospital, Southern Medical University, No. 1838, North Guangzhou Avenue, Baiyun District, Guangzhou, China.
- Big Data Centre, Nanfang Hospital, Southern Medical University, Guangzhou, China.
| |
Collapse
|
25
|
Qiao H, Chen Y, Qian C, Guo Y. Clinical data mining: challenges, opportunities, and recommendations for translational applications. J Transl Med 2024; 22:185. [PMID: 38378565 PMCID: PMC10880222 DOI: 10.1186/s12967-024-05005-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Accepted: 02/18/2024] [Indexed: 02/22/2024] Open
Abstract
Clinical data mining of predictive models offers significant advantages for re-evaluating and leveraging large amounts of complex clinical real-world data and experimental comparison data for tasks such as risk stratification, diagnosis, classification, and survival prediction. However, its translational application is still limited. One challenge is that the proposed clinical requirements and data mining are not synchronized. Additionally, the exotic predictions of data mining are difficult to apply directly in local medical institutions. Hence, it is necessary to incisively review the translational application of clinical data mining, providing an analytical workflow for developing and validating prediction models to ensure the scientific validity of analytic workflows in response to clinical questions. This review systematically revisits the purpose, process, and principles of clinical data mining and discusses the key causes contributing to the detachment from practice and the misuse of model verification in developing predictive models for research. Based on this, we propose a niche-targeting framework of four principles: Clinical Contextual, Subgroup-Oriented, Confounder- and False Positive-Controlled (CSCF), to provide guidance for clinical data mining prior to the model's development in clinical settings. Eventually, it is hoped that this review can help guide future research and develop personalized predictive models to achieve the goal of discovering subgroups with varied remedial benefits or risks and ensuring that precision medicine can deliver its full potential.
Collapse
Affiliation(s)
- Huimin Qiao
- Medical Big Data and Bioinformatics Research Centre, First Affiliated Hospital of Gannan Medical University, Ganzhou, China
| | - Yijing Chen
- School of Public Health and Health Management, Gannan Medical University, Ganzhou, China
| | - Changshun Qian
- School of Information Engineering, Jiangxi University of Science and Technology, Ganzhou, China
| | - You Guo
- Medical Big Data and Bioinformatics Research Centre, First Affiliated Hospital of Gannan Medical University, Ganzhou, China.
- School of Public Health and Health Management, Gannan Medical University, Ganzhou, China.
- School of Information Engineering, Jiangxi University of Science and Technology, Ganzhou, China.
- Ganzhou Key Laboratory of Medical Big Data, Ganzhou, China.
| |
Collapse
|
26
|
Ding H, Simmich J, Vaezipour A, Andrews N, Russell T. Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review. J Am Med Inform Assoc 2024; 31:746-761. [PMID: 38070173 PMCID: PMC10873847 DOI: 10.1093/jamia/ocad222] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2023] [Revised: 11/04/2023] [Accepted: 11/24/2023] [Indexed: 02/18/2024] Open
Abstract
OBJECTIVES Conversational agents (CAs) with emerging artificial intelligence present new opportunities to assist in health interventions but are difficult to evaluate, deterring their applications in the real world. We aimed to synthesize existing evidence and knowledge and outline an evaluation framework for CA interventions. MATERIALS AND METHODS We conducted a systematic scoping review to investigate designs and outcome measures used in the studies that evaluated CAs for health interventions. We then nested the results into an overarching digital health framework proposed by the World Health Organization (WHO). RESULTS The review included 81 studies evaluating CAs in experimental (n = 59), observational (n = 15) trials, and other research designs (n = 7). Most studies (n = 72, 89%) were published in the past 5 years. The proposed CA-evaluation framework includes 4 evaluation stages: (1) feasibility/usability, (2) efficacy, (3) effectiveness, and (4) implementation, aligning with WHO's stepwise evaluation strategy. Across these stages, this article presents the essential evidence of different study designs (n = 8), sample sizes, and main evaluation categories (n = 7) with subcategories (n = 40). The main evaluation categories included (1) functionality, (2) safety and information quality, (3) user experience, (4) clinical and health outcomes, (5) costs and cost benefits, (6) usage, adherence, and uptake, and (7) user characteristics for implementation research. Furthermore, the framework highlighted the essential evaluation areas (potential primary outcomes) and gaps across the evaluation stages. DISCUSSION AND CONCLUSION This review presents a new framework with practical design details to support the evaluation of CA interventions in healthcare research. PROTOCOL REGISTRATION The Open Science Framework (https://osf.io/9hq2v) on March 22, 2021.
Collapse
Affiliation(s)
- Hang Ding
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| | - Joshua Simmich
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| | - Atiyeh Vaezipour
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| | - Nicole Andrews
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
- The Tess Cramond Pain and Research Centre, Metro North Hospital and Health Service, Brisbane, QLD, Australia
- The Occupational Therapy Department, The Royal Brisbane and Women’s Hospital, Metro North Hospital and Health Service, Brisbane, QLD, Australia
| | - Trevor Russell
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| |
Collapse
|
27
|
Fisch U, Kliem P, Grzonka P, Sutter R. Performance of large language models on advocating the management of meningitis: a comparative qualitative study. BMJ Health Care Inform 2024; 31:e100978. [PMID: 38307617 PMCID: PMC10840049 DOI: 10.1136/bmjhci-2023-100978] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2023] [Accepted: 01/15/2024] [Indexed: 02/04/2024] Open
Abstract
OBJECTIVES We aimed to examine the adherence of large language models (LLMs) to bacterial meningitis guidelines using a hypothetical medical case, highlighting their utility and limitations in healthcare. METHODS A simulated clinical scenario of a patient with bacterial meningitis secondary to mastoiditis was presented in three independent sessions to seven publicly accessible LLMs (Bard, Bing, Claude-2, GTP-3.5, GTP-4, Llama, PaLM). Responses were evaluated for adherence to good clinical practice and two international meningitis guidelines. RESULTS A central nervous system infection was identified in 90% of LLM sessions. All recommended imaging, while 81% suggested lumbar puncture. Blood cultures and specific mastoiditis work-up were proposed in only 62% and 38% sessions, respectively. Only 38% of sessions provided the correct empirical antibiotic treatment, while antiviral treatment and dexamethasone were advised in 33% and 24%, respectively. Misleading statements were generated in 52%. No significant correlation was found between LLMs' text length and performance (r=0.29, p=0.20). Among all LLMs, GTP-4 demonstrated the best performance. DISCUSSION Latest LLMs provide valuable advice on differential diagnosis and diagnostic procedures but significantly vary in treatment-specific information for bacterial meningitis when introduced to a realistic clinical scenario. Misleading statements were common, with performance differences attributed to each LLM's unique algorithm rather than output length. CONCLUSIONS Users must be aware of such limitations and performance variability when considering LLMs as a support tool for medical decision-making. Further research is needed to refine these models' comprehension of complex medical scenarios and their ability to provide reliable information.
Collapse
Affiliation(s)
- Urs Fisch
- Department of Neurology, University Hospital Basel, Basel, Switzerland
| | - Paulina Kliem
- Clinic for Intensive Care Medicine, University Hospital Basel, Basel, Switzerland
| | - Pascale Grzonka
- Clinic for Intensive Care Medicine, University Hospital Basel, Basel, Switzerland
| | - Raoul Sutter
- Department of Neurology, University Hospital Basel, Basel, Switzerland
- Clinic for Intensive Care Medicine, University Hospital Basel, Basel, Switzerland
- Medical Faculty, University Basel, Basel, Switzerland
| |
Collapse
|
28
|
Chaurasia A, Namachivayam A, Koca-Ünsal RB, Lee JH. Deep-learning performance in identifying and classifying dental implant systems from dental imaging: a systematic review and meta-analysis. J Periodontal Implant Sci 2024; 54:3-12. [PMID: 37154107 PMCID: PMC10901682 DOI: 10.5051/jpis.2300160008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2022] [Revised: 02/10/2023] [Accepted: 02/21/2023] [Indexed: 05/10/2023] Open
Abstract
Deep learning (DL) offers promising performance in computer vision tasks and is highly suitable for dental image recognition and analysis. We evaluated the accuracy of DL algorithms in identifying and classifying dental implant systems (DISs) using dental imaging. In this systematic review and meta-analysis, we explored the MEDLINE/PubMed, Scopus, Embase, and Google Scholar databases and identified studies published between January 2011 and March 2022. Studies conducted on DL approaches for DIS identification or classification were included, and the accuracy of the DL models was evaluated using panoramic and periapical radiographic images. The quality of the selected studies was assessed using QUADAS-2. This review was registered with PROSPERO (CRDCRD42022309624). From 1,293 identified records, 9 studies were included in this systematic review and meta-analysis. The DL-based implant classification accuracy was no less than 70.75% (95% confidence interval [CI], 65.6%-75.9%) and no higher than 98.19 (95% CI, 97.8%-98.5%). The weighted accuracy was calculated, and the pooled sample size was 46,645, with an overall accuracy of 92.16% (95% CI, 90.8%-93.5%). The risk of bias and applicability concerns were judged as high for most studies, mainly regarding data selection and reference standards. DL models showed high accuracy in identifying and classifying DISs using panoramic and periapical radiographic images. Therefore, DL models are promising prospects for use as decision aids and decision-making tools; however, there are limitations with respect to their application in actual clinical practice.
Collapse
Affiliation(s)
- Akhilanand Chaurasia
- Department of Oral Medicine & Radiology, King George's Medical University, Lucknow, Uttar Pradesh, India
| | - Arunkumar Namachivayam
- Department of Biostatistics, Bapuji Dental College & Hospital, Davengere, Karnataka, India
| | - Revan Birke Koca-Ünsal
- Department of Periodontology, Faculty of Dentistry, University of Kyrenia, Kyrenia, Cyprus
| | - Jae-Hong Lee
- Department of Periodontology, College of Dentistry and Institute of Oral Bioscience, Jeonbuk National University, Jeonju, Korea
- Research Institute of Clinical Medicine of Jeonbuk National University-Biomedical Research Institute of Jeonbuk National University Hospital, Jeonju, Korea.
| |
Collapse
|
29
|
Allen MR, Webb S, Mandvi A, Frieden M, Tai-Seale M, Kallenberg G. Navigating the doctor-patient-AI relationship - a mixed-methods study of physician attitudes toward artificial intelligence in primary care. BMC PRIMARY CARE 2024; 25:42. [PMID: 38281026 PMCID: PMC10821550 DOI: 10.1186/s12875-024-02282-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Accepted: 01/19/2024] [Indexed: 01/29/2024]
Abstract
BACKGROUND Artificial intelligence (AI) is a rapidly advancing field that is beginning to enter the practice of medicine. Primary care is a cornerstone of medicine and deals with challenges such as physician shortage and burnout which impact patient care. AI and its application via digital health is increasingly presented as a possible solution. However, there is a scarcity of research focusing on primary care physician (PCP) attitudes toward AI. This study examines PCP views on AI in primary care. We explore its potential impact on topics pertinent to primary care such as the doctor-patient relationship and clinical workflow. By doing so, we aim to inform primary care stakeholders to encourage successful, equitable uptake of future AI tools. Our study is the first to our knowledge to explore PCP attitudes using specific primary care AI use cases rather than discussing AI in medicine in general terms. METHODS From June to August 2023, we conducted a survey among 47 primary care physicians affiliated with a large academic health system in Southern California. The survey quantified attitudes toward AI in general as well as concerning two specific AI use cases. Additionally, we conducted interviews with 15 survey respondents. RESULTS Our findings suggest that PCPs have largely positive views of AI. However, attitudes often hinged on the context of adoption. While some concerns reported by PCPs regarding AI in primary care focused on technology (accuracy, safety, bias), many focused on people-and-process factors (workflow, equity, reimbursement, doctor-patient relationship). CONCLUSION Our study offers nuanced insights into PCP attitudes towards AI in primary care and highlights the need for primary care stakeholder alignment on key issues raised by PCPs. AI initiatives that fail to address both the technological and people-and-process concerns raised by PCPs may struggle to make an impact.
Collapse
Affiliation(s)
- Matthew R Allen
- Department of Family Medicine, University of California San Diego, La Jolla, CA, 92093, USA.
- Division of Biomedical Informatics, University of California San Diego, La Jolla, CA, 92093, USA.
| | - Sophie Webb
- Department of Family Medicine, University of California San Diego, La Jolla, CA, 92093, USA
| | - Ammar Mandvi
- Department of Family Medicine, University of California San Diego, La Jolla, CA, 92093, USA
| | - Marshall Frieden
- Department of Family Medicine, University of California San Diego, La Jolla, CA, 92093, USA
| | - Ming Tai-Seale
- Department of Family Medicine, University of California San Diego, La Jolla, CA, 92093, USA
| | - Gene Kallenberg
- Department of Family Medicine, University of California San Diego, La Jolla, CA, 92093, USA
| |
Collapse
|
30
|
Tripathi S, Tabari A, Mansur A, Dabbara H, Bridge CP, Daye D. From Machine Learning to Patient Outcomes: A Comprehensive Review of AI in Pancreatic Cancer. Diagnostics (Basel) 2024; 14:174. [PMID: 38248051 PMCID: PMC10814554 DOI: 10.3390/diagnostics14020174] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2023] [Revised: 12/28/2023] [Accepted: 12/29/2023] [Indexed: 01/23/2024] Open
Abstract
Pancreatic cancer is a highly aggressive and difficult-to-detect cancer with a poor prognosis. Late diagnosis is common due to a lack of early symptoms, specific markers, and the challenging location of the pancreas. Imaging technologies have improved diagnosis, but there is still room for improvement in standardizing guidelines. Biopsies and histopathological analysis are challenging due to tumor heterogeneity. Artificial Intelligence (AI) revolutionizes healthcare by improving diagnosis, treatment, and patient care. AI algorithms can analyze medical images with precision, aiding in early disease detection. AI also plays a role in personalized medicine by analyzing patient data to tailor treatment plans. It streamlines administrative tasks, such as medical coding and documentation, and provides patient assistance through AI chatbots. However, challenges include data privacy, security, and ethical considerations. This review article focuses on the potential of AI in transforming pancreatic cancer care, offering improved diagnostics, personalized treatments, and operational efficiency, leading to better patient outcomes.
Collapse
Affiliation(s)
- Satvik Tripathi
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129, USA
- Harvard Medical School, Boston, MA 02115, USA
| | - Azadeh Tabari
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Harvard Medical School, Boston, MA 02115, USA
| | - Arian Mansur
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Harvard Medical School, Boston, MA 02115, USA
| | - Harika Dabbara
- Boston University Chobanian & Avedisian School of Medicine, Boston, MA 02118, USA;
| | - Christopher P. Bridge
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129, USA
- Harvard Medical School, Boston, MA 02115, USA
| | - Dania Daye
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02114, USA; (S.T.); (A.T.); (A.M.); (C.P.B.)
- Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA 02129, USA
- Harvard Medical School, Boston, MA 02115, USA
| |
Collapse
|
31
|
Wenderott K, Krups J, Luetkens JA, Gambashidze N, Weigl M. Prospective effects of an artificial intelligence-based computer-aided detection system for prostate imaging on routine workflow and radiologists' outcomes. Eur J Radiol 2024; 170:111252. [PMID: 38096741 DOI: 10.1016/j.ejrad.2023.111252] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Revised: 11/15/2023] [Accepted: 12/04/2023] [Indexed: 01/16/2024]
Abstract
OBJECTIVES Artificial intelligence (AI) is expected to alleviate the negative consequences of rising case numbers for radiologists. Currently, systematic evaluations of the impact of AI solutions in real-world radiological practice are missing. Our study addresses this gap by investigating the impact of the clinical implementation of an AI-based computer-aided detection system (CAD) for prostate MRI reading on clinicians' workflow, workflow throughput times, workload, and stress. MATERIALS AND METHODS CAD was newly implemented into radiology workflow and accompanied by a prospective pre-post study design. We assessed prostate MRI case readings using standardized work observations and questionnaires. The observation period was three months each in a single department. Workflow throughput times, PI-RADS score, CAD usage and radiologists' self-reported workload and stress were recorded. Linear mixed models were employed for effect identification. RESULTS In data analyses, 91 observed case readings (pre: 50, post: 41) were included. Variation of routine workflow was observed following CAD implementation. A non-significant increase in overall workflow throughput time was associated with CAD implementation (mean 16.99 ± 6.21 vs 18.77 ± 9.69 min, p = .51), along with an increase in diagnostic reading time for high suspicion cases (mean 15.73 ± 4.99 vs 23.07 ± 8.75 min, p = .02). Changes in radiologists' self-reported workload or stress were not found. CONCLUSION Implementation of an AI-based detection aid was associated with lower standardization and no effects over time on radiologists' workload or stress. Expectations of AI decreasing the workload of radiologists were not confirmed by our real-world study. PRE-REGISTRATION German register for clinical trials https://drks.de/; DRKS00027391.
Collapse
Affiliation(s)
| | - Jim Krups
- Institute for Patient Safety, University Hospital Bonn, Germany
| | - Julian A Luetkens
- Department of Radiology and Quantitative Imaging Lab Bonn (QILaB), University Hospital Bonn, Germany
| | | | - Matthias Weigl
- Institute for Patient Safety, University Hospital Bonn, Germany
| |
Collapse
|
32
|
S Alshuhri M, Al-Musawi SG, Al-Alwany AA, Uinarni H, Rasulova I, Rodrigues P, Alkhafaji AT, Alshanberi AM, Alawadi AH, Abbas AH. Artificial intelligence in cancer diagnosis: Opportunities and challenges. Pathol Res Pract 2024; 253:154996. [PMID: 38118214 DOI: 10.1016/j.prp.2023.154996] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Revised: 11/20/2023] [Accepted: 11/27/2023] [Indexed: 12/22/2023]
Abstract
Since cancer is one of the world's top causes of death, early diagnosis is critical to improving patient outcomes. Artificial intelligence (AI) has become a viable technique for cancer diagnosis by using machine learning algorithms to examine large volumes of data for accurate and efficient diagnosis. AI has the potential to alter the way cancer is detected fundamentally. Still, it has several disadvantages, such as requiring a large amount of data, technological limitations, and ethical concerns. This overview looks at the possibilities and restrictions of AI in cancer detection, as well as current applications and possible future developments. We can better understand how to use AI to improve patient outcomes and reduce cancer mortality rates by looking at its potential for cancer detection.
Collapse
Affiliation(s)
- Mohammed S Alshuhri
- Radiology and Medical Imaging Department, College of Applied Medical Sciences, Prince Sattam bin Abdulaziz University, Kharj, Saudi Arabia
| | | | | | - Herlina Uinarni
- Department of Anatomy, School of Medicine and Health Sciences Atma Jaya Catholic University of Indonesia, Indonesia; Radiology department of Pantai Indah Kapuk Hospital Jakarta, Jakarta, Indonesia.
| | - Irodakhon Rasulova
- School of Humanities, Natural & Social Sciences, New Uzbekistan University, 54 Mustaqillik Ave., Tashkent 100007, Uzbekistan; Department of Public Health, Samarkand State Medical University, Amir Temur Street 18, Samarkand, Uzbekistan
| | - Paul Rodrigues
- Department of Computer Engineering, College of Computer Science, King Khalid University, Al-Faraa, Abha, Asir, Kingdom of Saudi Arabia
| | | | - Asim Muhammed Alshanberi
- Department of Community Medicine & Pilgrim Healthcare, Umm Alqura University, Makkah 24382, Saudi Arabia; General Medicine Practice Program, Batterjee Medical College, Jeddah 21442, Saudi Arabia
| | - Ahmed Hussien Alawadi
- College of Technical Engineering, the Islamic University, Najaf, Iraq; College of Technical Engineering, the Islamic University of Al Diwaniyah, Iraq; College of Technical Engineering, the Islamic University of Babylon, Iraq
| | - Ali Hashim Abbas
- College of Technical Engineering, Imam Ja'afar Al-Sadiq University, Al-Muthanna 66002, Iraq
| |
Collapse
|
33
|
Kolasa K, Admassu B, Hołownia-Voloskova M, Kędzior KJ, Poirrier JE, Perni S. Systematic reviews of machine learning in healthcare: a literature review. Expert Rev Pharmacoecon Outcomes Res 2024; 24:63-115. [PMID: 37955147 DOI: 10.1080/14737167.2023.2279107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 10/31/2023] [Indexed: 11/14/2023]
Abstract
INTRODUCTION The increasing availability of data and computing power has made machine learning (ML) a viable approach to faster, more efficient healthcare delivery. METHODS A systematic literature review (SLR) of published SLRs evaluating ML applications in healthcare settings published between1 January 2010 and 27 March 2023 was conducted. RESULTS In total 220 SLRs covering 10,462 ML algorithms were reviewed. The main application of AI in medicine related to the clinical prediction and disease prognosis in oncology and neurology with the use of imaging data. Accuracy, specificity, and sensitivity were provided in 56%, 28%, and 25% SLRs respectively. Internal and external validation was reported in 53% and less than 1% of the cases respectively. The most common modeling approach was neural networks (2,454 ML algorithms), followed by support vector machine and random forest/decision trees (1,578 and 1,522 ML algorithms, respectively). EXPERT OPINION The review indicated considerable reporting gaps in terms of the ML's performance, both internal and external validation. Greater accessibility to healthcare data for developers can ensure the faster adoption of ML algorithms into clinical practice.
Collapse
Affiliation(s)
- Katarzyna Kolasa
- Division of Health Economics and Healthcare Management, Kozminski University, Warsaw, Poland
| | - Bisrat Admassu
- Division of Health Economics and Healthcare Management, Kozminski University, Warsaw, Poland
| | | | | | | | | |
Collapse
|
34
|
He X, Zheng X, Ding H. Existing Barriers Faced by and Future Design Recommendations for Direct-to-Consumer Health Care Artificial Intelligence Apps: Scoping Review. J Med Internet Res 2023; 25:e50342. [PMID: 38109173 PMCID: PMC10758939 DOI: 10.2196/50342] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2023] [Revised: 09/20/2023] [Accepted: 11/28/2023] [Indexed: 12/19/2023] Open
Abstract
BACKGROUND Direct-to-consumer (DTC) health care artificial intelligence (AI) apps hold the potential to bridge the spatial and temporal disparities in health care resources, but they also come with individual and societal risks due to AI errors. Furthermore, the manner in which consumers interact directly with health care AI is reshaping traditional physician-patient relationships. However, the academic community lacks a systematic comprehension of the research overview for such apps. OBJECTIVE This paper systematically delineated and analyzed the characteristics of included studies, identified existing barriers and design recommendations for DTC health care AI apps mentioned in the literature and also provided a reference for future design and development. METHODS This scoping review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews guidelines and was conducted according to Arksey and O'Malley's 5-stage framework. Peer-reviewed papers on DTC health care AI apps published until March 27, 2023, in Web of Science, Scopus, the ACM Digital Library, IEEE Xplore, PubMed, and Google Scholar were included. The papers were analyzed using Braun and Clarke's reflective thematic analysis approach. RESULTS Of the 2898 papers retrieved, 32 (1.1%) covering this emerging field were included. The included papers were recently published (2018-2023), and most (23/32, 72%) were from developed countries. The medical field was mostly general practice (8/32, 25%). In terms of users and functionalities, some apps were designed solely for single-consumer groups (24/32, 75%), offering disease diagnosis (14/32, 44%), health self-management (8/32, 25%), and health care information inquiry (4/32, 13%). Other apps connected to physicians (5/32, 16%), family members (1/32, 3%), nursing staff (1/32, 3%), and health care departments (2/32, 6%), generally to alert these groups to abnormal conditions of consumer users. In addition, 8 barriers and 6 design recommendations related to DTC health care AI apps were identified. Some more subtle obstacles that are particularly worth noting and corresponding design recommendations in consumer-facing health care AI systems, including enhancing human-centered explainability, establishing calibrated trust and addressing overtrust, demonstrating empathy in AI, improving the specialization of consumer-grade products, and expanding the diversity of the test population, were further discussed. CONCLUSIONS The booming DTC health care AI apps present both risks and opportunities, which highlights the need to explore their current status. This paper systematically summarized and sorted the characteristics of the included studies, identified existing barriers faced by, and made future design recommendations for such apps. To the best of our knowledge, this is the first study to systematically summarize and categorize academic research on these apps. Future studies conducting the design and development of such systems could refer to the results of this study, which is crucial to improve the health care services provided by DTC health care AI apps.
Collapse
Affiliation(s)
- Xin He
- School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China
| | - Xi Zheng
- School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China
| | - Huiyuan Ding
- School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China
| |
Collapse
|
35
|
Schneidereith TA, Thibault J. The Basics of Artificial Intelligence in Nursing: Fundamentals and Recommendations for Educators. J Nurs Educ 2023; 62:716-720. [PMID: 38049301 DOI: 10.3928/01484834-20231006-03] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/06/2023]
Abstract
BACKGROUND Artificial intelligence (AI) offers exciting possibilities; however, AI is a double-edged sword. The adoption of this technology offers many benefits but also presents risks to academic integrity and appropriately prepared graduates. Many of today's nurse educators are from generations that are unlikely to possess an understanding of AI. This article provides fundamental knowledge needed to understand the current state of AI in nursing and offers recommendations to nurse educators on ways to responsibly incorporate AI technologies into nursing curricula. METHOD AI literature from PubMed, CINAHL, and Google Scholar was reviewed and synthesized. RESULTS Definitions, explanations, and applications to nursing education are outlined. Recommendations are made for AI implementation, along with ideas to avoid potential AI-enabled plagiarism and academic dishonesty. CONCLUSION As professionals, nurse educators should understand the basics of AI and be able to judge the appropriateness of integration and also recognize opportunities to embrace future application. [J Nurs Educ. 2023;62(12):716-720.].
Collapse
|
36
|
Chekmeyan M, Baccei SJ, Garwood ER. Cross-Check QA: A Quality Assurance Workflow to Prevent Missed Diagnoses by Alerting Inadvertent Discordance Between the Radiologist and Artificial Intelligence in the Interpretation of High-Acuity CT Scans. J Am Coll Radiol 2023; 20:1225-1230. [PMID: 37423347 DOI: 10.1016/j.jacr.2023.06.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 06/02/2023] [Accepted: 06/09/2023] [Indexed: 07/11/2023]
Abstract
PURPOSE The aim of this study was to implement and evaluate a quality assurance (QA) workflow that leverages natural language processing to rapidly resolve inadvertent discordance between radiologists and an artificial intelligence (AI) decision support system (DSS) in the interpretation of high-acuity CT studies when the radiologist does not engage with AI DSS output. METHODS All consecutive high-acuity adult CT examinations performed in a health system between March 1, 2020, and September 20, 2022, were interpreted alongside an AI DSS (Aidoc) for intracranial hemorrhage, cervical spine fracture, and pulmonary embolus. CT studies were flagged for this QA workflow if they met three criteria: (1) negative results by radiologist report, (2) a high probability of positive results by the AI DSS, and (3) unviewed AI DSS output. In these cases, an automated e-mail notification was sent to our quality team. If discordance was confirmed on secondary review-an initially missed diagnosis-addendum and communication documentation was performed. RESULTS Of 111,674 high-acuity CT examinations interpreted alongside the AI DSS over this 2.5-year time period, the frequency of missed diagnoses (intracranial hemorrhage, pulmonary embolus, and cervical spine fracture) uncovered by this workflow was 0.02% (n = 26). Of 12,412 CT studies prioritized as depicting positive findings by the AI DSS, 0.4% (n = 46) were discordant, unengaged, and flagged for QA. Among these discordant cases, 57% (26 of 46) were determined to be true positives. Addendum and communication documentation was performed within 24 hours of the initial report signing in 85% of these cases. CONCLUSIONS Inadvertent discordance between radiologists and the AI DSS occurred in a small number of cases. This QA workflow leveraged natural language processing to rapidly detect, notify, and resolve these discrepancies and prevent potential missed diagnoses.
Collapse
Affiliation(s)
| | - Steven J Baccei
- Professor, Vice-Chair, Quality, Safety, and Process Improvement, and Interim Co-CMO, UMass Memorial Medical Center and Department of Radiology, UMass Chan Medical School, Worcester, Massachusetts
| | - Elisabeth R Garwood
- Assistant Professor and Director of Radiology AI and Clinical Innovation, Department of Radiology, UMass Chan Medical School, Worcester, Massachusetts
| |
Collapse
|
37
|
Lux TJ, Saßmannshausen Z, Kafetzis I, Sodmann P, Herold K, Sudarevic B, Schmitz R, Zoller WG, Meining A, Hann A. Assisted documentation as a new focus for artificial intelligence in endoscopy: the precedent of reliable withdrawal time and image reporting. Endoscopy 2023; 55:1118-1123. [PMID: 37399844 PMCID: PMC11321719 DOI: 10.1055/a-2122-1671] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/28/2022] [Accepted: 06/30/2023] [Indexed: 07/05/2023]
Abstract
BACKGROUND : Reliable documentation is essential for maintaining quality standards in endoscopy; however, in clinical practice, report quality varies. We developed an artificial intelligence (AI)-based prototype for the measurement of withdrawal and intervention times, and automatic photodocumentation. METHOD: A multiclass deep learning algorithm distinguishing different endoscopic image content was trained with 10 557 images (1300 examinations, nine centers, four processors). Consecutively, the algorithm was used to calculate withdrawal time (AI prediction) and extract relevant images. Validation was performed on 100 colonoscopy videos (five centers). The reported and AI-predicted withdrawal times were compared with video-based measurement; photodocumentation was compared for documented polypectomies. RESULTS: Video-based measurement in 100 colonoscopies revealed a median absolute difference of 2.0 minutes between the measured and reported withdrawal times, compared with 0.4 minutes for AI predictions. The original photodocumentation represented the cecum in 88 examinations compared with 98/100 examinations for the AI-generated documentation. For 39/104 polypectomies, the examiners' photographs included the instrument, compared with 68 for the AI images. Lastly, we demonstrated real-time capability (10 colonoscopies). CONCLUSION : Our AI system calculates withdrawal time, provides an image report, and is real-time ready. After further validation, the system may improve standardized reporting, while decreasing the workload created by routine documentation.
Collapse
Affiliation(s)
- Thomas J. Lux
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
| | - Zita Saßmannshausen
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
| | - Ioannis Kafetzis
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
| | - Philipp Sodmann
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
| | - Katja Herold
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
| | - Boban Sudarevic
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
- Department of Internal Medicine and Gastroenterology, Katharinenhospital,
Stuttgart, Germany
| | - Rüdiger Schmitz
- Department for Interdisciplinary Endoscopy; Department of Internal Medicine I;
and Department of Computational Neuroscience, University Hospital Hamburg - Eppendorf,
Hamburg, Germany
| | - Wolfram G. Zoller
- Department of Internal Medicine and Gastroenterology, Katharinenhospital,
Stuttgart, Germany
| | - Alexander Meining
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
| | - Alexander Hann
- Interventional and Experimental Endoscopy (InExEn), Internal Medicine II,
University Hospital Würzburg, Würzburg, Germany
| |
Collapse
|
38
|
Liao X, Yao C, Zhang J, Liu LZ. Recent advancement in integrating artificial intelligence and information technology with real-world data for clinical decision-making in China: A scoping review. J Evid Based Med 2023; 16:534-546. [PMID: 37772921 DOI: 10.1111/jebm.12549] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Accepted: 08/31/2023] [Indexed: 09/30/2023]
Abstract
OBJECTIVE Striking innovations and advancements have been achieved with the use of artificial intelligence and healthcare information technology being integrated into clinical real-world data. The current scoping review aimed to provide an overview of the current status of artificial intelligence-/information technology-based clinical decision support tools in China. METHODS PubMed/MEDLINE, Embase, China National Knowledge Internet, and Wanfang data were searched for both English and Chinese literature. The gray literature search was conducted for commercially available tools. Original studies that focused on clinical decision support tools driven by artificial intelligence or information technology in China and were published between 2010 and February 2022 were included. Information extracted from each article was further synthesized by themes based on three types of clinical decision-making. RESULTS A total of 37 peer-reviewed publications and 13 commercially available tools were included in the final analysis. Among them, 32.0% were developed for disease diagnosis, 54.0% for risk prediction and classification, and 14.0% for disease management. Chronic diseases were the most popular therapeutic areas of exploration, with particular emphasis on cardiovascular and cerebrovascular diseases. Single-center electronic medical records were the mainstream data sources leveraged to inform clinical decision-making, with internal validation being predominately used for model evaluation. CONCLUSIONS To effectively promote the extensive use of real-world data and drive a paradigm shift in clinical decision-making in China, multidisciplinary collaboration of key stakeholders is urgently needed.
Collapse
Affiliation(s)
- Xiwen Liao
- Peking University Clinical Research Institute, Peking University First Hospital, Beijing, China
| | - Chen Yao
- Peking University Clinical Research Institute, Peking University First Hospital, Beijing, China
- Hainan Institute of Real World Data, Qionghai, Hainan, China
| | - Jun Zhang
- Center for Observational and Real-world Evidence (CORE), MSD R&D (China) Co., Ltd., Beijing, China
| | - Larry Z Liu
- Center for Observational and Real-world Evidence (CORE), Merck & Co Inc, Rahway, Rahway, New Jersey, USA
- Department of Population Health Sciences, Weill Cornell Medical College, New York City, New York, USA
| |
Collapse
|
39
|
Pagano S, Holzapfel S, Kappenschneider T, Meyer M, Maderbacher G, Grifka J, Holzapfel DE. Arthrosis diagnosis and treatment recommendations in clinical practice: an exploratory investigation with the generative AI model GPT-4. J Orthop Traumatol 2023; 24:61. [PMID: 38015298 PMCID: PMC10684473 DOI: 10.1186/s10195-023-00740-4] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Accepted: 11/05/2023] [Indexed: 11/29/2023] Open
Abstract
BACKGROUND The spread of artificial intelligence (AI) has led to transformative advancements in diverse sectors, including healthcare. Specifically, generative writing systems have shown potential in various applications, but their effectiveness in clinical settings has been barely investigated. In this context, we evaluated the proficiency of ChatGPT-4 in diagnosing gonarthrosis and coxarthrosis and recommending appropriate treatments compared with orthopaedic specialists. METHODS A retrospective review was conducted using anonymized medical records of 100 patients previously diagnosed with either knee or hip arthrosis. ChatGPT-4 was employed to analyse these historical records, formulating both a diagnosis and potential treatment suggestions. Subsequently, a comparative analysis was conducted to assess the concordance between the AI's conclusions and the original clinical decisions made by the physicians. RESULTS In diagnostic evaluations, ChatGPT-4 consistently aligned with the conclusions previously drawn by physicians. In terms of treatment recommendations, there was an 83% agreement between the AI and orthopaedic specialists. The therapeutic concordance was verified by the calculation of a Cohen's Kappa coefficient of 0.580 (p < 0.001). This indicates a moderate-to-good level of agreement. In recommendations pertaining to surgical treatment, the AI demonstrated a sensitivity and specificity of 78% and 80%, respectively. Multivariable logistic regression demonstrated that the variables reduced quality of life (OR 49.97, p < 0.001) and start-up pain (OR 12.54, p = 0.028) have an influence on ChatGPT-4's recommendation for a surgery. CONCLUSION This study emphasises ChatGPT-4's notable potential in diagnosing conditions such as gonarthrosis and coxarthrosis and in aligning its treatment recommendations with those of orthopaedic specialists. However, it is crucial to acknowledge that AI tools such as ChatGPT-4 are not meant to replace the nuanced expertise and clinical judgment of seasoned orthopaedic surgeons, particularly in complex decision-making scenarios regarding treatment indications. Due to the exploratory nature of the study, further research with larger patient populations and more complex diagnoses is necessary to validate the findings and explore the broader potential of AI in healthcare. LEVEL OF EVIDENCE Level III evidence.
Collapse
Affiliation(s)
- Stefano Pagano
- Department of Orthopaedic Surgery, University of Regensburg, Asklepios Klinikum, Bad Abbach, Germany.
| | - Sabrina Holzapfel
- Department of Neonatology, University Children's Hospital Regensburg, Hospital St. Hedwig of the Order of St. John, University of Regensburg, Regensburg, Germany
| | - Tobias Kappenschneider
- Department of Orthopaedic Surgery, University of Regensburg, Asklepios Klinikum, Bad Abbach, Germany
| | - Matthias Meyer
- Department of Orthopaedic Surgery, University of Regensburg, Asklepios Klinikum, Bad Abbach, Germany
| | - Günther Maderbacher
- Department of Orthopaedic Surgery, University of Regensburg, Asklepios Klinikum, Bad Abbach, Germany
| | - Joachim Grifka
- Department of Orthopaedic Surgery, University of Regensburg, Asklepios Klinikum, Bad Abbach, Germany
| | - Dominik Emanuel Holzapfel
- Department of Orthopaedic Surgery, University of Regensburg, Asklepios Klinikum, Bad Abbach, Germany
| |
Collapse
|
40
|
Susanto AP, Lyell D, Widyantoro B, Berkovsky S, Magrabi F. Effects of machine learning-based clinical decision support systems on decision-making, care delivery, and patient outcomes: a scoping review. J Am Med Inform Assoc 2023; 30:2050-2063. [PMID: 37647865 PMCID: PMC10654852 DOI: 10.1093/jamia/ocad180] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Revised: 08/01/2023] [Accepted: 08/23/2023] [Indexed: 09/01/2023] Open
Abstract
OBJECTIVE This study aims to summarize the research literature evaluating machine learning (ML)-based clinical decision support (CDS) systems in healthcare settings. MATERIALS AND METHODS We conducted a review in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Review). Four databases, including PubMed, Medline, Embase, and Scopus were searched for studies published from January 2016 to April 2021 evaluating the use of ML-based CDS in clinical settings. We extracted the study design, care setting, clinical task, CDS task, and ML method. The level of CDS autonomy was examined using a previously published 3-level classification based on the division of clinical tasks between the clinician and CDS; effects on decision-making, care delivery, and patient outcomes were summarized. RESULTS Thirty-two studies evaluating the use of ML-based CDS in clinical settings were identified. All were undertaken in developed countries and largely in secondary and tertiary care settings. The most common clinical tasks supported by ML-based CDS were image recognition and interpretation (n = 12) and risk assessment (n = 9). The majority of studies examined assistive CDS (n = 23) which required clinicians to confirm or approve CDS recommendations for risk assessment in sepsis and for interpreting cancerous lesions in colonoscopy. Effects on decision-making, care delivery, and patient outcomes were mixed. CONCLUSION ML-based CDS are being evaluated in many clinical areas. There remain many opportunities to apply and evaluate effects of ML-based CDS on decision-making, care delivery, and patient outcomes, particularly in resource-constrained settings.
Collapse
Affiliation(s)
- Anindya Pradipta Susanto
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
- Faculty of Medicine, Universitas Indonesia, Jakarta, DKI Jakarta 10430, Indonesia
| | - David Lyell
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
| | - Bambang Widyantoro
- Faculty of Medicine, Universitas Indonesia, Jakarta, DKI Jakarta 10430, Indonesia
- National Cardiovascular Center Harapan Kita Hospital, Jakarta, DKI Jakarta 11420, Indonesia
| | - Shlomo Berkovsky
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
| | - Farah Magrabi
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
| |
Collapse
|
41
|
Xu J, Xu HL, Cao YN, Huang Y, Gao S, Wu QJ, Gong TT. The performance of deep learning on thyroid nodule imaging predicts thyroid cancer: A systematic review and meta-analysis of epidemiological studies with independent external test sets. Diabetes Metab Syndr 2023; 17:102891. [PMID: 37907027 DOI: 10.1016/j.dsx.2023.102891] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/22/2023] [Revised: 10/06/2023] [Accepted: 10/15/2023] [Indexed: 11/02/2023]
Abstract
BACKGROUND AND AIMS It is still controversial whether deep learning (DL) systems add accuracy to thyroid nodule imaging classification based on the recent available evidence. We conducted this study to analyze the current evidence of DL in thyroid nodule imaging diagnosis in both internal and external test sets. METHODS Until the end of December 2022, PubMed, IEEE, Embase, Web of Science, and the Cochrane Library were searched. We included primary epidemiological studies using externally validated DL techniques in image-based thyroid nodule appraisal. This systematic review was registered on PROSPERO (CRD42022362892). RESULTS We evaluated evidence from 17 primary epidemiological studies using externally validated DL techniques in image-based thyroid nodule appraisal. Fourteen studies were deemed eligible for meta-analysis. The pooled sensitivity, specificity, and area under the curve (AUC) of these DL algorithms were 0.89 (95% confidence interval 0.87-0.90), 0.84 (0.82-0.86), and 0.93 (0.91-0.95), respectively. For the internal validation set, the pooled sensitivity, specificity, and AUC were 0.91 (0.89-0.93), 0.88 (0.85-0.91), and 0.96 (0.93-0.97), respectively. In the external validation set, the pooled sensitivity, specificity, and AUC were 0.87 (0.85-0.89), 0.81 (0.77-0.83), and 0.91 (0.88-0.93), respectively. Notably, in subgroup analyses, DL algorithms still demonstrated exceptional diagnostic validity. CONCLUSIONS Current evidence suggests DL-based imaging shows diagnostic performances comparable to clinicians for differentiating thyroid nodules in both the internal and external test sets.
Collapse
Affiliation(s)
- Jin Xu
- Department of Obstetrics and Gynecology, Shengjing Hospital of China Medical University, Shenyang, China
| | - He-Li Xu
- Department of Clinical Epidemiology, Shengjing Hospital of China Medical University, Shenyang, China
| | - Yi-Ning Cao
- Department of Obstetrics and Gynecology, Shengjing Hospital of China Medical University, Shenyang, China; Department of Clinical Epidemiology, Shengjing Hospital of China Medical University, Shenyang, China
| | - Ying Huang
- Department of Ultrasound, Shengjing Hospital of China Medical University, Shenyang, China
| | - Song Gao
- Department of Obstetrics and Gynecology, Shengjing Hospital of China Medical University, Shenyang, China
| | - Qi-Jun Wu
- Department of Obstetrics and Gynecology, Shengjing Hospital of China Medical University, Shenyang, China; Department of Clinical Epidemiology, Shengjing Hospital of China Medical University, Shenyang, China; Key Laboratory of Reproductive and Genetic Medicine (China Medical University), National Health Commission, Shenyang, China.
| | - Ting-Ting Gong
- Department of Obstetrics and Gynecology, Shengjing Hospital of China Medical University, Shenyang, China.
| |
Collapse
|
42
|
Hummelsberger P, Koch TK, Rauh S, Dorn J, Lermer E, Raue M, Hudecek MFC, Schicho A, Colak E, Ghassemi M, Gaube S. Insights on the Current State and Future Outlook of AI in Health Care: Expert Interview Study. JMIR AI 2023; 2:e47353. [PMID: 38875571 PMCID: PMC11041415 DOI: 10.2196/47353] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Revised: 07/06/2023] [Accepted: 08/01/2023] [Indexed: 06/16/2024]
Abstract
BACKGROUND Artificial intelligence (AI) is often promoted as a potential solution for many challenges health care systems face worldwide. However, its implementation in clinical practice lags behind its technological development. OBJECTIVE This study aims to gain insights into the current state and prospects of AI technology from the stakeholders most directly involved in its adoption in the health care sector whose perspectives have received limited attention in research to date. METHODS For this purpose, the perspectives of AI researchers and health care IT professionals in North America and Western Europe were collected and compared for profession-specific and regional differences. In this preregistered, mixed methods, cross-sectional study, 23 experts were interviewed using a semistructured guide. Data from the interviews were analyzed using deductive and inductive qualitative methods for the thematic analysis along with topic modeling to identify latent topics. RESULTS Through our thematic analysis, four major categories emerged: (1) the current state of AI systems in health care, (2) the criteria and requirements for implementing AI systems in health care, (3) the challenges in implementing AI systems in health care, and (4) the prospects of the technology. Experts discussed the capabilities and limitations of current AI systems in health care in addition to their prevalence and regional differences. Several criteria and requirements deemed necessary for the successful implementation of AI systems were identified, including the technology's performance and security, smooth system integration and human-AI interaction, costs, stakeholder involvement, and employee training. However, regulatory, logistical, and technical issues were identified as the most critical barriers to an effective technology implementation process. In the future, our experts predicted both various threats and many opportunities related to AI technology in the health care sector. CONCLUSIONS Our work provides new insights into the current state, criteria, challenges, and outlook for implementing AI technology in health care from the perspective of AI researchers and IT professionals in North America and Western Europe. For the full potential of AI-enabled technologies to be exploited and for them to contribute to solving current health care challenges, critical implementation criteria must be met, and all groups involved in the process must work together.
Collapse
Affiliation(s)
- Pia Hummelsberger
- LMU Center for Leadership and People Management, Department of Psychology, LMU Munich, Munich, Germany
| | - Timo K Koch
- LMU Center for Leadership and People Management, Department of Psychology, LMU Munich, Munich, Germany
- Department of Psychology, LMU Munich, Munich, Germany
| | - Sabrina Rauh
- LMU Center for Leadership and People Management, Department of Psychology, LMU Munich, Munich, Germany
| | - Julia Dorn
- LMU Center for Leadership and People Management, Department of Psychology, LMU Munich, Munich, Germany
| | - Eva Lermer
- LMU Center for Leadership and People Management, Department of Psychology, LMU Munich, Munich, Germany
- Department of Business Psychology, Technical University of Applied Sciences Augsburg, Augsburg, Germany
| | - Martina Raue
- MIT AgeLab, Massachusetts Institute of Technology, Cambridge, MA, United States
| | - Matthias F C Hudecek
- Department of Experimental Psychology, University of Regensburg, Regensburg, Germany
| | - Andreas Schicho
- Department of Radiology, University Hospital Regensburg, Regensburg, Germany
| | - Errol Colak
- Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, ON, Canada
- Department of Medical Imaging, St. Michael's Hospital, Unity Health Toronto, Toronto, ON, Canada
- Department of Medical Imaging, Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Marzyeh Ghassemi
- Electrical Engineering and Computer Science, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, United States
- Vector Institute, Toronto, ON, Canada
| | - Susanne Gaube
- UCL Global Business School for Health, University College London, London, United Kingdom
| |
Collapse
|
43
|
Cresswell K, Rigby M, Magrabi F, Scott P, Brender J, Craven CK, Wong ZSY, Kukhareva P, Ammenwerth E, Georgiou A, Medlock S, De Keizer NF, Nykänen P, Prgomet M, Williams R. The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision. Health Policy 2023; 136:104889. [PMID: 37579545 DOI: 10.1016/j.healthpol.2023.104889] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 08/04/2023] [Indexed: 08/16/2023]
Abstract
Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.
Collapse
Affiliation(s)
- Kathrin Cresswell
- The University of Edinburgh, Usher Institute, Edinburgh, United Kingdom.
| | - Michael Rigby
- Keele University, School of Social, Political and Global Studies and School of Primary, Community and Social Care, Keele, United Kingdom
| | - Farah Magrabi
- Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
| | - Philip Scott
- University of Wales Trinity Saint David, Swansea, United Kingdom
| | - Jytte Brender
- Department of Health Science and Technology, Aalborg University, Aalborg, Denmark
| | - Catherine K Craven
- University of Texas Health Science Center at San Antonio, San Antonio, TX, United States
| | - Zoie Shui-Yee Wong
- St. Luke's International University, Graduate School of Public Health, Tokyo, Japan
| | - Polina Kukhareva
- Department of Biomedical Informatics, University of Utah, United States of America
| | - Elske Ammenwerth
- UMIT TIROL, Private University for Health Sciences and Health Informatics, Institute of Medical Informatics, Hall in Tirol, Austria
| | - Andrew Georgiou
- Macquarie University, Australian Institute of Health Innovation, Sydney, Australia
| | - Stephanie Medlock
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Digital Health and Quality of Care Amsterdam, the Netherlands
| | - Nicolette F De Keizer
- Amsterdam UMC location University of Amsterdam, Department of Medical Informatics, Meibergdreef 9, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Digital Health and Quality of Care Amsterdam, the Netherlands
| | - Pirkko Nykänen
- Tampere University, Faculty for Information Technology and Communication Sciences, Finland
| | - Mirela Prgomet
- Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, Australia
| | - Robin Williams
- The University of Edinburgh, Institute for the Study of Science, Technology and Innovation, Edinburgh, United Kingdom
| |
Collapse
|
44
|
Yang J, Hao S, Huang J, Chen T, Liu R, Zhang P, Feng M, He Y, Xiao W, Hong Y, Zhang Z. The application of artificial intelligence in the management of sepsis. MEDICAL REVIEW (2021) 2023; 3:369-380. [PMID: 38283255 PMCID: PMC10811352 DOI: 10.1515/mr-2023-0039] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/22/2023] [Accepted: 11/08/2023] [Indexed: 01/30/2024]
Abstract
Sepsis is a complex and heterogeneous syndrome that remains a serious challenge to healthcare worldwide. Patients afflicted by severe sepsis or septic shock are customarily placed under intensive care unit (ICU) supervision, where a multitude of apparatus is poised to produce high-granularity data. This reservoir of high-quality data forms the cornerstone for the integration of AI into clinical practice. However, existing reviews currently lack the inclusion of the latest advancements. This review examines the evolving integration of artificial intelligence (AI) in sepsis management. Applications of artificial intelligence include early detection, subtyping analysis, precise treatment and prognosis assessment. AI-driven early warning systems provide enhanced recognition and intervention capabilities, while profiling analyzes elucidate distinct sepsis manifestations for targeted therapy. Precision medicine harnesses the potential of artificial intelligence for pathogen identification, antibiotic selection, and fluid optimization. In conclusion, the seamless amalgamation of artificial intelligence into the domain of sepsis management heralds a transformative shift, ushering in novel prospects to elevate diagnostic precision, therapeutic efficacy, and prognostic acumen. As AI technologies develop, their impact on shaping the future of sepsis care warrants ongoing research and thoughtful implementation.
Collapse
Affiliation(s)
- Jie Yang
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, Zhenjiang Province, China
| | - Sicheng Hao
- Duke University School of Medicine, Durham, NC, USA
| | - Jiajie Huang
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, Zhenjiang Province, China
| | - Tianqi Chen
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, Zhenjiang Province, China
| | - Ruoqi Liu
- Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA
| | - Ping Zhang
- Department of Computer Science and Engineering, The Ohio State University, Columbus, OH, USA
| | - Mengling Feng
- Saw Swee Hock School of Public Health and Institute of Data science, National University of Singapore, Singapore, Singapore
| | - Yang He
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, Zhenjiang Province, China
| | - Wei Xiao
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, Zhenjiang Province, China
| | - Yucai Hong
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, Zhenjiang Province, China
| | - Zhongheng Zhang
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, Zhenjiang Province, China
| |
Collapse
|
45
|
Wang DY, Ding J, Sun AL, Liu SG, Jiang D, Li N, Yu JK. Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias. J Am Med Inform Assoc 2023; 30:1684-1692. [PMID: 37561535 PMCID: PMC10531198 DOI: 10.1093/jamia/ocad118] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2022] [Revised: 05/30/2023] [Accepted: 06/19/2023] [Indexed: 08/11/2023] Open
Abstract
BACKGROUND Incorporating artificial intelligence (AI) into clinics brings the risk of automation bias, which potentially misleads the clinician's decision-making. The purpose of this study was to propose a potential strategy to mitigate automation bias. METHODS This was a laboratory study with a randomized cross-over design. The diagnosis of anterior cruciate ligament (ACL) rupture, a common injury, on magnetic resonance imaging (MRI) was used as an example. Forty clinicians were invited to diagnose 200 ACLs with and without AI assistance. The AI's correcting and misleading (automation bias) effects on the clinicians' decision-making processes were analyzed. An ordinal logistic regression model was employed to predict the correcting and misleading probabilities of the AI. We further proposed an AI suppression strategy that retracted AI diagnoses with a higher misleading probability and provided AI diagnoses with a higher correcting probability. RESULTS The AI significantly increased clinicians' accuracy from 87.2%±13.1% to 96.4%±1.9% (P < .001). However, the clinicians' errors in the AI-assisted round were associated with automation bias, accounting for 45.5% of the total mistakes. The automation bias was found to affect clinicians of all levels of expertise. Using a logistic regression model, we identified an AI output zone with higher probability to generate misleading diagnoses. The proposed AI suppression strategy was estimated to decrease clinicians' automation bias by 41.7%. CONCLUSION Although AI improved clinicians' diagnostic performance, automation bias was a serious problem that should be addressed in clinical practice. The proposed AI suppression strategy is a practical method for decreasing automation bias.
Collapse
Affiliation(s)
- Ding-Yu Wang
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| | - Jia Ding
- Beijing Yizhun Medical AI Co., Ltd, Beijing, China
| | - An-Lan Sun
- Beijing Yizhun Medical AI Co., Ltd, Beijing, China
| | - Shang-Gui Liu
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| | - Dong Jiang
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| | - Nan Li
- Research Center of Clinical Epidemiology, Peking University Third Hospital, Beijing, China
| | - Jia-Kuo Yu
- Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China
- Beijing Key Laboratory of Sports Injuries, Beijing, China
- Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China
| |
Collapse
|
46
|
Di Bidino R, Piaggio D, Andellini M, Merino-Barbancho B, Lopez-Perez L, Zhu T, Raza Z, Ni M, Morrison A, Borsci S, Fico G, Pecchia L, Iadanza E. Scoping Meta-Review of Methods Used to Assess Artificial Intelligence-Based Medical Devices for Heart Failure. Bioengineering (Basel) 2023; 10:1109. [PMID: 37892839 PMCID: PMC10604154 DOI: 10.3390/bioengineering10101109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 09/13/2023] [Accepted: 09/17/2023] [Indexed: 10/29/2023] Open
Abstract
Artificial intelligence and machine learning (AI/ML) are playing increasingly important roles, permeating the field of medical devices (MDs). This rapid progress has not yet been matched by the Health Technology Assessment (HTA) process, which still needs to define a common methodology for assessing AI/ML-based MDs. To collect existing evidence from the literature about the methods used to assess AI-based MDs, with a specific focus on those used for the management of heart failure (HF), the International Federation of Medical and Biological Engineering (IFMBE) conducted a scoping meta-review. This manuscript presents the results of this search, which covered the period from January 1974 to October 2022. After careful independent screening, 21 reviews, mainly conducted in North America and Europe, were retained and included. Among the findings were that deep learning is the most commonly utilised method and that electronic health records and registries are among the most prevalent sources of data for AI/ML algorithms. Out of the 21 included reviews, 19 focused on risk prediction and/or the early diagnosis of HF. Furthermore, 10 reviews provided evidence of the impact on the incidence/progression of HF, and 13 on the length of stay. From an HTA perspective, the main areas requiring improvement are the quality assessment of studies on AI/ML (included in 11 out of 21 reviews) and their data sources, as well as the definition of the criteria used to assess the selection of the most appropriate AI/ML algorithm.
Collapse
Affiliation(s)
- Rossella Di Bidino
- Fondazione Policlinico Universitario Agostino Gemelli IRCCS—The Graduate School of Health Economics and Management (ALTEMS), 00168 Rome, Italy
| | - Davide Piaggio
- School of Engineering, University of Warwick, Coventry CV4 7AL, UK; (D.P.); (M.A.); (Z.R.); (L.P.)
| | - Martina Andellini
- School of Engineering, University of Warwick, Coventry CV4 7AL, UK; (D.P.); (M.A.); (Z.R.); (L.P.)
| | - Beatriz Merino-Barbancho
- Life Supporting Technologies, Photonics Technology and Bioengineering Department, School of Telecommunication Engineering, Universidad Politécnica de Madrid, 28040 Madrid, Spain (L.L.-P.); (G.F.)
| | - Laura Lopez-Perez
- Life Supporting Technologies, Photonics Technology and Bioengineering Department, School of Telecommunication Engineering, Universidad Politécnica de Madrid, 28040 Madrid, Spain (L.L.-P.); (G.F.)
| | - Tianhui Zhu
- NIHR London In-Vitro Diagnostics Cooperative, Imperial College of London, London W2 1NY, UK
| | - Zeeshan Raza
- School of Engineering, University of Warwick, Coventry CV4 7AL, UK; (D.P.); (M.A.); (Z.R.); (L.P.)
| | - Melody Ni
- NIHR London In-Vitro Diagnostics Cooperative, Imperial College of London, London W2 1NY, UK
| | - Andra Morrison
- Canadian Agency for Drugs and Technologies in Health, Ottawa, ON K1S 5S8, Canada;
| | - Simone Borsci
- NIHR London In-Vitro Diagnostics Cooperative, Imperial College of London, London W2 1NY, UK
- Department of Learning, Data Analysis, and Technology, Cognition, Data and Education (CODE) Group, Faculty of Behavioural Management and Social Sciences, University of Twente, 7522 Enschede, The Netherlands
| | - Giuseppe Fico
- Life Supporting Technologies, Photonics Technology and Bioengineering Department, School of Telecommunication Engineering, Universidad Politécnica de Madrid, 28040 Madrid, Spain (L.L.-P.); (G.F.)
| | - Leandro Pecchia
- School of Engineering, University of Warwick, Coventry CV4 7AL, UK; (D.P.); (M.A.); (Z.R.); (L.P.)
- School of Engineering, University Campus Bio-Medico, 00128 Rome, Italy
- International Federation of Medical and Biological Engineering, B-1090 Brussels, Belgium
| | - Ernesto Iadanza
- International Federation of Medical and Biological Engineering, B-1090 Brussels, Belgium
- Department of Medical Biotechnologies, University of Siena, 53100 Siena, Italy
| |
Collapse
|
47
|
Cuthbert R, Simpson AI. Artificial intelligence in orthopaedics: can Chat Generative Pre-trained Transformer (ChatGPT) pass Section 1 of the Fellowship of the Royal College of Surgeons (Trauma & Orthopaedics) examination? Postgrad Med J 2023; 99:1110-1114. [PMID: 37410674 DOI: 10.1093/postmj/qgad053] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2023] [Revised: 05/26/2023] [Accepted: 06/01/2023] [Indexed: 07/08/2023]
Abstract
PURPOSE Chat Generative Pre-trained Transformer (ChatGPT) is a large language artificial intelligence (AI) model which generates contextually relevant text in response to questioning. After ChatGPT successfully passed the United States Medical Licensing Examinations, proponents have argued it should play an increasing role in medical service provision and education. AI in healthcare remains in its infancy, and the reliability of AI systems must be scrutinized. This study assessed whether ChatGPT could pass Section 1 of the Fellowship of the Royal College of Surgeons (FRCS) examination in Trauma and Orthopaedic Surgery. METHODS The UK and Ireland In-Training Examination (UKITE) was used as a surrogate for the FRCS. Papers 1 and 2 of UKITE 2022 were directly inputted into ChatGPT. All questions were in a single-best-answer format without wording alterations. Imaging was trialled to ensure ChatGPT utilized this information. RESULTS ChatGPT scored 35.8%: 30% lower than the FRCS pass rate and 8.2% lower than the mean score achieved by human candidates of all training levels. Subspecialty analysis demonstrated ChatGPT scored highest in basic science (53.3%) and lowest in trauma (0%). In 87 questions answered incorrectly, ChatGPT only stated it did not know the answer once and gave incorrect explanatory answers for the remaining questions. CONCLUSION ChatGPT is currently unable to exert the higher-order judgement and multilogical thinking required to pass the FRCS examination. Further, the current model fails to recognize its own limitations. ChatGPT's deficiencies should be publicized equally as much as its successes to ensure clinicians remain aware of its fallibility. KEY MESSAGES
Collapse
Affiliation(s)
- Rory Cuthbert
- Guy's and St Thomas' Hospital National Health Service Foundation Trust, London, SE1 9RT, United Kingdom
| | - Ashley I Simpson
- Guy's and St Thomas' Hospital National Health Service Foundation Trust, London, SE1 9RT, United Kingdom
| |
Collapse
|
48
|
Wang SM, Hogg HDJ, Sangvai D, Patel MR, Weissler EH, Kellogg KC, Ratliff W, Balu S, Sendak M. Development and Integration of Machine Learning Algorithm to Identify Peripheral Arterial Disease: Multistakeholder Qualitative Study. JMIR Form Res 2023; 7:e43963. [PMID: 37733427 PMCID: PMC10557008 DOI: 10.2196/43963] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 01/20/2023] [Accepted: 04/30/2023] [Indexed: 09/22/2023] Open
Abstract
BACKGROUND Machine learning (ML)-driven clinical decision support (CDS) continues to draw wide interest and investment as a means of improving care quality and value, despite mixed real-world implementation outcomes. OBJECTIVE This study aimed to explore the factors that influence the integration of a peripheral arterial disease (PAD) identification algorithm to implement timely guideline-based care. METHODS A total of 12 semistructured interviews were conducted with individuals from 3 stakeholder groups during the first 4 weeks of integration of an ML-driven CDS. The stakeholder groups included technical, administrative, and clinical members of the team interacting with the ML-driven CDS. The ML-driven CDS identified patients with a high probability of having PAD, and these patients were then reviewed by an interdisciplinary team that developed a recommended action plan and sent recommendations to the patient's primary care provider. Pseudonymized transcripts were coded, and thematic analysis was conducted by a multidisciplinary research team. RESULTS Three themes were identified: positive factors translating in silico performance to real-world efficacy, organizational factors and data structure factors affecting clinical impact, and potential challenges to advancing equity. Our study found that the factors that led to successful translation of in silico algorithm performance to real-world impact were largely nontechnical, given adequate efficacy in retrospective validation, including strong clinical leadership, trustworthy workflows, early consideration of end-user needs, and ensuring that the CDS addresses an actionable problem. Negative factors of integration included failure to incorporate the on-the-ground context, the lack of feedback loops, and data silos limiting the ML-driven CDS. The success criteria for each stakeholder group were also characterized to better understand how teams work together to integrate ML-driven CDS and to understand the varying needs across stakeholder groups. CONCLUSIONS Longitudinal and multidisciplinary stakeholder engagement in the development and integration of ML-driven CDS underpins its effective translation into real-world care. Although previous studies have focused on the technical elements of ML-driven CDS, our study demonstrates the importance of including administrative and operational leaders as well as an early consideration of clinicians' needs. Seeing how different stakeholder groups have this more holistic perspective also permits more effective detection of context-driven health care inequities, which are uncovered or exacerbated via ML-driven CDS integration through structural and organizational challenges. Many of the solutions to these inequities lie outside the scope of ML and require coordinated systematic solutions for mitigation to help reduce disparities in the care of patients with PAD.
Collapse
Affiliation(s)
- Sabrina M Wang
- Duke University School of Medicine, Durham, NC, United States
| | - H D Jeffry Hogg
- Population Health Science Institute, Faculty of Medical Sciences, Newcastle University, Newcastle upon Tyne, United Kingdom
- Newcastle Eye Centre, Royal Victoria Infirmary, Newcastle upon Tyne, United Kingdom
| | - Devdutta Sangvai
- Population Health Management, Duke Health, Durham, NC, United States
| | - Manesh R Patel
- Department of Cardiology, Duke University, Durham, NC, United States
| | - E Hope Weissler
- Department of Vascular Surgery, Duke University, Durham, NC, United States
| | | | - William Ratliff
- Duke Institute for Health Innovation, Durham, NC, United States
| | - Suresh Balu
- Duke Institute for Health Innovation, Durham, NC, United States
| | - Mark Sendak
- Duke Institute for Health Innovation, Durham, NC, United States
| |
Collapse
|
49
|
Jacobs IN, Giordano T, Soaper A, Din TF, Faig W, de Alarcon A, Balakrishnan K, Prager JD, Michael R, Douglas S, Piccione J. A multicenter study analyzing the impact of pre-existing comorbidities on laryngotracheal reconstruction (LTR) outcomes. Int J Pediatr Otorhinolaryngol 2023; 172:111631. [PMID: 37567085 DOI: 10.1016/j.ijporl.2023.111631] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Revised: 05/30/2023] [Accepted: 06/11/2023] [Indexed: 08/13/2023]
Abstract
INTRODUCTION Comorbidities such as chronic lung disease and gastroesophageal reflux (GERD), prematurity, and numerous other conditions may impact the success of LTR. Single-center studies are limited in terms of patient numbers and may be underpowered. OBJECTIVES To analyze the impact of specific comorbidities on the operation-specific and overall surgical success of LTR in a large multicenter cohort and validate a predictive model for surgical success. METHODS A large retrospective multicenter 10-year review was undertaken to validate the data of a previous single-center study (Wertz et al. Laryngoscope 2020) which identified specific predictive comorbidities which impacted LTR outcomes. A Monte Carlo simulation based on the previous data set suggested that 300-400 cases would be needed to optimize the statistical power of a Bayesian model developed from the single-center data to predict surgical success. An IRB-approved data-sharing agreement was executed for 4 large U.S. CENTERS A virtual REDCap® data entry form inquired about patient characteristics that best predicted surgical success in the single-center model. These included demographics, surgical approaches, cardiac, airway, genetic, endocrine, musculoskeletal, gastrointestinal, and pulmonary comorbidities; details of the surgical procedures, and results of esophagogastroduodenoscopy (EGD), esophageal pH/impedance and flexible bronchoscopy with bronchioalveolar lavage (BAL) were included. Surgical success defined as successful decannulation or resolution of airway symptoms was recorded as single surgery success and overall success following open surgical revision surgery. Multivariate Bayesian analysis, logistical regression, and Kaplan-Meier analysis were performed. RESULTS 542 patients were identified, including 165 from the single-center study and an additional 377 patients from the multicenter group. The median age was 36 months at the time of the most recent surgery. 70.9% of the LTRs were double-staged procedures. The overall success rate was 86.4% and operation-specific success rate was 69.2%. The specific comorbidities and aerodigestive test results that impacted success based on univariate analysis included staging, bronchiectasis, pulmonary hypertension, GERD, ASD, PDA, grade of stenosis, advanced levels of stenosis, Trisomy 21, MRSA, prior open surgery at another hospital, and gross appearance on EGD. Bayesian model averaging with backward selection was used to validate and refine a predictive model for surgical success with favorable receiver operating curve characteristics - AUC values of 0.827 for single surgery success and 0.797 for overall success. DISCUSSION With over 500 patients reviewed, this was the largest multicenter study of LTR to date, which elucidated the impact of comorbidities on success with LTR and was able to improve upon the predictive modeling based on single-center data. Patient factors are most critical in the outcome of LTR. Stage and levels of stenosis, as well as pulmonary and GI conditions most strongly impact the likelihood of success. Future prospective case-control studies will be performed to further optimize the current model for outcome prediction and patient management.
Collapse
Affiliation(s)
- Ian N Jacobs
- Division of Otolaryngology, Children's Hospital of Philadelphia, 3500 Civic Center Boulevard, 5th Floor, Philadelphia, PA, 19104, USA.
| | - Teresa Giordano
- Division of Otolaryngology, Children's Hospital of Philadelphia, 3500 Civic Center Boulevard, 5th Floor, Philadelphia, PA, 19104, USA.
| | - Ashley Soaper
- Division of Pediatric Otolaryngology-HNS, Cincinnati Children's Hospital Medical Center, 3333 Burnet Ave, Cincinnati, OH, 45229, USA.
| | - Taseer Feroze Din
- Division of Pediatric Otolaryngology, Lucile Packard Children's Hospital of Stanford University, 730 Welch Rd, 1st Floor, Palo Alto, CA, 94304, USA.
| | - Walter Faig
- Wescott Department of Biostatistics, Children's Hospital of Philadelphia, Roberts Center for Pediatric Research, 2716 South Street, Philadelphia, PA, 19146, USA.
| | - Alessandro de Alarcon
- Division of Pediatric Otolaryngology-HNS, Cincinnati Children's Hospital Medical Center, 3333 Burnet Ave, Cincinnati, OH, 45229, USA.
| | - Karthik Balakrishnan
- Division of Pediatric Otolaryngology, Lucile Packard Children's Hospital of Stanford University, 730 Welch Rd, 1st Floor, Palo Alto, CA, 94304, USA.
| | - Jeremy D Prager
- Department of Pediatric Otolaryngology, Children's Hospital of Colorado, 13123 E. 16th Avenue, Aurora, CO, 80045, USA.
| | - Rutter Michael
- Division of Pediatric Otolaryngology-HNS, Cincinnati Children's Hospital Medical Center, 3333 Burnet Ave, Cincinnati, OH, 45229, USA.
| | - Sidell Douglas
- Division of Pediatric Otolaryngology, Lucile Packard Children's Hospital of Stanford University, 730 Welch Rd, 1st Floor, Palo Alto, CA, 94304, USA.
| | - Joseph Piccione
- Division of Pulmonary and Sleep Medicine, Children's Hospital of Philadelphia, 3500 Civic Center Boulevard, 14th Floor, Philadelphia, PA, 19104, USA.
| |
Collapse
|
50
|
Singareddy S, Sn VP, Jaramillo AP, Yasir M, Iyer N, Hussein S, Nath TS. Artificial Intelligence and Its Role in the Management of Chronic Medical Conditions: A Systematic Review. Cureus 2023; 15:e46066. [PMID: 37900468 PMCID: PMC10607642 DOI: 10.7759/cureus.46066] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Accepted: 09/27/2023] [Indexed: 10/31/2023] Open
Abstract
Due to the increased burden of chronic medical conditions in recent years, artificial intelligence (AI) is suggested in the medical field to optimize health care. Physicians could implement these automated problem-solving tools for their benefit, reducing their workload, assisting in diagnostics, and supporting clinical decision-making. These tools are being considered for future medical assistance in real life. A literature review was performed to assess the impact of AI on the patient population with chronic medical conditions, using standardized guidelines. A MeSH strategy was created, and the database was searched for appropriate studies using specific inclusion and exclusion criteria. The online database yielded 93 results from various databases, of which 10 moderate to high-quality studies were selected to be included in our systematic review after removing the duplicates, screening titles, and articles. Of the 10 studies, nine recommended using AI after considering the potential limitations such as privacy protection, medicolegal implications, and psychosocial aspects. Due to its non-fatigable nature, AI was found to be of immense help in image recognition. It was also found to be valuable in various disciplines related to administration, physician burden, and patient adherence. The newer technologies of Chatbots and eHealth applications are of great help when used safely and effectively after proper patient education. After a careful review conducted by our team members, it is safe to conclude that implementing AI in daily clinical practice could potentiate the cognitive ability of physicians and decrease the workload through various automated technologies such as image recognition, speech recognition, and voice recognition due to its unmatchable speed and non-fatigable nature when compared to clinicians. Despite its vast benefits to the medical field, a few limitations could hinder its effective implementation into real-life practice, which requires enormous research and strict regulations to support its role as a physician's aid. However, AI should only be used as a medical support system, in order to improve the primary outcomes such as reducing waiting time, healthcare costs, and workload. AI should not be meant to replace physicians.
Collapse
Affiliation(s)
- Sanjana Singareddy
- Internal Medicine, California Institute of Behavioral Neurosciences & Psychology, Fairfield, USA
| | - Vijay Prabhu Sn
- Internal Medicine, California Institute of Behavioral Neurosciences & Psychology, Fairfield, USA
| | - Arturo P Jaramillo
- General Practice, California Institute of Behavioral Neurosciences & Psychology, Fairfield, USA
| | - Mohamed Yasir
- Research, California Institute of Behavioral Neurosciences & Psychology, Fairfield, USA
| | - Nandhini Iyer
- Internal Medicine, California Institute of Behavioral Neurosciences & Psychology, Fairfield, USA
| | - Sally Hussein
- Internal Medicine, California Institute of Behavioral Neurosciences & Psychology, Fairfield, USA
| | - Tuheen Sankar Nath
- Surgical Oncology, California Institute of Behavioral Neurosciences & Psychology, Fairfield, USA
| |
Collapse
|