1
|
Jemaa S, Ounadjela S, Wang X, El-Galaly TC, Kostakoglu L, Knapp A, Ku G, Musick L, Sahin D, Wei MC, Yin S, Bengtsson T, De Crespigny A, Carano RAD. Automated Lugano Metabolic Response Assessment in 18F-Fluorodeoxyglucose-Avid Non-Hodgkin Lymphoma With Deep Learning on 18F-Fluorodeoxyglucose-Positron Emission Tomography. J Clin Oncol 2024; 42:2966-2977. [PMID: 38843483 DOI: 10.1200/jco.23.01978] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Revised: 02/12/2024] [Accepted: 03/14/2024] [Indexed: 08/30/2024] Open
Abstract
PURPOSE Artificial intelligence can reduce the time used by physicians on radiological assessments. For 18F-fluorodeoxyglucose-avid lymphomas, obtaining complete metabolic response (CMR) by end of treatment is prognostic. METHODS Here, we present a deep learning-based algorithm for fully automated treatment response assessments according to the Lugano 2014 classification. The proposed four-stage method, trained on a multicountry clinical trial (ClinicalTrials.gov identifier: NCT01287741) and tested in three independent multicenter and multicountry test sets on different non-Hodgkin lymphoma subtypes and different lines of treatment (ClinicalTrials.gov identifiers NCT02257567, NCT02500407; 20% holdout in ClinicalTrials.gov identifier NCT01287741), outputs the detected lesions at baseline and follow-up to enable focused radiologist review. RESULTS The method's response assessment achieved high agreement with the adjudicated radiologic responses (eg, agreement for overall response assessment of 93%, 87%, and 85% in ClinicalTrials.gov identifiers NCT01287741, NCT02500407, and NCT02257567, respectively) similar to inter-radiologist agreement and was strongly prognostic of outcomes with a trend toward higher accuracy for death risk than adjudicated radiologic responses (hazard ratio for end of treatment by-model CMR of 0.123, 0.054, and 0.205 in ClinicalTrials.gov identifiers NCT01287741, NCT02500407, and NCT02257567, compared with, respectively, 0.226, 0.292, and 0.272 for CMR by the adjudicated responses). Furthermore, a radiologist review of the algorithm's assessments was conducted. The radiologist median review time was 1.38 minutes/assessment, and no statistically significant differences were observed in the level of agreement of the radiologist with the model's response compared with the level of agreement of the radiologist with the adjudicated responses. CONCLUSION These results suggest that the proposed method can be incorporated into radiologic response assessment workflows in cancer imaging for significant time savings and with performance similar to trained medical experts.
Collapse
Affiliation(s)
| | | | | | - Tarec C El-Galaly
- Department of Hematology, Aalborg University Hospital, Aalborg, Denmark
- Hematology Research Unit, Department of Hematology, Odense University Hospital, Odense, Denmark
- Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- Department of Medicine Solna, Clinical Epidemiology Division, Karolinska Institutet, Stockholm, Sweden
| | - Lale Kostakoglu
- Department of Radiology and Medical Imaging, University of Virginia, Charlottesville, VA
| | | | - Grace Ku
- Genentech, Inc, South San Francisco, CA
| | | | | | | | - Shen Yin
- Genentech, Inc, South San Francisco, CA
| | - Thomas Bengtsson
- Department of Statistics, University of California, Berkeley, CA
| | | | | |
Collapse
|
2
|
Dubin JA, Bains SS, DeRogatis MJ, Moore MC, Hameed D, Mont MA, Nace J, Delanois RE. Appropriateness of Frequently Asked Patient Questions Following Total Hip Arthroplasty From ChatGPT Compared to Arthroplasty-Trained Nurses. J Arthroplasty 2024; 39:S306-S311. [PMID: 38626863 DOI: 10.1016/j.arth.2024.04.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 04/02/2024] [Accepted: 04/05/2024] [Indexed: 05/16/2024] Open
Abstract
BACKGROUND The use of ChatGPT (Generative Pretrained Transformer), which is a natural language artificial intelligence model, has gained unparalleled attention with the accumulation of over 100 million users within months of launching. As such, we aimed to compare the following: 1) orthopaedic surgeons' evaluation of the appropriateness of the answers to the most frequently asked patient questions after total hip arthroplasty; and 2) patients' evaluation of ChatGPT and arthroplasty-trained nurses responses to answer their postoperative questions. METHODS We prospectively created 60 questions to address the most commonly asked patient questions following total hip arthroplasty. We obtained answers from arthroplasty-trained nurses and from the ChatGPT-3.5 version for each of the questions. Surgeons graded each set of responses based on clinical judgment as 1) "appropriate," 2) "inappropriate" if the response contained inappropriate information, or 3) "unreliable" if the responses provided inconsistent content. Each patient was given a randomly selected question from the 60 aforementioned questions, with responses provided by ChatGPT and arthroplasty-trained nurses, using a Research Electronic Data Capture survey hosted at our local hospital. RESULTS The 3 fellowship-trained surgeons graded 56 out of 60 (93.3%) responses for the arthroplasty-trained nurses and 57 out of 60 (95.0%) for ChatGPT to be "appropriate." There were 175 out of 252 (69.4%) patients who were more comfortable following the ChatGPT responses and 77 out of 252 (30.6%) who preferred arthroplasty-trained nurses' responses. However, 199 out of 252 patients (79.0%) responded that they were "uncertain" with regard to trusting AI to answer their postoperative questions. CONCLUSIONS ChatGPT provided appropriate answers from a physician perspective. Patients were also more comfortable with the ChatGPT responses than those from arthroplasty-trained nurses. Inevitably, its successful implementation is dependent on its ability to provide credible information that is consistent with the goals of the physician and patient alike.
Collapse
Affiliation(s)
- Jeremy A Dubin
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Sandeep S Bains
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Michael J DeRogatis
- Department of Orthopaedic Surgery, St. Luke's University Health Network, Bethlehem, Pennsylvania
| | - Mallory C Moore
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Daniel Hameed
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Michael A Mont
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - James Nace
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| | - Ronald E Delanois
- LifeBridge Health, Sinai Hospital of Baltimore, Rubin Institute for Advanced Orthopedics, Baltimore, Maryland
| |
Collapse
|
3
|
Chen Z, Liang N, Li H, Zhang H, Li H, Yan L, Hu Z, Chen Y, Zhang Y, Wang Y, Ke D, Shi N. Exploring explainable AI features in the vocal biomarkers of lung disease. Comput Biol Med 2024; 179:108844. [PMID: 38981214 DOI: 10.1016/j.compbiomed.2024.108844] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2024] [Revised: 05/15/2024] [Accepted: 06/04/2024] [Indexed: 07/11/2024]
Abstract
This review delves into the burgeoning field of explainable artificial intelligence (XAI) in the detection and analysis of lung diseases through vocal biomarkers. Lung diseases, often elusive in their early stages, pose a significant public health challenge. Recent advancements in AI have ushered in innovative methods for early detection, yet the black-box nature of many AI models limits their clinical applicability. XAI emerges as a pivotal tool, enhancing transparency and interpretability in AI-driven diagnostics. This review synthesizes current research on the application of XAI in analyzing vocal biomarkers for lung diseases, highlighting how these techniques elucidate the connections between specific vocal features and lung pathology. We critically examine the methodologies employed, the types of lung diseases studied, and the performance of various XAI models. The potential for XAI to aid in early detection, monitor disease progression, and personalize treatment strategies in pulmonary medicine is emphasized. Furthermore, this review identifies current challenges, including data heterogeneity and model generalizability, and proposes future directions for research. By offering a comprehensive analysis of explainable AI features in the context of lung disease detection, this review aims to bridge the gap between advanced computational approaches and clinical practice, paving the way for more transparent, reliable, and effective diagnostic tools.
Collapse
Affiliation(s)
- Zhao Chen
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Ning Liang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Haoyuan Li
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Haili Zhang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Huizhen Li
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Lijiao Yan
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Ziteng Hu
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Yaxin Chen
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Yujing Zhang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Yanping Wang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Dandan Ke
- Special Disease Clinic, Huaishuling Branch of Beijing Fengtai Hospital of Integrated Traditional Chinese and Western Medicine, Beijing, China.
| | - Nannan Shi
- State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China; Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China.
| |
Collapse
|
4
|
Gondode P, Duggal S, Garg N, Lohakare P, Jakhar J, Bharti S, Dewangan S. Comparative Analysis of Accuracy, Readability, Sentiment, and Actionability: Artificial Intelligence Chatbots (ChatGPT and Google Gemini) versus Traditional Patient Information Leaflets for Local Anesthesia in Eye Surgery. Br Ir Orthopt J 2024; 20:183-192. [PMID: 39183761 PMCID: PMC11342839 DOI: 10.22599/bioj.377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2024] [Accepted: 07/31/2024] [Indexed: 08/27/2024] Open
Abstract
Background and Aim Eye surgeries often evoke strong negative emotions in patients, including fear and anxiety. Patient education material plays a crucial role in informing and empowering individuals. Traditional sources of medical information may not effectively address individual patient concerns or cater to varying levels of understanding. This study aims to conduct a comparative analysis of the accuracy, completeness, readability, tone, and understandability of patient education material generated by AI chatbots versus traditional Patient Information Leaflets (PILs), focusing on local anesthesia in eye surgery. Methods Expert reviewers evaluated responses generated by AI chatbots (ChatGPT and Google Gemini) and a traditional PIL (Royal College of Anaesthetists' PIL) based on accuracy, completeness, readability, sentiment, and understandability. Statistical analyses, including ANOVA and Tukey HSD tests, were conducted to compare the performance of the sources. Results Readability analysis showed variations in complexity among the sources, with AI chatbots offering simplified language and PILs maintaining better overall readability and accessibility. Sentiment analysis revealed differences in emotional tone, with Google Gemini exhibiting the most positive sentiment. AI chatbots demonstrated superior understandability and actionability, while PILs excelled in completeness. Overall, ChatGPT showed slightly higher accuracy (scores expressed as mean ± standard deviation) (4.71 ± 0.5 vs 4.61 ± 0.62) and completeness (4.55 ± 0.58 vs 4.47 ± 0.58) compared to Google Gemini, but PILs performed best (4.84 ± 0.37 vs 4.88 ± 0.33) in terms of both accuracy and completeness (p-value for completeness <0.05). Conclusion AI chatbots show promise as innovative tools for patient education, complementing traditional PILs. By leveraging the strengths of both AI-driven technologies and human expertise, healthcare providers can enhance patient education and empower individuals to make informed decisions about their health and medical care.
Collapse
Affiliation(s)
- Prakash Gondode
- Department of Anaesthesiology, Pain medicine and Critical Care, All India Institute of Medical Sciences (AIIMS), New Delhi, India
| | - Sakshi Duggal
- Department of Anaesthesiology, Pain medicine and Critical Care, All India Institute of Medical Sciences (AIIMS), New Delhi, India
| | - Neha Garg
- Department of Anaesthesiology, Pain medicine and Critical Care, All India Institute of Medical Sciences (AIIMS), New Delhi, India
| | - Pooja Lohakare
- Department of Microbiology, Mahatma Gandhi Institute of Medical Sciences (MGIMS), Wardha, India
| | - Jubin Jakhar
- Department of Anaesthesiology, Pain medicine and Critical Care, University college of Medical Sciences (UCMS), Delhi, India
| | - Swati Bharti
- Department of Anaesthesiology, Pain medicine and Critical Care, All India Institute of Medical Sciences (AIIMS), New Delhi, India
| | - Shraddha Dewangan
- Department of Anaesthesiology, Pain medicine and Critical Care, All India Institute of Medical Sciences (AIIMS), New Delhi, India
| |
Collapse
|
5
|
Balas M, Mandelcorn ED, Yan P, Ing EB, Crawford SA, Arjmand P. ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines. CANADIAN JOURNAL OF OPHTHALMOLOGY 2024:S0008-4182(24)00175-3. [PMID: 39097289 DOI: 10.1016/j.jcjo.2024.06.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Revised: 02/11/2024] [Accepted: 06/03/2024] [Indexed: 08/05/2024]
Abstract
OBJECTIVE To evaluate the performance of an artificial intelligence (AI) large language model, ChatGPT (version 4.0), for common retinal diseases, in accordance with the American Academy of Ophthalmology (AAO) Preferred Practice Pattern (PPP) guidelines. DESIGN A cross-sectional survey study design was employed to compare the responses made by ChatGPT to established clinical guidelines. PARTICIPANTS Responses by the AI were reviewed by a panel of three vitreoretinal specialists for evaluation. METHODS To investigate ChatGPT's comprehension of clinical guidelines, we designed 130 questions covering a broad spectrum of topics within 12 AAO PPP domains of retinal disease These questions were crafted to encompass diagnostic criteria, treatment guidelines, and management strategies, including both medical and surgical aspects of retinal care. A panel of 3 retinal specialists independently evaluated responses on a Likert scale from 1 to 5 based on their relevance, accuracy, and adherence to AAO PPP guidelines. Response readability was evaluated using Flesch Readability Ease and Flesch-Kincaid grade level scores. RESULTS ChatGPT achieved an overall average score of 4.9/5.0, suggesting high alignment with the AAO PPP guidelines. Scores varied across domains, with the lowest in the surgical management of disease. The responses had a low reading ease score and required a college-to-graduate level of comprehension. Identified errors were related to diagnostic criteria, treatment options, and methodological procedures. CONCLUSION ChatGPT 4.0 demonstrated significant potential in generating guideline-concordant responses, particularly for common medical retinal diseases. However, its performance slightly decreased in surgical retina, highlighting the ongoing need for clinician input, further model refinement, and improved comprehensibility.
Collapse
Affiliation(s)
- Michael Balas
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Efrem D Mandelcorn
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, ON, Canada; University Health Network, University of Toronto, Toronto, ON, Canada; Kensington Eye Institute, Toronto, ON, Canada
| | - Peng Yan
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, ON, Canada; University Health Network, University of Toronto, Toronto, ON, Canada; Kensington Eye Institute, Toronto, ON, Canada
| | - Edsel B Ing
- Department of Ophthalmology and Vision Sciences, University of Toronto, Toronto, ON, Canada; Department of Ophthalmology and Visual Sciences, University of Alberta, Edmonton, AB, Canada
| | - Sean A Crawford
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada; University Health Network, University of Toronto, Toronto, ON, Canada; Division of Vascular Surgery, Department of Surgery, University of Toronto, Toronto, ON, Canada; Peter Munk Cardiac Centre, Toronto General Hospital, University Health Network, Toronto, ON, Canada
| | | |
Collapse
|
6
|
Hadi A, Tran E, Nagarajan B, Kirpalani A. Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians. PLoS One 2024; 19:e0307383. [PMID: 39083523 PMCID: PMC11290643 DOI: 10.1371/journal.pone.0307383] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Accepted: 07/03/2024] [Indexed: 08/02/2024] Open
Abstract
BACKGROUND ChatGPT is a large language model (LLM) trained on over 400 billion words from books, articles, and websites. Its extensive training draws from a large database of information, making it valuable as a diagnostic aid. Moreover, its capacity to comprehend and generate human language allows medical trainees to interact with it, enhancing its appeal as an educational resource. This study aims to investigate ChatGPT's diagnostic accuracy and utility in medical education. METHODS 150 Medscape case challenges (September 2021 to January 2023) were inputted into ChatGPT. The primary outcome was the number (%) of cases for which the answer given was correct. Secondary outcomes included diagnostic accuracy, cognitive load, and quality of medical information. A qualitative content analysis was also conducted to assess its responses. RESULTS ChatGPT answered 49% (74/150) cases correctly. It had an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an AUC of 0.66. Most answers were considered low cognitive load 51% (77/150) and most answers were complete and relevant 52% (78/150). DISCUSSION ChatGPT in its current form is not accurate as a diagnostic tool. ChatGPT does not necessarily give factual correctness, despite the vast amount of information it was trained on. Based on our qualitative analysis, ChatGPT struggles with the interpretation of laboratory values, imaging results, and may overlook key information relevant to the diagnosis. However, it still offers utility as an educational tool. ChatGPT was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Additionally, answers were easy to understand, showcasing a potential benefit in simplifying complex concepts for medical learners. Our results should guide future research into harnessing ChatGPT's potential educational benefits, such as simplifying medical concepts and offering guidance on differential diagnoses and next steps.
Collapse
Affiliation(s)
- Ali Hadi
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Edward Tran
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Branavan Nagarajan
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Amrit Kirpalani
- Department of Paediatrics, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Division of Nephrology, Children’s Hospital, London Health Sciences Centre, London, Ontario, Canada
| |
Collapse
|
7
|
Semerci ZM, Toru HS, Çobankent Aytekin E, Tercanlı H, Chiorean DM, Albayrak Y, Cotoi OS. The Role of Artificial Intelligence in Early Diagnosis and Molecular Classification of Head and Neck Skin Cancers: A Multidisciplinary Approach. Diagnostics (Basel) 2024; 14:1477. [PMID: 39061614 PMCID: PMC11276319 DOI: 10.3390/diagnostics14141477] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2024] [Revised: 07/01/2024] [Accepted: 07/09/2024] [Indexed: 07/28/2024] Open
Abstract
Cancer remains a significant global health concern, with increasing genetic and metabolic irregularities linked to its onset. Among various forms of cancer, skin cancer, including squamous cell carcinoma, basal cell carcinoma, and melanoma, is on the rise worldwide, often triggered by ultraviolet (UV) radiation. The propensity of skin cancer to metastasize highlights the importance of early detection for successful treatment. This narrative review explores the evolving role of artificial intelligence (AI) in diagnosing head and neck skin cancers from both radiological and pathological perspectives. In the past two decades, AI has made remarkable progress in skin cancer research, driven by advances in computational capabilities, digitalization of medical images, and radiomics data. AI has shown significant promise in image-based diagnosis across various medical domains. In dermatology, AI has played a pivotal role in refining diagnostic and treatment strategies, including genomic risk assessment. This technology offers substantial potential to aid primary clinicians in improving patient outcomes. Studies have demonstrated AI's effectiveness in identifying skin lesions, categorizing them, and assessing their malignancy, contributing to earlier interventions and better prognosis. The rising incidence and mortality rates of skin cancer, coupled with the high cost of treatment, emphasize the need for early diagnosis. Further research and integration of AI into clinical practice are warranted to maximize its benefits in skin cancer diagnosis and treatment.
Collapse
Affiliation(s)
- Zeliha Merve Semerci
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Akdeniz University, 07070 Antalya, Turkey; (Z.M.S.); (H.T.)
| | - Havva Serap Toru
- Department of Pathology, Faculty of Medicine, Akdeniz University, 07070 Antalya, Turkey
| | | | - Hümeyra Tercanlı
- Department of Oral and Maxillofacial Radiology, Faculty of Dentistry, Akdeniz University, 07070 Antalya, Turkey; (Z.M.S.); (H.T.)
| | - Diana Maria Chiorean
- Department of Pathology, County Clinical Hospital of Targu Mures, 540072 Targu Mures, Romania; (D.M.C.); (O.S.C.)
- Department of Pathophysiology, “George Emil Palade” University of Medicine, Pharmacy, Science, and Technology of Targu Mures, 38 Gheorghe Marinescu Street, 540142 Targu Mures, Romania
| | - Yalçın Albayrak
- Department of Electric and Electronic Engineering, Faculty of Engineering, Akdeniz University, 07010 Antalya, Turkey;
| | - Ovidiu Simion Cotoi
- Department of Pathology, County Clinical Hospital of Targu Mures, 540072 Targu Mures, Romania; (D.M.C.); (O.S.C.)
- Department of Pathophysiology, “George Emil Palade” University of Medicine, Pharmacy, Science, and Technology of Targu Mures, 38 Gheorghe Marinescu Street, 540142 Targu Mures, Romania
| |
Collapse
|
8
|
Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, Vielhauer J, Makowski M, Braren R, Kaissis G, Rueckert D. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med 2024:10.1038/s41591-024-03097-1. [PMID: 38965432 DOI: 10.1038/s41591-024-03097-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2024] [Accepted: 05/29/2024] [Indexed: 07/06/2024]
Abstract
Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.
Collapse
Affiliation(s)
- Paul Hager
- Institute for AI and Informatics, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany.
- Institute for Diagnostic and Interventional Radiology, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany.
| | - Friederike Jungmann
- Institute for AI and Informatics, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Institute for Diagnostic and Interventional Radiology, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | | | - Kunal Bhagat
- Department of Medicine, ChristianaCare Health System, Wilmington, DE, USA
| | - Inga Hubrecht
- Department of Medicine III, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Manuel Knauer
- Department of Medicine III, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Jakob Vielhauer
- Department of Medicine II, University Hospital of the Ludwig Maximilian University of Munich, Munich, Germany
| | - Marcus Makowski
- Institute for Diagnostic and Interventional Radiology, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Rickmer Braren
- Institute for Diagnostic and Interventional Radiology, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Georgios Kaissis
- Institute for AI and Informatics, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Institute for Diagnostic and Interventional Radiology, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Department of Computing, Imperial College, London, UK
- Reliable AI Group, Institute for Machine Learning in Biomedical Imaging, Helmholtz Munich, Munich, Germany
| | - Daniel Rueckert
- Institute for AI and Informatics, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Department of Computing, Imperial College, London, UK
| |
Collapse
|
9
|
Ramgopal S, Belanger T, Lorenz D, Lipsett SC, Neuman MI, Liebovitz D, Florin TA. Preferences for Management of Pediatric Pneumonia: A Clinician Survey of Artificially Generated Patient Cases. Pediatr Emerg Care 2024:00006565-990000000-00488. [PMID: 38950412 DOI: 10.1097/pec.0000000000003231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 07/03/2024]
Abstract
BACKGROUND It is unknown which factors are associated with chest radiograph (CXR) and antibiotic use for suspected community-acquired pneumonia (CAP) in children. We evaluated factors associated with CXR and antibiotic preferences among clinicians for children with suspected CAP using case scenarios generated through artificial intelligence (AI). METHODS We performed a survey of general pediatric, pediatric emergency medicine, and emergency medicine attending physicians employed by a private physician contractor. Respondents were given 5 unique, AI-generated case scenarios. We used generalized estimating equations to identify factors associated with CXR and antibiotic use. We evaluated the cluster-weighted correlation between clinician suspicion and clinical prediction model risk estimates for CAP using 2 predictive models. RESULTS A total of 172 respondents provided responses to 839 scenarios. Factors associated with CXR acquisition (OR, [95% CI]) included presence of crackles (4.17 [2.19, 7.95]), prior pneumonia (2.38 [1.32, 4.20]), chest pain (1.90 [1.18, 3.05]) and fever (1.82 [1.32, 2.52]). The decision to use antibiotics before knowledge of CXR results included past hospitalization for pneumonia (4.24 [1.88, 9.57]), focal decreased breath sounds (3.86 [1.98, 7.52]), and crackles (3.45 [2.15, 5.53]). After revealing CXR results to clinicians, these results were the sole predictor associated with antibiotic decision-making. Suspicion for CAP correlated with one of 2 prediction models for CAP (Spearman's rho = 0.25). Factors associated with a greater suspicion of pneumonia included prior pneumonia, duration of illness, worsening course of illness, shortness of breath, vomiting, decreased oral intake or urinary output, respiratory distress, head nodding, focal decreased breath sounds, focal rhonchi, fever, and crackles, and lower pulse oximetry. CONCLUSIONS Ordering preferences for CXRs demonstrated similarities and differences with evidence-based risk models for CAP. Clinicians relied heavily on CXR findings to guide antibiotic ordering. These findings can be used within decision support systems to promote evidence-based management practices for pediatric CAP.
Collapse
Affiliation(s)
- Sriram Ramgopal
- From the Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL
| | | | - Douglas Lorenz
- Department of Bioinformatics and Biostatistics, University of Louisville, Louisville, KY
| | - Susan C Lipsett
- Department of Pediatrics, Division of Emergency Medicine, Boston Children's Hospital, Harvard Medical School, Boston, MA
| | - Mark I Neuman
- Department of Pediatrics, Division of Emergency Medicine, Boston Children's Hospital, Harvard Medical School, Boston, MA
| | - David Liebovitz
- Department of General Internal Medicine, Northwestern University Feinberg School of Medicine, Northwestern University, Chicago, IL
| | - Todd A Florin
- From the Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL
| |
Collapse
|
10
|
Zou X, Cui N, Ma Q, Lin Z, Zhang J, Li X. Development of a machine learning model for predicting pneumothorax risk in coaxial core needle biopsy (≤3 cm). Eur J Radiol 2024; 176:111508. [PMID: 38759543 DOI: 10.1016/j.ejrad.2024.111508] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2023] [Revised: 03/31/2024] [Accepted: 05/13/2024] [Indexed: 05/19/2024]
Abstract
PURPOSE The aim is to devise a machine learning algorithm exploiting preoperative clinical data to forecast the hazard of pneumothorax post-coaxial needle lung biopsy (CCNB), thereby informing clinical decision-making and enhancing perioperative care. METHOD This retrospective analysis aggregated clinical and imaging data from patients with lung nodules (≤3 cm) biopsies. Variable selection was done using univariate analysis and LASSO regression, with the dataset subsequently divided into training (80 %) and validation (20 %) subsets. Various machine learning (ML) classifiers were employed in a consolidated approach to ascertain the paramount model, which was followed by individualized risk profiling showcased through Shapley Additive eXplanations (SHAP). RESULTS Out of the 325 patients included in the study, 19.6% (64/325) experienced postoperative pneumothorax. High-risk factors determined were Cancer, Lesion_type, GOLD, Size, and Depth. The Gaussian Naive Bayes (GNB) classifier demonstrated superior prediction with an Area Under the Curve (AUC) of 0.82 (95% CI 0.71-0.94), complemented by an accuracy rate of 0.8, sensitivity of 0.71, specificity of 0.84, and an F1 score of 0.61 in the test cohort. CONCLUSION The formulated prognostic algorithm exhibited commendable efficacy in preoperatively prognosticating CCNB-induced pneumothorax, harboring the potential to refine personalized risk appraisals, steer clinical judgment, and ameliorate perioperative patient stewardship.
Collapse
Affiliation(s)
- Xugong Zou
- Department of Interventional Medicine, Zhongshan People's Hospital, Zhongshan City 528403, Guangdong Province, China
| | - Ning Cui
- Medical Imaging Center, Taihe Hospital, Shiyan City, Hubei Province, China
| | - Qiang Ma
- Department of Interventional Medicine, Zhongshan People's Hospital, Zhongshan City 528403, Guangdong Province, China
| | - Zhipeng Lin
- Department of Interventional Medicine, Zhongshan People's Hospital, Zhongshan City 528403, Guangdong Province, China
| | - Jian Zhang
- Department of Interventional Medicine, Zhongshan People's Hospital, Zhongshan City 528403, Guangdong Province, China
| | - Xiaoqun Li
- Department of Interventional Medicine, Zhongshan People's Hospital, Zhongshan City 528403, Guangdong Province, China.
| |
Collapse
|
11
|
Yoon S, Goh H, Lee PC, Tan HC, Teh MM, Lim DST, Kwee A, Suresh C, Carmody D, Swee DS, Tan SYT, Wong AJW, Choo CHM, Wee Z, Bee YM. Assessing the Utility, Impact, and Adoption Challenges of an Artificial Intelligence-Enabled Prescription Advisory Tool for Type 2 Diabetes Management: Qualitative Study. JMIR Hum Factors 2024; 11:e50939. [PMID: 38869934 PMCID: PMC11211700 DOI: 10.2196/50939] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Revised: 11/07/2023] [Accepted: 05/05/2024] [Indexed: 06/14/2024] Open
Abstract
BACKGROUND The clinical management of type 2 diabetes mellitus (T2DM) presents a significant challenge due to the constantly evolving clinical practice guidelines and growing array of drug classes available. Evidence suggests that artificial intelligence (AI)-enabled clinical decision support systems (CDSSs) have proven to be effective in assisting clinicians with informed decision-making. Despite the merits of AI-driven CDSSs, a significant research gap exists concerning the early-stage implementation and adoption of AI-enabled CDSSs in T2DM management. OBJECTIVE This study aimed to explore the perspectives of clinicians on the use and impact of the AI-enabled Prescription Advisory (APA) tool, developed using a multi-institution diabetes registry and implemented in specialist endocrinology clinics, and the challenges to its adoption and application. METHODS We conducted focus group discussions using a semistructured interview guide with purposively selected endocrinologists from a tertiary hospital. The focus group discussions were audio-recorded and transcribed verbatim. Data were thematically analyzed. RESULTS A total of 13 clinicians participated in 4 focus group discussions. Our findings suggest that the APA tool offered several useful features to assist clinicians in effectively managing T2DM. Specifically, clinicians viewed the AI-generated medication alterations as a good knowledge resource in supporting the clinician's decision-making on drug modifications at the point of care, particularly for patients with comorbidities. The complication risk prediction was seen as positively impacting patient care by facilitating early doctor-patient communication and initiating prompt clinical responses. However, the interpretability of the risk scores, concerns about overreliance and automation bias, and issues surrounding accountability and liability hindered the adoption of the APA tool in clinical practice. CONCLUSIONS Although the APA tool holds great potential as a valuable resource for improving patient care, further efforts are required to address clinicians' concerns and improve the tool's acceptance and applicability in relevant contexts.
Collapse
Affiliation(s)
- Sungwon Yoon
- Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore
- Centre for Population Health Research and Implementation, SingHealth Regional Health System, SingHealth, Singapore, Singapore
| | - Hendra Goh
- Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore
| | - Phong Ching Lee
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Hong Chang Tan
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Ming Ming Teh
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Dawn Shao Ting Lim
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Ann Kwee
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Chandran Suresh
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - David Carmody
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Du Soon Swee
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Sarah Ying Tse Tan
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Andy Jun-Wei Wong
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | | | - Zongwen Wee
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| | - Yong Mong Bee
- Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
| |
Collapse
|
12
|
Li M, Xiong X, Xu B, Dickson C. Chinese Oncologists' Perspectives on Integrating AI into Clinical Practice: Cross-Sectional Survey Study. JMIR Form Res 2024; 8:e53918. [PMID: 38838307 PMCID: PMC11187515 DOI: 10.2196/53918] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Revised: 02/21/2024] [Accepted: 04/03/2024] [Indexed: 06/07/2024] Open
Abstract
BACKGROUND The rapid development of artificial intelligence (AI) has brought significant interest to its potential applications in oncology. Although AI-powered tools are already being implemented in some Chinese hospitals, their integration into clinical practice raises several concerns for Chinese oncologists. OBJECTIVE This study aims to explore the concerns of Chinese oncologists regarding the integration of AI into clinical practice and to identify the factors influencing these concerns. METHODS A total of 228 Chinese oncologists participated in a cross-sectional web-based survey from April to June in 2023 in mainland China. The survey gauged their worries about AI with multiple-choice questions. The survey evaluated their views on the statements of "The impact of AI on the doctor-patient relationship" and "AI will replace doctors." The data were analyzed using descriptive statistics, and variate analyses were used to find correlations between the oncologists' backgrounds and their concerns. RESULTS The study revealed that the most prominent concerns were the potential for AI to mislead diagnosis and treatment (163/228, 71.5%); an overreliance on AI (162/228, 71%); data and algorithm bias (123/228, 54%); issues with data security and patient privacy (123/228, 54%); and a lag in the adaptation of laws, regulations, and policies in keeping up with AI's development (115/228, 50.4%). Oncologists with a bachelor's degree expressed heightened concerns related to data and algorithm bias (34/49, 69%; P=.03) and the lagging nature of legal, regulatory, and policy issues (32/49, 65%; P=.046). Regarding AI's impact on doctor-patient relationships, 53.1% (121/228) saw a positive impact, whereas 35.5% (81/228) found it difficult to judge, 9.2% (21/228) feared increased disputes, and 2.2% (5/228) believed that there is no impact. Although sex differences were not significant (P=.08), perceptions varied-male oncologists tended to be more positive than female oncologists (74/135, 54.8% vs 47/93, 50%). Oncologists with a bachelor's degree (26/49, 53%; P=.03) and experienced clinicians (≥21 years; 28/56, 50%; P=.054). found it the hardest to judge. Those with IT experience were significantly more positive (25/35, 71%) than those without (96/193, 49.7%; P=.02). Opinions regarding the possibility of AI replacing doctors were diverse, with 23.2% (53/228) strongly disagreeing, 14% (32/228) disagreeing, 29.8% (68/228) being neutral, 16.2% (37/228) agreeing, and 16.7% (38/228) strongly agreeing. There were no significant correlations with demographic and professional factors (all P>.05). CONCLUSIONS Addressing oncologists' concerns about AI requires collaborative efforts from policy makers, developers, health care professionals, and legal experts. Emphasizing transparency, human-centered design, bias mitigation, and education about AI's potential and limitations is crucial. Through close collaboration and a multidisciplinary strategy, AI can be effectively integrated into oncology, balancing benefits with ethical considerations and enhancing patient care.
Collapse
Affiliation(s)
- Ming Li
- Department of Health Policy Management, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States
| | - XiaoMin Xiong
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing University School of Medicine, Chongqing, China
| | - Bo Xu
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing University School of Medicine, Chongqing, China
- Department of Biochemistry and Molecular Biology, Key Laboratory of Breast Cancer Prevention and Therapy, Ministry of Education, National Cancer Research Center, Tianjin Medical University Cancer Institute and Hospital, Tianjin, China
| | - Conan Dickson
- Department of Health Policy Management, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States
| |
Collapse
|
13
|
Rhodius-Meester HFM, van Maurik IS, Collij LE, van Gils AM, Koikkalainen J, Tolonen A, Pijnenburg YAL, Berkhof J, Barkhof F, van de Giessen E, Lötjönen J, van der Flier WM. Computerized decision support is an effective approach to select memory clinic patients for amyloid-PET. PLoS One 2024; 19:e0303111. [PMID: 38768188 PMCID: PMC11104589 DOI: 10.1371/journal.pone.0303111] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Accepted: 04/18/2024] [Indexed: 05/22/2024] Open
Abstract
BACKGROUND The use of amyloid-PET in dementia workup is upcoming. At the same time, amyloid-PET is costly and limitedly available. While the appropriate use criteria (AUC) aim for optimal use of amyloid-PET, their limited sensitivity hinders the translation to clinical practice. Therefore, there is a need for tools that guide selection of patients for whom amyloid-PET has the most clinical utility. We aimed to develop a computerized decision support approach to select patients for amyloid-PET. METHODS We included 286 subjects (135 controls, 108 Alzheimer's disease dementia, 33 frontotemporal lobe dementia, and 10 vascular dementia) from the Amsterdam Dementia Cohort, with available neuropsychology, APOE, MRI and [18F]florbetaben amyloid-PET. In our computerized decision support approach, using supervised machine learning based on the DSI classifier, we first classified the subjects using only neuropsychology, APOE, and quantified MRI. Then, for subjects with uncertain classification (probability of correct class (PCC) < 0.75) we enriched classification by adding (hypothetical) amyloid positive (AD-like) and negative (normal) PET visual read results and assessed whether the diagnosis became more certain in at least one scenario (PPC≥0.75). If this was the case, the actual visual read result was used in the final classification. We compared the proportion of PET scans and patients diagnosed with sufficient certainty in the computerized approach with three scenarios: 1) without amyloid-PET, 2) amyloid-PET according to the AUC, and 3) amyloid-PET for all patients. RESULTS The computerized approach advised PET in n = 60(21%) patients, leading to a diagnosis with sufficient certainty in n = 188(66%) patients. This approach was more efficient than the other three scenarios: 1) without amyloid-PET, diagnostic classification was obtained in n = 155(54%), 2) applying the AUC resulted in amyloid-PET in n = 113(40%) and diagnostic classification in n = 156(55%), and 3) performing amyloid-PET in all resulted in diagnostic classification in n = 154(54%). CONCLUSION Our computerized data-driven approach selected 21% of memory clinic patients for amyloid-PET, without compromising diagnostic performance. Our work contributes to a cost-effective implementation and could support clinicians in making a balanced decision in ordering additional amyloid PET during the dementia workup.
Collapse
Affiliation(s)
- Hanneke F. M. Rhodius-Meester
- Alzheimer Center Amsterdam, Neurology, Amsterdam UMC Location VUmc, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Neuroscience, Neurodegeneration, Amsterdam, The Netherlands
- Department of Internal Medicine, Geriatric Medicine Section, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Department of Geriatric Medicine, The Memory Clinic, Oslo University Hospital, Oslo, Norway
| | - Ingrid S. van Maurik
- Alzheimer Center Amsterdam, Neurology, Amsterdam UMC Location VUmc, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Neuroscience, Neurodegeneration, Amsterdam, The Netherlands
- Epidemiology and Data Science, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Public Health, Methodology, Amsterdam, The Netherlands
| | - Lyduine E. Collij
- Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Aniek M. van Gils
- Alzheimer Center Amsterdam, Neurology, Amsterdam UMC Location VUmc, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Neuroscience, Neurodegeneration, Amsterdam, The Netherlands
| | | | | | - Yolande A. L. Pijnenburg
- Alzheimer Center Amsterdam, Neurology, Amsterdam UMC Location VUmc, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Neuroscience, Neurodegeneration, Amsterdam, The Netherlands
| | - Johannes Berkhof
- Epidemiology and Data Science, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Public Health, Methodology, Amsterdam, The Netherlands
| | - Frederik Barkhof
- Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Queen Square Institute of Neurology and Centre for Medical Image Computing, University College London, London, United Kingdom
| | - Elsmarieke van de Giessen
- Alzheimer Center Amsterdam, Neurology, Amsterdam UMC Location VUmc, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Neuroscience, Neurodegeneration, Amsterdam, The Netherlands
- Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | | | - Wiesje M. van der Flier
- Alzheimer Center Amsterdam, Neurology, Amsterdam UMC Location VUmc, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Neuroscience, Neurodegeneration, Amsterdam, The Netherlands
- Epidemiology and Data Science, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Public Health, Methodology, Amsterdam, The Netherlands
| |
Collapse
|
14
|
Pozza A, Zanella L, Castaldi B, Di Salvo G. How Will Artificial Intelligence Shape the Future of Decision-Making in Congenital Heart Disease? J Clin Med 2024; 13:2996. [PMID: 38792537 PMCID: PMC11122569 DOI: 10.3390/jcm13102996] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2024] [Revised: 05/10/2024] [Accepted: 05/13/2024] [Indexed: 05/26/2024] Open
Abstract
Improvements in medical technology have significantly changed the management of congenital heart disease (CHD), offering novel tools to predict outcomes and personalize follow-up care. By using sophisticated imaging modalities, computational models and machine learning algorithms, clinicians can experiment with unprecedented insights into the complex anatomy and physiology of CHD. These tools enable early identification of high-risk patients, thus allowing timely, tailored interventions and improved outcomes. Additionally, the integration of genetic testing offers valuable prognostic information, helping in risk stratification and treatment optimisation. The birth of telemedicine platforms and remote monitoring devices facilitates customised follow-up care, enhancing patient engagement and reducing healthcare disparities. Taking into consideration challenges and ethical issues, clinicians can make the most of the full potential of artificial intelligence (AI) to further refine prognostic models, personalize care and improve long-term outcomes for patients with CHD. This narrative review aims to provide a comprehensive illustration of how AI has been implemented as a new technological method for enhancing the management of CHD.
Collapse
Affiliation(s)
- Alice Pozza
- Paediatric Cardiology Unit, Department of Women’s and Children’s Health, University of Padua, 35122 Padova, Italy; (A.P.)
| | - Luca Zanella
- Heart Surgery, Department of Medical and Surgical Sciences, University of Bologna, 40138 Bologna, Italy
- Cardiac Surgery Unit, Department of Cardiac-Thoracic-Vascular Diseases, IRCCS Azienda Ospedaliero-Universitaria di Bologna, 40138 Bologna, Italy
| | - Biagio Castaldi
- Paediatric Cardiology Unit, Department of Women’s and Children’s Health, University of Padua, 35122 Padova, Italy; (A.P.)
| | - Giovanni Di Salvo
- Paediatric Cardiology Unit, Department of Women’s and Children’s Health, University of Padua, 35122 Padova, Italy; (A.P.)
| |
Collapse
|
15
|
Nair D, Raveendran KU. Consumer satisfaction, palliative care and artificial intelligence (AI). BMJ Support Palliat Care 2024; 14:171-177. [PMID: 38490720 DOI: 10.1136/spcare-2023-004634] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Accepted: 02/23/2024] [Indexed: 03/17/2024]
Abstract
The scope of artificial intelligence (AI) in healthcare is promising, and AI has the potential to revolutionise the field of palliative care services also. Consumer satisfaction in palliative care is a critical aspect of providing high-quality end-of-life support. It encompasses various elements that contribute to a positive experience for both patients and their families. AI-based tools and technologies can help in early identification of the beneficiaries, reduce the cost, improve the quality of care and satisfaction to the patients with chronic life-limiting illnesses. However, it is essential to ensure that AI is used ethically and in a way that complements, rather than replaces, the human touch and compassionate care, which are the core components of palliative care. This article tries to analyse the scope and challenges of improving consumer satisfaction through AI-based technology in palliative care services.
Collapse
Affiliation(s)
- Devi Nair
- Health Management, Goa Institute of Management, Sanquelim, Goa, India
| | | |
Collapse
|
16
|
Soukup T, Dean Franklin B. Quality, safety and artificial intelligence. BMJ Qual Saf 2024; 33:406-410. [PMID: 38760073 DOI: 10.1136/bmjqs-2024-017382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Accepted: 04/01/2024] [Indexed: 05/19/2024]
Affiliation(s)
- Tayana Soukup
- Department of Surgery and Cancer, Faculty of Medicine, Imperial College London, London, UK
| | - Bryony Dean Franklin
- Imperial College Healthcare NHS Trust, London, UK
- UCL School of Pharmacy, London, UK
| |
Collapse
|
17
|
Sezgin E, Sirrianni JW, Kranz K. Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls. Appl Clin Inform 2024; 15:600-611. [PMID: 38749477 PMCID: PMC11268986 DOI: 10.1055/a-2327-4121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Accepted: 05/14/2024] [Indexed: 07/26/2024] Open
Abstract
OBJECTIVE We present a proof-of-concept digital scribe system as an Emergency Department (ED) consultation call-based clinical conversation summarization pipeline to support clinical documentation, and report its performance. MATERIALS AND METHODS We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries. RESULTS The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate. DISCUSSION The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories. CONCLUSION The study provides evidence towards the potential of AI-assisted tools in assisting clinical documentation. Future work is suggested on expanding the research scope with additional language models and hybrid approaches, and comparative analysis to measure documentation burden and human factors.
Collapse
Affiliation(s)
- Emre Sezgin
- Center for Biobehavioral Health, The Abigail Wexner Research Institute at Nationwide Children's Hospital, Columbus, Ohio, United States
- The Ohio State University College of Medicine, Columbus, Ohio, United States
| | - Joseph W. Sirrianni
- IT Research and Innovation, The Abigail Wexner Research Institute at Nationwide Children's Hospital, Columbus, Ohio, United States
| | - Kelly Kranz
- Physician Consult and Transfer Center, Nationwide Children's Hospital, Columbus, Ohio, United States
| |
Collapse
|
18
|
Zeng S, Qing Q, Xu W, Yu S, Zheng M, Tan H, Peng J, Huang J. Personalized anesthesia and precision medicine: a comprehensive review of genetic factors, artificial intelligence, and patient-specific factors. Front Med (Lausanne) 2024; 11:1365524. [PMID: 38784235 PMCID: PMC11111965 DOI: 10.3389/fmed.2024.1365524] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 04/22/2024] [Indexed: 05/25/2024] Open
Abstract
Precision medicine, characterized by the personalized integration of a patient's genetic blueprint and clinical history, represents a dynamic paradigm in healthcare evolution. The emerging field of personalized anesthesia is at the intersection of genetics and anesthesiology, where anesthetic care will be tailored to an individual's genetic make-up, comorbidities and patient-specific factors. Genomics and biomarkers can provide more accurate anesthetic protocols, while artificial intelligence can simplify anesthetic procedures and reduce anesthetic risks, and real-time monitoring tools can improve perioperative safety and efficacy. The aim of this paper is to present and summarize the applications of these related fields in anesthesiology by reviewing them, exploring the potential of advanced technologies in the implementation and development of personalized anesthesia, realizing the future integration of new technologies into clinical practice, and promoting multidisciplinary collaboration between anesthesiology and disciplines such as genomics and artificial intelligence.
Collapse
Affiliation(s)
- Shiyue Zeng
- Zhuzhou Clinical College, Jishou University, Jishou, China
| | - Qi Qing
- Zhuzhou Clinical College, Jishou University, Jishou, China
| | - Wei Xu
- Department of Anesthesiology, Zhuzhou Central Hospital, Zhuzhou, China
| | - Simeng Yu
- Zhuzhou Clinical College, Jishou University, Jishou, China
| | - Mingzhi Zheng
- Department of Anesthesiology, Zhuzhou Central Hospital, Zhuzhou, China
| | - Hongpei Tan
- Department of Radiology, Third Xiangya Hospital, Central South University, Changsha, China
| | - Junmin Peng
- Department of Anesthesiology, Zhuzhou Central Hospital, Zhuzhou, China
| | - Jing Huang
- Department of Anesthesiology, Zhuzhou Central Hospital, Zhuzhou, China
| |
Collapse
|
19
|
Zhou Y, Lin CJ, Yu Q, Blais JE, Wan EYF, Lee M, Wong E, Siu DCW, Wong V, Chan EWY, Lam TW, Chui W, Wong ICK, Luo R, Chui CSL. Development and validation of risk prediction model for recurrent cardiovascular events among Chinese: the Personalized CARdiovascular DIsease risk Assessment for Chinese model. EUROPEAN HEART JOURNAL. DIGITAL HEALTH 2024; 5:363-370. [PMID: 38774379 PMCID: PMC11104455 DOI: 10.1093/ehjdh/ztae018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Revised: 12/14/2023] [Accepted: 01/30/2024] [Indexed: 05/24/2024]
Abstract
Aims Cardiovascular disease (CVD) is a leading cause of mortality, especially in developing countries. This study aimed to develop and validate a CVD risk prediction model, Personalized CARdiovascular DIsease risk Assessment for Chinese (P-CARDIAC), for recurrent cardiovascular events using machine learning technique. Methods and results Three cohorts of Chinese patients with established CVD were included if they had used any of the public healthcare services provided by the Hong Kong Hospital Authority (HA) since 2004 and categorized by their geographical locations. The 10-year CVD outcome was a composite of diagnostic or procedure codes with specific International Classification of Diseases, Ninth Revision, Clinical Modification. Multivariate imputation with chained equations and XGBoost were applied for the model development. The comparison with Thrombolysis in Myocardial Infarction Risk Score for Secondary Prevention (TRS-2°P) and Secondary Manifestations of ARTerial disease (SMART2) used the validation cohorts with 1000 bootstrap replicates. A total of 48 799, 119 672 and 140 533 patients were included in the derivation and validation cohorts, respectively. A list of 125 risk variables were used to make predictions on CVD risk, of which 8 classes of CVD-related drugs were considered interactive covariates. Model performance in the derivation cohort showed satisfying discrimination and calibration with a C statistic of 0.69. Internal validation showed good discrimination and calibration performance with C statistic over 0.6. The P-CARDIAC also showed better performance than TRS-2°P and SMART2. Conclusion Compared with other risk scores, the P-CARDIAC enables to identify unique patterns of Chinese patients with established CVD. We anticipate that the P-CARDIAC can be applied in various settings to prevent recurrent CVD events, thus reducing the related healthcare burden.
Collapse
Affiliation(s)
- Yekai Zhou
- Department of Computer Science, The University of Hong Kong, Rm 301 Chow Yei Ching Building, Pokfulam Road, Pokfulam, Hong Kong Special Administrative Region, 999077, China
| | - Celia Jiaxi Lin
- School of Nursing, The University of Hong Kong, 5/F Academic Building, 3 Sassoon Road, Pokfulam, Hong Kong Special Administrative Region, 999077, China
| | - Qiuyan Yu
- Centre for Safe Medication Practice and Research, Department of Pharmacology and Pharmacy, The University of Hong Kong, Hong Kong Special Administrative Region, 999077, China
| | - Joseph Edgar Blais
- Centre for Safe Medication Practice and Research, Department of Pharmacology and Pharmacy, The University of Hong Kong, Hong Kong Special Administrative Region, 999077, China
| | - Eric Yuk Fai Wan
- Centre for Safe Medication Practice and Research, Department of Pharmacology and Pharmacy, The University of Hong Kong, Hong Kong Special Administrative Region, 999077, China
- Department of Family Medicine and Primary Care, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Queen Mary Hospital, Hong Kong Special Administrative Region, 999077, China
| | - Marco Lee
- Centre for Safe Medication Practice and Research, Department of Pharmacology and Pharmacy, The University of Hong Kong, Hong Kong Special Administrative Region, 999077, China
| | - Emmanuel Wong
- Department of Medicine, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Queen Mary Hospital, Hong Kong Special Administrative Region, 999077, China
| | - David Chung-Wah Siu
- Department of Medicine, School of Clinical Medicine, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Queen Mary Hospital, Hong Kong Special Administrative Region, 999077, China
| | - Vincent Wong
- Department of Pharmacy, Queen Mary Hospital, Hospital Authority, Hong Kong Special Administrative Region, 999077, China
| | - Esther Wai Yin Chan
- Centre for Safe Medication Practice and Research, Department of Pharmacology and Pharmacy, The University of Hong Kong, Hong Kong Special Administrative Region, 999077, China
- Laboratory of Data Discovery for Health (D4H), Hong Kong Science Park, Hong Kong Science and Technology Park, Hong Kong Special Administrative Region, 999077, China
| | - Tak-Wah Lam
- Department of Computer Science, The University of Hong Kong, Rm 301 Chow Yei Ching Building, Pokfulam Road, Pokfulam, Hong Kong Special Administrative Region, 999077, China
| | - William Chui
- Department of Pharmacy, Queen Mary Hospital, Hospital Authority, Hong Kong Special Administrative Region, 999077, China
| | - Ian Chi Kei Wong
- Centre for Safe Medication Practice and Research, Department of Pharmacology and Pharmacy, The University of Hong Kong, Hong Kong Special Administrative Region, 999077, China
- Laboratory of Data Discovery for Health (D4H), Hong Kong Science Park, Hong Kong Science and Technology Park, Hong Kong Special Administrative Region, 999077, China
- Aston Pharmacy School, Aston University, Birmingham, B4 7ET, United Kingdom
| | - Ruibang Luo
- Department of Computer Science, The University of Hong Kong, Rm 301 Chow Yei Ching Building, Pokfulam Road, Pokfulam, Hong Kong Special Administrative Region, 999077, China
| | - Celine Sze Ling Chui
- School of Nursing, The University of Hong Kong, 5/F Academic Building, 3 Sassoon Road, Pokfulam, Hong Kong Special Administrative Region, 999077, China
- Laboratory of Data Discovery for Health (D4H), Hong Kong Science Park, Hong Kong Science and Technology Park, Hong Kong Special Administrative Region, 999077, China
- School of Public Health, The University of Hong Kong, Hong Kong Special Administrative Region, China
| |
Collapse
|
20
|
Sparrow R, Hatherley J, Oakley J, Bain C. Should the Use of Adaptive Machine Learning Systems in Medicine be Classified as Research? THE AMERICAN JOURNAL OF BIOETHICS : AJOB 2024:1-12. [PMID: 38662360 DOI: 10.1080/15265161.2024.2337429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/26/2024]
Abstract
A novel advantage of the use of machine learning (ML) systems in medicine is their potential to continue learning from new data after implementation in clinical practice. To date, considerations of the ethical questions raised by the design and use of adaptive machine learning systems in medicine have, for the most part, been confined to discussion of the so-called "update problem," which concerns how regulators should approach systems whose performance and parameters continue to change even after they have received regulatory approval. In this paper, we draw attention to a prior ethical question: whether the continuous learning that will occur in such systems after their initial deployment should be classified, and regulated, as medical research? We argue that there is a strong prima facie case that the use of continuous learning in medical ML systems should be categorized, and regulated, as research and that individuals whose treatment involves such systems should be treated as research subjects.
Collapse
|
21
|
Huguet N, Chen J, Parikh RB, Marino M, Flocke SA, Likumahuwa-Ackman S, Bekelman J, DeVoe JE. Applying Machine Learning Techniques to Implementation Science. Online J Public Health Inform 2024; 16:e50201. [PMID: 38648094 DOI: 10.2196/50201] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 11/15/2023] [Accepted: 03/14/2024] [Indexed: 04/25/2024] Open
Abstract
Machine learning (ML) approaches could expand the usefulness and application of implementation science methods in clinical medicine and public health settings. The aim of this viewpoint is to introduce a roadmap for applying ML techniques to address implementation science questions, such as predicting what will work best, for whom, under what circumstances, and with what predicted level of support, and what and when adaptation or deimplementation are needed. We describe how ML approaches could be used and discuss challenges that implementation scientists and methodologists will need to consider when using ML throughout the stages of implementation.
Collapse
Affiliation(s)
- Nathalie Huguet
- Department of Family Medicine, Oregon Health & Science University, Portland, OR, United States
- BRIDGE-C2 Implementation Science Center for Cancer Control, Oregon Health & Science University, Portland, OR, United States
| | - Jinying Chen
- Section of Preventive Medicine and Epidemiology, Department of Medicine, Boston University Chobanian & Avedisian School of Medicine, Boston, MA, United States
- Data Science Core, Boston University Chobanian & Avedisian School of Medicine, Boston, MA, United States
- iDAPT Implementation Science Center for Cancer Control, Wake Forest School of Medicine, Winston-Salem, NC, United States
| | - Ravi B Parikh
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - Miguel Marino
- Department of Family Medicine, Oregon Health & Science University, Portland, OR, United States
- BRIDGE-C2 Implementation Science Center for Cancer Control, Oregon Health & Science University, Portland, OR, United States
| | - Susan A Flocke
- Department of Family Medicine, Oregon Health & Science University, Portland, OR, United States
- BRIDGE-C2 Implementation Science Center for Cancer Control, Oregon Health & Science University, Portland, OR, United States
| | - Sonja Likumahuwa-Ackman
- Department of Family Medicine, Oregon Health & Science University, Portland, OR, United States
- BRIDGE-C2 Implementation Science Center for Cancer Control, Oregon Health & Science University, Portland, OR, United States
| | - Justin Bekelman
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
- Penn Center for Cancer Care Innovation, Abramson Cancer Center, Penn Medicine, Philadelphia, PA, United States
| | - Jennifer E DeVoe
- Department of Family Medicine, Oregon Health & Science University, Portland, OR, United States
- BRIDGE-C2 Implementation Science Center for Cancer Control, Oregon Health & Science University, Portland, OR, United States
| |
Collapse
|
22
|
Sideris K, Weir CR, Schmalfuss C, Hanson H, Pipke M, Tseng PH, Lewis N, Sallam K, Bozkurt B, Hanff T, Schofield R, Larimer K, Kyriakopoulos CP, Taleb I, Brinker L, Curry T, Knecht C, Butler JM, Stehlik J. Artificial intelligence predictive analytics in heart failure: results of the pilot phase of a pragmatic randomized clinical trial. J Am Med Inform Assoc 2024; 31:919-928. [PMID: 38341800 PMCID: PMC10990545 DOI: 10.1093/jamia/ocae017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 12/20/2023] [Accepted: 01/17/2024] [Indexed: 02/13/2024] Open
Abstract
OBJECTIVES We conducted an implementation planning process during the pilot phase of a pragmatic trial, which tests an intervention guided by artificial intelligence (AI) analytics sourced from noninvasive monitoring data in heart failure patients (LINK-HF2). MATERIALS AND METHODS A mixed-method analysis was conducted at 2 pilot sites. Interviews were conducted with 12 of 27 enrolled patients and with 13 participating clinicians. iPARIHS constructs were used for interview construction to identify workflow, communication patterns, and clinician's beliefs. Interviews were transcribed and analyzed using inductive coding protocols to identify key themes. Behavioral response data from the AI-generated notifications were collected. RESULTS Clinicians responded to notifications within 24 hours in 95% of instances, with 26.7% resulting in clinical action. Four implementation themes emerged: (1) High anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring. (2) The AI notifications required a differential and tailored balance of trust and action advice related to role. (3) Clinic experience with other home-based programs influenced utilization. (4) Responding to notifications involved significant effort, including electronic health record (EHR) review, patient contact, and consultation with other clinicians. DISCUSSION Clinician's use of AI data is a function of beliefs regarding the trustworthiness and usefulness of the data, the degree of autonomy in professional roles, and the cognitive effort involved. CONCLUSION The implementation planning analysis guided development of strategies that addressed communication technology, patient education, and EHR integration to reduce clinician and patient burden in the subsequent main randomized phase of the trial. Our results provide important insights into the unique implications of implementing AI analytics into clinical workflow.
Collapse
Affiliation(s)
- Konstantinos Sideris
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Charlene R Weir
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Carsten Schmalfuss
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | - Heather Hanson
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Matt Pipke
- PhysIQ, Inc., Chicago, IL 60563, United States
| | - Po-He Tseng
- PhysIQ, Inc., Chicago, IL 60563, United States
| | - Neil Lewis
- Cardiology Section, Medical Service, Hunter Holmes McGuire Veterans Medical Center, Richmond, VA 23249, United States
- Department of Internal Medicine, Division of Cardiovascular Disease, Virginia Commonwealth University, Richmond, VA 23249, United States
| | - Karim Sallam
- Cardiology Section, Medical Service, VA Palo Alto Health Care System, Palo Alto, CA 94304, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, Stanford University School of Medicine, Stanford, CA 94305, United States
| | - Biykem Bozkurt
- Cardiology Section, Medical Service, Michael E. DeBakey VA Medical Center, Houston, TX 77030, United States
- Section of Cardiology, Department of Medicine, Baylor College of Medicine, Houston, TX 77030, United States
| | - Thomas Hanff
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Richard Schofield
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | | | - Christos P Kyriakopoulos
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Iosif Taleb
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Lina Brinker
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| | - Tempa Curry
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | - Cheri Knecht
- Cardiology Section, Medical Service, Malcom Randall VA Medical Center, Gainesville, FL 32608, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Florida College of Medicine, Gainesville, FL 32610, United States
| | - Jorie M Butler
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Josef Stehlik
- Cardiology Section, Medical Service, George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, UT 84148, United States
- Division of Cardiovascular Medicine, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84112, United States
| |
Collapse
|
23
|
Michel J, Manns A, Boudersa S, Jaubert C, Dupic L, Vivien B, Burgun A, Campeotto F, Tsopra R. Clinical decision support system in emergency telephone triage: A scoping review of technical design, implementation and evaluation. Int J Med Inform 2024; 184:105347. [PMID: 38290244 DOI: 10.1016/j.ijmedinf.2024.105347] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2023] [Revised: 01/09/2024] [Accepted: 01/21/2024] [Indexed: 02/01/2024]
Abstract
OBJECTIVES Emergency department overcrowding could be improved by upstream telephone triage. Emergency telephone triage aims at managing and orientating adequately patients as early as possible and distributing limited supply of staff and materials. This complex task could be improved with the use of Clinical decision support systems (CDSS). The aim of this scoping review was to identify literature gaps for the future development and evaluation of CDSS for Emergency telephone triage. MATERIALS AND METHODS We present here a scoping review of CDSS designed for emergency telephone triage, and compared them in terms of functional characteristics, technical design, health care implementation and methodologies used for evaluation, following the PRISMA-ScR guidelines. RESULTS Regarding design, 19 CDSS were retrieved: 12 were knowledge based CDSS (decisional algorithms built according to guidelines or clinical expertise) and 7 were data driven (statistical, machine learning, or deep learning models). Most of them aimed at assisting nurses or non-medical staff by providing patient orientation and/or severity/priority assessment. Eleven were implemented in real life, and only three were connected to the Electronic Health Record. Regarding evaluation, CDSS were assessed through various aspects: intrinsic characteristics, impact on clinical practice or user apprehension. Only one pragmatic trial and one randomized controlled trial were conducted. CONCLUSION This review highlights the potential of a hybrid system, user tailored, flexible, connected to the electronic health record, which could work with oral, video and digital data; and the need to evaluate CDSS on intrinsic characteristics and impact on clinical practice, iteratively at each distinct stage of the IT lifecycle.
Collapse
Affiliation(s)
- Julie Michel
- SAMU 93-UF Recherche-Enseignement-Qualité, Université Paris 13, Sorbonne Paris Cité, Inserm U942, Hôpital Avicenne, 125, rue de Stalingrad, 93009 Bobigny, France
| | - Aurélia Manns
- Université Paris Cité, Sorbonne Université, Inserm, Centre de Recherche des Cordeliers, F-75006 Paris, France; Department of Medical Informatics, AP-HP, Hôpital Européen Georges-Pompidou et Hôpital Necker-Enfants Malades, F-75015 Paris, France.
| | - Sofia Boudersa
- Department of Medical Informatics, AP-HP, Hôpital Européen Georges-Pompidou et Hôpital Necker-Enfants Malades, F-75015 Paris, France
| | - Côme Jaubert
- Université Paris Cité, Sorbonne Université, Inserm, Centre de Recherche des Cordeliers, F-75006 Paris, France
| | - Laurent Dupic
- Régulation Régionale Pédiatrique, SAMU de Paris, Hôpital Necker - Enfants Malades, AP-HP, Paris, France
| | - Benoit Vivien
- Digital Health Program of Université de Paris Cité, Paris, France; Régulation Régionale Pédiatrique, SAMU de Paris, Hôpital Necker - Enfants Malades, AP-HP, Paris, France
| | - Anita Burgun
- Université Paris Cité, Sorbonne Université, Inserm, Centre de Recherche des Cordeliers, F-75006 Paris, France; Department of Medical Informatics, AP-HP, Hôpital Européen Georges-Pompidou et Hôpital Necker-Enfants Malades, F-75015 Paris, France
| | - Florence Campeotto
- Digital Health Program of Université de Paris Cité, Paris, France; Régulation Régionale Pédiatrique, SAMU de Paris, Hôpital Necker - Enfants Malades, AP-HP, Paris, France; Faculté de Pharmacie, Université de Paris Cité, Inserm UMR S1139, Paris, France
| | - Rosy Tsopra
- Université Paris Cité, Sorbonne Université, Inserm, Centre de Recherche des Cordeliers, F-75006 Paris, France; Department of Medical Informatics, AP-HP, Hôpital Européen Georges-Pompidou et Hôpital Necker-Enfants Malades, F-75015 Paris, France
| |
Collapse
|
24
|
Yangöz ŞT, Turan Kavradim S, Özer Z. Global Trends and Hotspots in Nursing Research on Decision Support Systems: A Bibliometric Analysis in CiteSpace. Comput Inform Nurs 2024; 42:207-217. [PMID: 38241720 DOI: 10.1097/cin.0000000000001090] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2024]
Abstract
Decision support systems have been widely used in healthcare in recent years; however, there is lack of evidence on global trends and hotspots. This descriptive bibliometric study aimed to analyze bibliometric patterns of decision support systems in nursing. Data were extracted from the Web of Science Core Collection. Published research articles on decision support systems in nursing were identified. Co-occurrence and co-citation analysis was performed using CiteSpace version 6.1.R2. In total, 165 articles were analyzed. A total of 358 authors and 257 institutions from 20 countries contributed to this research field. The most productive authors were Andrew Johnson, Suzanne Bakken, Alessandro Febretti, Eileen S. O'Neill, and Kathryn H. Bowles. The most productive country and institution were the United States and Duke University, respectively. The top 10 keywords were "care," "clinical decision support," "clinical decision support system," "decision support system," "electronic health record," "system," "nursing informatics," "guideline," "decision support," and "outcomes." Common themes on keywords were planning intervention, national health information infrastructure, and methodological challenge. This study will help to find potential partners, countries, and institutions for future researchers, practitioners, and scholars. Additionally, it will contribute to health policy development, evidence-based practice, and further studies for researchers, practitioners, and scholars.
Collapse
Affiliation(s)
- Şefika Tuğba Yangöz
- Author Affiliations: Department of Internal Medicine Nursing, Faculty of Health Sciences, Pamukkale University (Dr Yangöz), Denizli; and Department of Internal Medicine Nursing, Faculty of Nursing, Akdeniz University (Drs Kavradim and Özer), Antalya, Turkey
| | | | | |
Collapse
|
25
|
Hsueh J, Fritz C, Thomas CE, Reimer AP, Reisner AT, Schoenfeld D, Haimovich A, Thomas SH. Applications of Artificial Intelligence in Helicopter Emergency Medical Services: A Scoping Review. Air Med J 2024; 43:90-95. [PMID: 38490791 DOI: 10.1016/j.amj.2023.11.012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2023] [Revised: 11/15/2023] [Accepted: 11/18/2023] [Indexed: 03/17/2024]
Abstract
OBJECTIVE Recent systematic reviews of acute care medicine applications of artificial intelligence (AI) have focused on hospital and general prehospital uses. The purpose of this scoping review was to identify and describe the literature on AI use with a focus on applications in helicopter emergency medical services (HEMS). METHODS A literature search was performed with specific inclusion and exclusion criteria. Articles were grouped by characteristics such as publication year and general subject matter with categoric and temporal trend analyses. RESULTS We identified 21 records focused on the use of AI in HEMS. These applications included both clinical and triage uses and nonclinical uses. The earliest study appeared in 2006, but over one third of the identified studies have been published in 2021 or later. The passage of time has seen an increased likelihood of HEMS AI studies focusing on nonclinical issues; for each year, the likelihood of a nonclinical focus had an odds ratio of 1.3. CONCLUSION This scoping review provides overview and hypothesis-generating information regarding AI applications specific to HEMS. HEMS AI may be ultimately deployed in nonclinical arenas as much as or more than for clinical decision support. Future studies will inform future decisions as to how AI may improve HEMS systems design, asset deployment, and clinical care.
Collapse
Affiliation(s)
- Jennifer Hsueh
- Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA.
| | - Christie Fritz
- Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | | | - Andrew P Reimer
- Case Western Reserve University Frances Payne Bolton School of Nursing, Cleveland, OH; Cleveland Clinic Critical Care Transport, Cleveland, OH
| | - Andrew T Reisner
- Department of Emergency Medicine, Massachusetts General Hospital, Harvard Medical School, Boston, MA
| | - David Schoenfeld
- Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Adrian Haimovich
- Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Stephen H Thomas
- Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA; Blizard Institute, Barts and The London School of Medicine, London, United Kingdom
| |
Collapse
|
26
|
Laka M, Carter D, Merlin T. Evaluating clinical decision support software (CDSS): challenges for robust evidence generation. Int J Technol Assess Health Care 2024; 40:e16. [PMID: 38328905 DOI: 10.1017/s0266462324000059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/09/2024]
Abstract
OBJECTIVES Computerized clinical decision support software (CDSS) are digital health technologies that have been traditionally categorized as medical devices. However, the evaluation frameworks for traditional medical devices are not well adapted to assess the value and safety of CDSS. In this study, we identified a range of challenges associated with CDSS evaluation as a medical device and investigated whether and how CDSS are evaluated in Australia. METHODS Using a qualitative approach, we interviewed 11 professionals involved in the implementation and evaluation of digital health technologies at national and regional levels. Data were thematically analyzed using both data-driven (inductive) and theory-based (deductive) approaches. RESULTS Our results suggest that current CDSS evaluations have an overly narrow perspective on the risks and benefits of CDSS due to an inability to capture the impact of the technology on the sociotechnical environment. By adopting a static view of the CDSS, these evaluation frameworks are unable to discern how rapidly evolving technologies and a dynamic clinical environment can impact CDSS performance. After software upgrades, CDSS can transition from providing information to specifying diagnoses and treatments. Therefore, it is not clear how CDSS can be monitored continuously when changes in the software can directly affect patient safety. CONCLUSION Our findings emphasize the importance of taking a living health technology assessment approach to the evaluation of digital health technologies that evolve rapidly. There is a role for observational (real-world) evidence to understand the impact of changes to the technology and the sociotechnical environment on CDSS performance.
Collapse
Affiliation(s)
- Mah Laka
- Adelaide Health Technology Assessment (AHTA), School of Public Health, University of Adelaide, Adelaide, SA, Australia
| | - Drew Carter
- Adelaide Health Technology Assessment (AHTA), School of Public Health, University of Adelaide, Adelaide, SA, Australia
| | - Tracy Merlin
- Adelaide Health Technology Assessment (AHTA), School of Public Health, University of Adelaide, Adelaide, SA, Australia
| |
Collapse
|
27
|
Pietiläinen L, Hästbacka J, Bäcklund M, Selander T, Reinikainen M. A novel score for predicting 1-year mortality of intensive care patients. Acta Anaesthesiol Scand 2024; 68:195-205. [PMID: 37771172 DOI: 10.1111/aas.14336] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 08/22/2023] [Accepted: 09/18/2023] [Indexed: 09/30/2023]
Abstract
BACKGROUND We aimed to develop a simple scoring table for predicting probability of death within 1-year after admission to an intensive care unit. We analysed data on emergency admissions from the nationwide Finnish intensive care quality registry. METHODS We included first admissions of adult patients with data available on 1-year vital status (dead or alive) and all five variables included in a premorbid functional status score, which is the number of activities the person can manage independently of the following five: get out of bed, move indoors, dress, climb stairs and walk 400 m. We analysed data on patient characteristics and admission-associated factors from 2012 to 2014 to find predictors of 1-year mortality and to develop a score for predicting probability of death. We tested the performance of this score in data from 2015. We assessed the 1-year functional status score of survivors with data available. RESULTS Out of 25,261 patients, 20,628 (81.7%) patients were able to perform all five functional activities independently prior to the intensive care unit admission. At 1-year post admission, 19,625 (77.7%) patients were alive. 1-year functional status score was known for 11,011 patients and 8970 (81.5%) patients achieved functional status score 5, managing all five activities independently. The score based on age, sex, preceding functional status, type of intensive care unit admission, severity of acute illness and the most significant diagnoses predicted 1-year mortality with an area under the receiver operating characteristic curve 0.78 (95% CI, 0.76-0.79). The calibration of our prediction model was good, with calibration intercept -0.01 (-0.07 to 0.05) and calibration slope 0.96 (0.90 to 1.02). CONCLUSION Our score based on data available at intensive care unit admission predicted 1-year mortality with fairly good discrimination. Most survivors achieved good functional recovery.
Collapse
Affiliation(s)
- Laura Pietiläinen
- Department of Anaesthesiology and Intensive Care, Kuopio University Hospital, Kuopio, Finland
- University of Eastern Finland, Kuopio, Finland
| | - Johanna Hästbacka
- Department of Anesthesia and Intensive Care, Tampere University Hospital, and Tampere University, Tampere, Finland
| | - Minna Bäcklund
- Division of Intensive Care Medicine, Department of Perioperative, Intensive Care and Pain Medicine, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
| | - Tuomas Selander
- Science Service Center, Kuopio University Hospital, Kuopio, Finland
| | - Matti Reinikainen
- Department of Anaesthesiology and Intensive Care, Kuopio University Hospital, Kuopio, Finland
- University of Eastern Finland, Kuopio, Finland
| |
Collapse
|
28
|
Yang F, Yan D, Wang Z. Large-Scale assessment of ChatGPT's performance in benign and malignant bone tumors imaging report diagnosis and its potential for clinical applications. J Bone Oncol 2024; 44:100525. [PMID: 38314324 PMCID: PMC10834989 DOI: 10.1016/j.jbo.2024.100525] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Revised: 01/03/2024] [Accepted: 01/07/2024] [Indexed: 02/06/2024] Open
Abstract
Objective This study was designed to delve into the complexities involved in diagnosing of benign and malignant bone tumors and to assess the potential of AI technologies like ChatGPT in improving diagnostic accuracy and efficiency. The study also explores the few-shot learning as a method to optimize ChatGPT's performance in specialized medical domains such as benign and malignant bone tumors diagnosis. Methods A total of 1366 benign and malignant bone tumors-related imaging reports were collected and diagnosed by 25 experienced physicians. The gold standard of diagnosis was established by combining clinical, imaging and pathological principles.These reports were then input into the ChatGPT model which underwent a few-shot learning method to generate diagnostic results. The diagnostic results of the physicians and the AI model were compared to evaluate the performance of ChatGPT. An experiment was conducted to assess the influence of different radiologist's reporting styles on the model's diagnostic performance. Furthermore, in-depth analysis of misdiagnosed cases was carried out, categorizing diagnostic errors and exploring possible causes. Results The diagnostic results generated by ChatGPT showed an accuracy of 0.73, sensitivity of 0.95, and specificity of 0.58. After few-shot learning, ChatGPT demonstrated significant improvement, achieving an accuracy of 0.87, sensitivity of 0.99, and specificity of 0.73, bringing it much closer to the level of physician diagnostics. In an experiment analyzing the influence of the radiologist's reporting style, the model demonstrated higher sensitivity when interpreting reports written by high-level radiologists. In 56 benign cases, ChatGPT misdiagnosed them as malignant. Among these, 35 benign lesions- fibrous dysplasia and osteofibrous dysplasia- were incorrectly identified as metastatic tumors or osteosarcomas; 8 cases of myositis ossificans were wrongly diagnosed as extraosseous osteosarcoma. 7 cases of giant cell tumor of bone at the end of long bone were misdiagnosed as osteosarcoma by intermediate doctors. Chondroblastoma was misdiagnosed as malignant tumor in 6 cases -2 osteosarcoma and 4 chondrosarcoma-In this study, 23 osteosarcoma cases were misdiagnosed by ChatGPT as osteomyelitis; Chondrosarcoma was misdiagnosed as fibrous dysplasia or aneurysmal bone cyst in 8 cases. Four cases of spinal chordoma were misdiagnosed as spinal tuberculosis. Conclusion Our findings highlight the potential of ChatGPT in the diagnosis of benign and malignant bone tumors, offering advantages like enhanced efficiency and a reduction in missed diagnoses. However, the necessity of collaborative interactions between physicians and ChatGPT in practical settings was underscored. With an examination into AI's capacity in benign and malignant bone tumors diagnosis, this study lays the groundwork for future AI advancements in medicine. Additionally, the benefits of few-shot learning in fine-tuning ChatGPT applications in specialized fields were also demonstrated.
Collapse
Affiliation(s)
- Fan Yang
- Department of Radiation, Beijing Jishuitan Hospital,Capital Medical University, Beijing 100035, China
| | - Dong Yan
- Department of Radiation, Beijing Jishuitan Hospital,Capital Medical University, Beijing 100035, China
| | - Zhixiang Wang
- Department of Ultrasound, Beijing Friendship Hospital, Capital Medical University, Beijing 100050, China
| |
Collapse
|
29
|
Li H, Zhou M, Sun Y, Yang J, Zeng X, Qiu Y, Xia Y, Zheng Z, Yu J, Feng Y, Shi Z, Huang T, Tan L, Lin R, Li J, Fan X, Ye J, Duan H, Shi S, Shu Q. A Patient Similarity Network (CHDmap) to Predict Outcomes After Congenital Heart Surgery: Development and Validation Study. JMIR Med Inform 2024; 12:e49138. [PMID: 38297829 PMCID: PMC10850852 DOI: 10.2196/49138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Revised: 08/21/2023] [Accepted: 11/16/2023] [Indexed: 02/02/2024] Open
Abstract
Background Although evidence-based medicine proposes personalized care that considers the best evidence, it still fails to address personal treatment in many real clinical scenarios where the complexity of the situation makes none of the available evidence applicable. "Medicine-based evidence" (MBE), in which big data and machine learning techniques are embraced to derive treatment responses from appropriately matched patients in real-world clinical practice, was proposed. However, many challenges remain in translating this conceptual framework into practice. Objective This study aimed to technically translate the MBE conceptual framework into practice and evaluate its performance in providing general decision support services for outcomes after congenital heart disease (CHD) surgery. Methods Data from 4774 CHD surgeries were collected. A total of 66 indicators and all diagnoses were extracted from each echocardiographic report using natural language processing technology. Combined with some basic clinical and surgical information, the distances between each patient were measured by a series of calculation formulas. Inspired by structure-mapping theory, the fusion of distances between different dimensions can be modulated by clinical experts. In addition to supporting direct analogical reasoning, a machine learning model can be constructed based on similar patients to provide personalized prediction. A user-operable patient similarity network (PSN) of CHD called CHDmap was proposed and developed to provide general decision support services based on the MBE approach. Results Using 256 CHD cases, CHDmap was evaluated on 2 different types of postoperative prognostic prediction tasks: a binary classification task to predict postoperative complications and a multiple classification task to predict mechanical ventilation duration. A simple poll of the k-most similar patients provided by the PSN can achieve better prediction results than the average performance of 3 clinicians. Constructing logistic regression models for prediction using similar patients obtained from the PSN can further improve the performance of the 2 tasks (best area under the receiver operating characteristic curve=0.810 and 0.926, respectively). With the support of CHDmap, clinicians substantially improved their predictive capabilities. Conclusions Without individual optimization, CHDmap demonstrates competitive performance compared to clinical experts. In addition, CHDmap has the advantage of enabling clinicians to use their superior cognitive abilities in conjunction with it to make decisions that are sometimes even superior to those made using artificial intelligence models. The MBE approach can be embraced in clinical practice, and its full potential can be realized.
Collapse
Affiliation(s)
- Haomin Li
- Clinical Data Center, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Mengying Zhou
- Clinical Data Center, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
- The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China
| | - Yuhan Sun
- Clinical Data Center, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
- The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China
| | - Jian Yang
- Clinical Data Center, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
- The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China
| | - Xian Zeng
- Clinical Data Center, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
- The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China
| | - Yunxiang Qiu
- Cardiac Intensive Care Unit, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Yuanyuan Xia
- Cardiac Intensive Care Unit, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Zhijie Zheng
- Cardiac Intensive Care Unit, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Jin Yu
- Ultrasonography Department, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Yuqing Feng
- Clinical Data Center, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Zhuo Shi
- Cardiac Surgery, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Ting Huang
- Cardiac Surgery, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Linhua Tan
- Cardiac Intensive Care Unit, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Ru Lin
- Cardiac Surgery, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Jianhua Li
- Cardiac Surgery, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Xiangming Fan
- Cardiac Surgery, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Jingjing Ye
- Ultrasonography Department, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Huilong Duan
- The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, China
| | - Shanshan Shi
- Cardiac Intensive Care Unit, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| | - Qiang Shu
- Cardiac Surgery, The Children’s Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health, Hangzhou, China
| |
Collapse
|
30
|
Howard A, Aston S, Gerada A, Reza N, Bincalar J, Mwandumba H, Butterworth T, Hope W, Buchan I. Antimicrobial learning systems: an implementation blueprint for artificial intelligence to tackle antimicrobial resistance. Lancet Digit Health 2024; 6:e79-e86. [PMID: 38123255 DOI: 10.1016/s2589-7500(23)00221-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Revised: 10/17/2023] [Accepted: 10/21/2023] [Indexed: 12/23/2023]
Abstract
The proliferation of various forms of artificial intelligence (AI) brings many opportunities to improve health care. AI models can harness complex evolving data, inform and augment human actions, and learn from health outcomes such as morbidity and mortality. The global public health challenge of antimicrobial resistance (AMR) needs large-scale optimisation of antimicrobial use and wider infection care, which could be enabled by carefully constructed AI models. As AI models become increasingly useful and robust, health-care systems remain challenging places for their deployment. An implementation gap exists between the promise of AI models and their use in patient and population care. Here, we outline an adaptive implementation and maintenance framework for AI models to improve antimicrobial use and infection care as a learning system. The roles of AMR problem identification, law and regulation, organisational support, data processing, and AI development, assessment, maintenance, and scalability in the implementation of AMR-targeted AI models are considered.
Collapse
Affiliation(s)
- Alex Howard
- Department of Antimicrobial Pharmacodynamics and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK; Liverpool University Hospitals NHS Foundation Trust, Liverpool, UK.
| | - Stephen Aston
- Department of Antimicrobial Pharmacodynamics and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK; Liverpool University Hospitals NHS Foundation Trust, Liverpool, UK
| | - Alessandro Gerada
- Department of Antimicrobial Pharmacodynamics and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK; Liverpool University Hospitals NHS Foundation Trust, Liverpool, UK
| | - Nada Reza
- Department of Antimicrobial Pharmacodynamics and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK; Liverpool University Hospitals NHS Foundation Trust, Liverpool, UK
| | - Jason Bincalar
- Department of Health Data Science, University of Liverpool, Liverpool, UK; Liverpool University Hospitals NHS Foundation Trust, Liverpool, UK
| | - Henry Mwandumba
- Malawi Liverpool Wellcome Programme, Kamuzu University of Health Sciences, Blantyre, Malawi; Department of Clinical Sciences, Liverpool School of Tropical Medicine, Liverpool, UK
| | - Tom Butterworth
- Combined Intelligence for Public Health Action, NHS Cheshire and Merseyside, Warrington, UK
| | - William Hope
- Department of Antimicrobial Pharmacodynamics and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK; Liverpool University Hospitals NHS Foundation Trust, Liverpool, UK
| | - Iain Buchan
- Department of Public Health, Policy and Systems, Institute of Population Health, University of Liverpool, Liverpool, UK; Combined Intelligence for Public Health Action, NHS Cheshire and Merseyside, Warrington, UK
| |
Collapse
|
31
|
Savic-Pesic D, Chamorro N, Lopez-Rodriguez V, Daniel-Diez J, Torres Creixenti A, El Mesnaoui MI, Benavides Navas VK, Castellanos Cotte JD, Abellan Cano I, Da Costa Azevedo FA, Trenza Peñas M, Voelcker-Sala I, Villalobos F, Satue-Gracia EM, Martin-Lujan F. Validity of the Espiro Mobile Application in the Interpretation of Spirometric Patterns: An App Accuracy Study. Diagnostics (Basel) 2023; 14:29. [PMID: 38201338 PMCID: PMC10795716 DOI: 10.3390/diagnostics14010029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 12/13/2023] [Accepted: 12/14/2023] [Indexed: 01/12/2024] Open
Abstract
Spirometry is a pulmonary function test where correct interpretation of the results is crucial for accurate diagnosis of disease. There are online tools to assist in the interpretation of spirometry results; however, as yet none are validated. We evaluated the interpretation accuracy of the Espiro app using pulmonologist interpretations as the gold standard. This is an observational descriptive study in which 118 spirometry results were interpreted by the Espiro app, two pulmonologists, two primary care physicians, and two residents of a primary care training program. We determined the interpretation accuracy of the Espiro app and the concordance of the pattern and severity interpretation between the Espiro app and each of the observers using Cohen's kappa coefficient (k). We obtained a sensitivity and specificity for the Espiro app of 97.5% (95% confidence interval (CI): 86.8-99.9%) and 94.9% (95%CI: 87.4-98.6%) with pulmonologist 1 and 100% (95%CI: 91.6-100%) and 98.7% (95%CI: 92.9-99.9%) with pulmonologist 2. The concordance for the pattern interpretation was greater than k 0.907, representing almost perfect agreement. The concordance of the severity interpretation was greater than k 0.807, representing substantial to almost perfect agreement. We concluded that the Espiro app is a valid tool for spirometry interpretation.
Collapse
Affiliation(s)
- Darinka Savic-Pesic
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
- ISAC Research Group, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut IDIAP Jordi Gol, Gran Vía de Les Corts Catalanes, 591 Ático, 08007 Barcelona, Spain;
- School of Medicine and Health Sciences, Universitat Rovira i Virgili, Carrer de Sant Llorenç, 21, 43201 Reus, Spain
| | - Nuria Chamorro
- Pneumology Service, Hospital Universitari de Tarragona Joan XXII, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain
| | - Vanesa Lopez-Rodriguez
- Pneumology Service, Hospital Universitari de Tarragona Joan XXII, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain
| | - Jordi Daniel-Diez
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
| | - Anna Torres Creixenti
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
| | - Mohamed Issam El Mesnaoui
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
| | - Viviana Katherine Benavides Navas
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
| | - Jose David Castellanos Cotte
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
| | - Iván Abellan Cano
- Primary Care Unit, Sanitat Conselleria, Generalitat Valenciana, Dpto 18, Carretera de Sax s/n, 03600 Elda, Spain
| | | | - María Trenza Peñas
- Centro de Salud Aguilas Sur, Primary Care Unit, Servicio Murciano de Salud, Calle Rey Carlos III, s/n, 30880 Aguilas, Spain
| | - Iñaki Voelcker-Sala
- College of Medicine and Public Health, Flinders University, Flinders Drive, Bedford Park, SA 5042, Australia
| | - Felipe Villalobos
- ISAC Research Group, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut IDIAP Jordi Gol, Gran Vía de Les Corts Catalanes, 591 Ático, 08007 Barcelona, Spain;
| | - Eva-María Satue-Gracia
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
- Primary Care Research Support Unit Reus-Tarragona, Institut Català de la Salut, Camí de Riudoms, 53–55, 43202 Reus, Spain
| | - Francisco Martin-Lujan
- Camp de Tarragona Primary Care Unit, Institut Català de la Salut, Doctor Mallafrè Guasch, 4, 43005 Tarragona, Spain; (D.S.-P.); (E.-M.S.-G.)
- ISAC Research Group, Fundació Institut Universitari per a la Recerca a l’Atenció Primària de Salut IDIAP Jordi Gol, Gran Vía de Les Corts Catalanes, 591 Ático, 08007 Barcelona, Spain;
- School of Medicine and Health Sciences, Universitat Rovira i Virgili, Carrer de Sant Llorenç, 21, 43201 Reus, Spain
- Primary Care Research Support Unit Reus-Tarragona, Institut Català de la Salut, Camí de Riudoms, 53–55, 43202 Reus, Spain
| |
Collapse
|
32
|
Staes CJ, Beck AC, Chalkidis G, Scheese CH, Taft T, Guo JW, Newman MG, Kawamoto K, Sloss EA, McPherson JP. Design of an interface to communicate artificial intelligence-based prognosis for patients with advanced solid tumors: a user-centered approach. J Am Med Inform Assoc 2023; 31:174-187. [PMID: 37847666 PMCID: PMC10746322 DOI: 10.1093/jamia/ocad201] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Revised: 09/18/2023] [Accepted: 10/02/2023] [Indexed: 10/19/2023] Open
Abstract
OBJECTIVES To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design. MATERIALS AND METHODS Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements. RESULTS Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist. CONCLUSION User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.
Collapse
Affiliation(s)
- Catherine J Staes
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Anna C Beck
- Department of Internal Medicine, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT 84112, United States
| | - George Chalkidis
- Healthcare IT Research Department, Center for Digital Services, Hitachi Ltd., Tokyo, Japan
| | - Carolyn H Scheese
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Teresa Taft
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Jia-Wen Guo
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Michael G Newman
- Department of Population Sciences, Huntsman Cancer Institute, Salt Lake City, UT 84112, United States
| | - Kensaku Kawamoto
- Department of Biomedical Informatics, School of Medicine, University of Utah, Salt Lake City, UT 84108, United States
| | - Elizabeth A Sloss
- College of Nursing, University of Utah, Salt Lake City, UT 84112, United States
| | - Jordan P McPherson
- Department of Pharmacotherapy, College of Pharmacy, University of Utah, Salt Lake City, UT 84108, United States
- Department of Pharmacy, Huntsman Cancer Institute, Salt Lake City, UT 84112, United States
| |
Collapse
|
33
|
Farič N, Hinder S, Williams R, Ramaesh R, Bernabeu MO, van Beek E, Cresswell K. Early experiences of integrating an artificial intelligence-based diagnostic decision support system into radiology settings: a qualitative study. J Am Med Inform Assoc 2023; 31:24-34. [PMID: 37748456 PMCID: PMC10746311 DOI: 10.1093/jamia/ocad191] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Revised: 08/23/2023] [Accepted: 09/13/2023] [Indexed: 09/27/2023] Open
Abstract
OBJECTIVES Artificial intelligence (AI)-based clinical decision support systems to aid diagnosis are increasingly being developed and implemented but with limited understanding of how such systems integrate with existing clinical work and organizational practices. We explored the early experiences of stakeholders using an AI-based imaging software tool Veye Lung Nodules (VLN) aiding the detection, classification, and measurement of pulmonary nodules in computed tomography scans of the chest. MATERIALS AND METHODS We performed semistructured interviews and observations across early adopter deployment sites with clinicians, strategic decision-makers, suppliers, patients with long-term chest conditions, and academics with expertise in the use of diagnostic AI in radiology settings. We coded the data using the Technology, People, Organizations, and Macroenvironmental factors framework. RESULTS We conducted 39 interviews. Clinicians reported VLN to be easy to use with little disruption to the workflow. There were differences in patterns of use between experts and novice users with experts critically evaluating system recommendations and actively compensating for system limitations to achieve more reliable performance. Patients also viewed the tool positively. There were contextual variations in tool performance and use between different hospital sites and different use cases. Implementation challenges included integration with existing information systems, data protection, and perceived issues surrounding wider and sustained adoption, including procurement costs. DISCUSSION Tool performance was variable, affected by integration into workflows and divisions of labor and knowledge, as well as technical configuration and infrastructure. CONCLUSION The socio-organizational factors affecting performance of diagnostic AI are under-researched and require attention and further research.
Collapse
Affiliation(s)
- Nuša Farič
- Usher Institute, University of Edinburgh, Edinburgh, United Kingdom
| | - Sue Hinder
- Usher Institute, University of Edinburgh, Edinburgh, United Kingdom
| | - Robin Williams
- Institute for the Study of Science, Technology and Innovation, University of Edinburgh, Edinburgh, United Kingdom
| | - Rishi Ramaesh
- Department of Radiology, Royal Infirmary Hospital, Edinburgh, United Kingdom
| | - Miguel O Bernabeu
- Usher Institute, University of Edinburgh, Edinburgh, United Kingdom
- The Bayes Centre, University of Edinburgh, Edinburgh, United Kingdom
| | - Edwin van Beek
- Centre for Cardiovascular Science, Edinburgh Imaging and Neuroscience, University of Edinburgh, Edinburgh, United Kingdom
| | | |
Collapse
|
34
|
Sezgin E, Sirrianni J, Kranz K. Development and Evaluation of a Digital Scribe: Conversation Summarization Pipeline for Emergency Department Counseling Sessions towards Reducing Documentation Burden. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.12.06.23299573. [PMID: 38106162 PMCID: PMC10723557 DOI: 10.1101/2023.12.06.23299573] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/19/2023]
Abstract
Objective We present a proof-of-concept digital scribe system as an ED clinical conversation summarization pipeline and report its performance. Materials and Methods We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries. Results The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate. Discussion The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories. Conclusion The study provides evidence towards the potential of AI-assisted tools in reducing clinical documentation burden. Future work is suggested on expanding the research scope with larger language models, and comparative analysis to measure documentation efforts and time.
Collapse
Affiliation(s)
- Emre Sezgin
- Nationwide Children's Hospital, Columbus OH
- Ohio State University College of Medicine, Columbus OH
| | | | | |
Collapse
|
35
|
Reinders P, Augustin M, Kirsten N, Fleyder A, Otten M. Digital health interventions in dermatology-Mapping technology and study parameters of systematically identified publications. J Eur Acad Dermatol Venereol 2023; 37:2440-2449. [PMID: 37528462 DOI: 10.1111/jdv.19392] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 07/12/2023] [Indexed: 08/03/2023]
Abstract
Digital health interventions (DHI) potentially improve the efficiency and effectiveness of dermatological care. Currently, an overview clustering and characterizing the evidence on DHIs is missing. This systematic mapping of the literature aims to analyse published research on DHIs in dermatology to identify trends and gaps in research. For this purpose, a systematic search of the MEDLINE database was conducted in August 2022 to identify original publications on DHIs in dermatology. Data on country, targeted audience, DHI category, indication, outcome parameter and study design were extracted. Out of 12,009 records identified in MEDLINE, 403 studies were included in the final analysis. Studies on DHIs mainly performed in western countries, headed by the United States (n = 133), Germany (n = 32) and Spain (n = 23). Of all identified DHIs, 261 targeted healthcare providers (HCP), 66 clients (e.g. patients, caregivers, healthy individuals) and 67 both clients and HCPs. A majority of DHIs focussed on establishing a diagnosis (n = 254). Every other study analysed store-and-forward teledermatology (n = 187), followed by artificial intelligence applications for image analysis (n = 65). The most often analysed DHI category for clients was a support of health behaviour change (n = 31). Monitoring of clients was targeted by 77 studies. Skin cancer (n = 148), wounds (n = 29) and psoriasis (n = 29) were the most targeted indications by DHIs. Most studies analysed diagnostic performance (n = 166), fewer studies analysed acceptance (n = 92) and effectiveness (n = 98). Usability (n = 32) and efficiency (n = 36) were investigated only to a small extent. Studies on DHIs in dermatology have focused on teledermatology and AI applications, with an emphasis on skin cancer diagnosis. Apart from that, a range of DHIs for different user groups, purposes and indications were identified, demonstrating the broad potential for DHIs in dermatology. Further research with a wider set of outcome parameters is needed to fully understand the potential of DHIs and ensure their sustainable implementation into dermatological care.
Collapse
Affiliation(s)
- Patrick Reinders
- Institute for Health Services Research in Dermatology and Nursing (IVDP), University Medical Center Hamburg-Eppendorf (UKE), Hamburg, Germany
| | - Matthias Augustin
- Institute for Health Services Research in Dermatology and Nursing (IVDP), University Medical Center Hamburg-Eppendorf (UKE), Hamburg, Germany
| | - Natalia Kirsten
- Institute for Health Services Research in Dermatology and Nursing (IVDP), University Medical Center Hamburg-Eppendorf (UKE), Hamburg, Germany
| | - Anastasia Fleyder
- Institute for Health Services Research in Dermatology and Nursing (IVDP), University Medical Center Hamburg-Eppendorf (UKE), Hamburg, Germany
| | - Marina Otten
- Institute for Health Services Research in Dermatology and Nursing (IVDP), University Medical Center Hamburg-Eppendorf (UKE), Hamburg, Germany
| |
Collapse
|
36
|
Lee JT, Moffett AT, Maliha G, Faraji Z, Kanter GP, Weissman GE. Analysis of Devices Authorized by the FDA for Clinical Decision Support in Critical Care. JAMA Intern Med 2023; 183:1399-1401. [PMID: 37812404 PMCID: PMC10562983 DOI: 10.1001/jamainternmed.2023.5002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/03/2023] [Accepted: 07/22/2023] [Indexed: 10/10/2023]
Abstract
This case series study examines the clinical evidence cited for US Food and Drug Administration–approved clinical decision support devices for use in the critical care setting.
Collapse
Affiliation(s)
- Jessica T. Lee
- Palliative and Advanced Illness Research Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia
| | - Alexander T. Moffett
- Palliative and Advanced Illness Research Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia
| | - George Maliha
- Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia
- Division of General Internal Medicine, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia
| | - Zahra Faraji
- Palliative and Advanced Illness Research Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia
| | - Genevieve P. Kanter
- Department of Health Policy and Management, Sol Price School of Public Policy, University of Southern California, Los Angeles
- Leonard D. Schaeffer Center for Health Policy and Economics, University of Southern California, Los Angeles
| | - Gary E. Weissman
- Palliative and Advanced Illness Research Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia
- Penn Institute for Biomedical Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia
| |
Collapse
|
37
|
Smith WR, Appelbaum PS, Lebowitz MS, Gülöksüz S, Calkins ME, Kohler CG, Gur RE, Barzilay R. The Ethics of Risk Prediction for Psychosis and Suicide Attempt in Youth Mental Health. J Pediatr 2023; 263:113583. [PMID: 37353146 PMCID: PMC10828819 DOI: 10.1016/j.jpeds.2023.113583] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Revised: 06/01/2023] [Accepted: 06/16/2023] [Indexed: 06/25/2023]
Abstract
OBJECTIVE To identify potential clinical utility of polygenic risk scores (PRS) and exposomic risk scores (ERS) for psychosis and suicide attempt in youth and assess the ethical implications of these tools. STUDY DESIGN We conducted a narrative literature review of emerging findings on PRS and ERS for suicide and psychosis as well as a literature review on the ethics of PRS. We discuss the ethical implications of the emerging findings for the clinical potential of PRS and ERS. RESULTS Emerging evidence suggests that PRS and ERS may offer clinical utility in the relatively near future but that this utility will be limited to specific, narrow clinical questions, in contrast to the suggestion that population-level screening will have sweeping impact. Combining PRS and ERS might optimize prediction. This clinical utility would change the risk-benefit balance of PRS, and further empirical assessment of proposed risks would be necessary. Some concerns for PRS, such as those about counseling, privacy, and inequities, apply to ERS. ERS raise distinct ethical challenges as well, including some that involve informed consent and direct-to-consumer advertising. Both raise questions about the ethics of machine-learning/artificial intelligence approaches. CONCLUSIONS Predictive analytics using PRS and ERS may soon play a role in youth mental health settings. Our findings help educate clinicians about potential capabilities, limitations, and ethical implications of these tools. We suggest that a broader discussion with the public is needed to avoid overenthusiasm and determine regulations and guidelines for use of predictive scores.
Collapse
Affiliation(s)
- William R Smith
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA; Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA.
| | - Paul S Appelbaum
- Department of Psychiatry, Vagelos College of Physicians & Surgeons, Columbia University, New York, NY; New York State Psychiatric Institute, New York, NY
| | - Matthew S Lebowitz
- Department of Psychiatry, Vagelos College of Physicians & Surgeons, Columbia University, New York, NY
| | - Sinan Gülöksüz
- Department of Psychiatry, Yale School of Medicine, New Haven, CT; Department of Psychiatry and Neuropsychology, School for Mental Health and Neuroscience, Maastricht University Medical Center, Maastricht, The Netherlands
| | - Monica E Calkins
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
| | - Christian G Kohler
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
| | - Raquel E Gur
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA; Lifespan Brain Institute, Children's Hospital of Philadelphia and Penn Medicine, University of Pennsylvania, Philadelphia, PA; Department of Child and Adolescent Psychiatry, Children's Hospital of Philadelphia, University of Pennsylvania, Philadelphia, PA
| | - Ran Barzilay
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA; Department of Child and Adolescent Psychiatry, Children's Hospital of Philadelphia, University of Pennsylvania, Philadelphia, PA; Department of Child and Adolescent Psychiatry, Children's Hospital of Philadelphia, University of Pennsylvania, Philadelphia, PA
| |
Collapse
|
38
|
Nafees A, Khan M, Chow R, Fazelzad R, Hope A, Liu G, Letourneau D, Raman S. Evaluation of clinical decision support systems in oncology: An updated systematic review. Crit Rev Oncol Hematol 2023; 192:104143. [PMID: 37742884 DOI: 10.1016/j.critrevonc.2023.104143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 09/17/2023] [Accepted: 09/21/2023] [Indexed: 09/26/2023] Open
Abstract
With increasing reliance on technology in oncology, the impact of digital clinical decision support (CDS) tools needs to be examined. A systematic review update was conducted and peer-reviewed literature from 2016 to 2022 were included if CDS tools were used for live decision making and comparatively assessed quantitative outcomes. 3369 studies were screened and 19 were included in this updated review. Combined with a previous review of 24 studies, a total of 43 studies were analyzed. Improvements in outcomes were observed in 42 studies, and 34 of these were of statistical significance. Computerized physician order entry and clinical practice guideline systems comprise the greatest number of evaluated CDS tools (13 and 10 respectively), followed by those that utilize patient-reported outcomes (8), clinical pathway systems (8) and prescriber alerts for best-practice advisories (4). Our review indicates that CDS can improve guideline adherence, patient-centered care, and care delivery processes in oncology.
Collapse
Affiliation(s)
- Abdulwadud Nafees
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Canada
| | - Maha Khan
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Canada
| | - Ronald Chow
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Canada; Institute of Biomedical Engineering, Faculty of Applied Sciences & Engineering, University of Toronto, Toronto, Canada; Library and Information Services, Princess Margaret Cancer Centre, Toronto, Canada
| | - Rouhi Fazelzad
- Institute of Biomedical Engineering, Faculty of Applied Sciences & Engineering, University of Toronto, Toronto, Canada; Library and Information Services, Princess Margaret Cancer Centre, Toronto, Canada
| | - Andrew Hope
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Canada; Department of Radiation Oncology, University of Toronto, Toronto, Canada
| | - Geoffrey Liu
- Department of Medical Oncology, Princess Margaret Cancer Centre, University of Toronto, Toronto, Canada
| | - Daniel Letourneau
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Canada; Department of Radiation Oncology, University of Toronto, Toronto, Canada
| | - Srinivas Raman
- Radiation Medicine Program, Princess Margaret Hospital Cancer Centre, Toronto, Canada; Department of Radiation Oncology, University of Toronto, Toronto, Canada.
| |
Collapse
|
39
|
Chen Z, Liang N, Zhang H, Li H, Yang Y, Zong X, Chen Y, Wang Y, Shi N. Harnessing the power of clinical decision support systems: challenges and opportunities. Open Heart 2023; 10:e002432. [PMID: 38016787 PMCID: PMC10685930 DOI: 10.1136/openhrt-2023-002432] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Accepted: 10/31/2023] [Indexed: 11/30/2023] Open
Abstract
Clinical decision support systems (CDSSs) are increasingly integrated into healthcare settings to improve patient outcomes, reduce medical errors and enhance clinical efficiency by providing clinicians with evidence-based recommendations at the point of care. However, the adoption and optimisation of these systems remain a challenge. This review aims to provide an overview of the current state of CDSS, discussing their development, implementation, benefits, limitations and future directions. We also explore the potential for enhancing their effectiveness and provide an outlook for future developments in this field. There are several challenges in CDSS implementation, including data privacy concerns, system integration and clinician acceptance. While CDSS have demonstrated significant potential, their adoption and optimisation remain a challenge.
Collapse
Affiliation(s)
- Zhao Chen
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Ning Liang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Haili Zhang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Huizhen Li
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Yijiu Yang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Xingyu Zong
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Yaxin Chen
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Yanping Wang
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| | - Nannan Shi
- Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing, China
| |
Collapse
|
40
|
Saab R, Balachandar A, Mahdi H, Nashnoush E, Perri LX, Waldron AL, Sadeghian A, Rubenfeld G, Crowley M, Boulos MI, Murray BJ, Khosravani H. Machine-learning assisted swallowing assessment: a deep learning-based quality improvement tool to screen for post-stroke dysphagia. Front Neurosci 2023; 17:1302132. [PMID: 38130696 PMCID: PMC10734030 DOI: 10.3389/fnins.2023.1302132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Accepted: 11/07/2023] [Indexed: 12/23/2023] Open
Abstract
Introduction Post-stroke dysphagia is common and associated with significant morbidity and mortality, rendering bedside screening of significant clinical importance. Using voice as a biomarker coupled with deep learning has the potential to improve patient access to screening and mitigate the subjectivity associated with detecting voice change, a component of several validated screening protocols. Methods In this single-center study, we developed a proof-of-concept model for automated dysphagia screening and evaluated the performance of this model on training and testing cohorts. Patients were admitted to a comprehensive stroke center, where primary English speakers could follow commands without significant aphasia and participated on a rolling basis. The primary outcome was classification either as a pass or fail equivalent using a dysphagia screening test as a label. Voice data was recorded from patients who spoke a standardized set of vowels, words, and sentences from the National Institute of Health Stroke Scale. Seventy patients were recruited and 68 were included in the analysis, with 40 in training and 28 in testing cohorts, respectively. Speech from patients was segmented into 1,579 audio clips, from which 6,655 Mel-spectrogram images were computed and used as inputs for deep-learning models (DenseNet and ConvNext, separately and together). Clip-level and participant-level swallowing status predictions were obtained through a voting method. Results The models demonstrated clip-level dysphagia screening sensitivity of 71% and specificity of 77% (F1 = 0.73, AUC = 0.80 [95% CI: 0.78-0.82]). At the participant level, the sensitivity and specificity were 89 and 79%, respectively (F1 = 0.81, AUC = 0.91 [95% CI: 0.77-1.05]). Discussion This study is the first to demonstrate the feasibility of applying deep learning to classify vocalizations to detect post-stroke dysphagia. Our findings suggest potential for enhancing dysphagia screening in clinical settings. https://github.com/UofTNeurology/masa-open-source.
Collapse
Affiliation(s)
- Rami Saab
- Hurvitz Brain Sciences Program, Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Arjun Balachandar
- Hurvitz Brain Sciences Program, Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Hamza Mahdi
- Hurvitz Brain Sciences Program, Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Eptehal Nashnoush
- Hurvitz Brain Sciences Program, Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Lucas X. Perri
- Goodfellow-Waldron Initiative in Stroke Innovation and Recovery, Division of Neurology, Neurology Quality and Innovation Lab, University of Toronto, Toronto, ON, Canada
| | - Ashley L. Waldron
- Goodfellow-Waldron Initiative in Stroke Innovation and Recovery, Division of Neurology, Neurology Quality and Innovation Lab, University of Toronto, Toronto, ON, Canada
| | - Alireza Sadeghian
- Department of Computer Science, Faculty of Science, Toronto Metropolitan University, Toronto, ON, Canada
| | - Gordon Rubenfeld
- Institute of Medical Science, University of Toronto, Toronto, ON, Canada
- Interdepartmental Division of Critical Care, Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Mark Crowley
- Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada
| | - Mark I. Boulos
- Hurvitz Brain Sciences Program, Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
- Institute of Medical Science, University of Toronto, Toronto, ON, Canada
| | - Brian J. Murray
- Hurvitz Brain Sciences Program, Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Houman Khosravani
- Hurvitz Brain Sciences Program, Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
- Goodfellow-Waldron Initiative in Stroke Innovation and Recovery, Division of Neurology, Neurology Quality and Innovation Lab, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
41
|
Prasad V, Aydemir B, Kehoe IE, Kotturesh C, O’Connell A, Biebelberg B, Wang Y, Lynch JC, Pepino JA, Filbin MR, Heldt T, Reisner AT. Diagnostic suspicion bias and machine learning: Breaking the awareness deadlock for sepsis detection. PLOS DIGITAL HEALTH 2023; 2:e0000365. [PMID: 37910497 PMCID: PMC10619833 DOI: 10.1371/journal.pdig.0000365] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Accepted: 09/11/2023] [Indexed: 11/03/2023]
Abstract
Many early warning algorithms are downstream of clinical evaluation and diagnostic testing, which means that they may not be useful when clinicians fail to suspect illness and fail to order appropriate tests. Depending on how such algorithms handle missing data, they could even indicate "low risk" simply because the testing data were never ordered. We considered predictive methodologies to identify sepsis at triage, before diagnostic tests are ordered, in a busy Emergency Department (ED). One algorithm used "bland clinical data" (data available at triage for nearly every patient). The second algorithm added three yes/no questions to be answered after the triage interview. Retrospectively, we studied adult patients from a single ED between 2014-16, separated into training (70%) and testing (30%) cohorts, and a final validation cohort of patients from four EDs between 2016-2018. Sepsis was defined per the Rhee criteria. Investigational predictors were demographics and triage vital signs (downloaded from the hospital EMR); past medical history; and the auxiliary queries (answered by chart reviewers who were blinded to all data except the triage note and initial HPI). We developed L2-regularized logistic regression models using a greedy forward feature selection. There were 1164, 499, and 784 patients in the training, testing, and validation cohorts, respectively. The bland clinical data model yielded ROC AUC's 0.78 (0.76-0.81) and 0.77 (0.73-0.81), for training and testing, respectively, and ranged from 0.74-0.79 in four hospital validation. The second model which included auxiliary queries yielded 0.84 (0.82-0.87) and 0.83 (0.79-0.86), and ranged from 0.78-0.83 in four hospital validation. The first algorithm did not require clinician input but yielded middling performance. The second showed a trend towards superior performance, though required additional user effort. These methods are alternatives to predictive algorithms downstream of clinical evaluation and diagnostic testing. For hospital early warning algorithms, consideration should be given to bias and usability of various methods.
Collapse
Affiliation(s)
- Varesh Prasad
- Harvard-MIT Program in Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| | - Baturay Aydemir
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Iain E. Kehoe
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Chaya Kotturesh
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Abigail O’Connell
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Brett Biebelberg
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Yang Wang
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - James C. Lynch
- Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| | - Jeremy A. Pepino
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Michael R. Filbin
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Thomas Heldt
- Harvard-MIT Program in Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| | - Andrew T. Reisner
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States of America
| |
Collapse
|
42
|
Vicente L, Matute H. Humans inherit artificial intelligence biases. Sci Rep 2023; 13:15737. [PMID: 37789032 PMCID: PMC10547752 DOI: 10.1038/s41598-023-42384-8] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Accepted: 09/09/2023] [Indexed: 10/05/2023] Open
Abstract
Artificial intelligence recommendations are sometimes erroneous and biased. In our research, we hypothesized that people who perform a (simulated) medical diagnostic task assisted by a biased AI system will reproduce the model's bias in their own decisions, even when they move to a context without AI support. In three experiments, participants completed a medical-themed classification task with or without the help of a biased AI system. The biased recommendations by the AI influenced participants' decisions. Moreover, when those participants, assisted by the AI, moved on to perform the task without assistance, they made the same errors as the AI had made during the previous phase. Thus, participants' responses mimicked AI bias even when the AI was no longer making suggestions. These results provide evidence of human inheritance of AI bias.
Collapse
Affiliation(s)
- Lucía Vicente
- Department of Psychology, Deusto University, Avenida Universidades 24, 48007, Bilbao, Spain
| | - Helena Matute
- Department of Psychology, Deusto University, Avenida Universidades 24, 48007, Bilbao, Spain.
| |
Collapse
|
43
|
Sharma V, Joon T, Kulkarni V, Samanani S, Simpson SH, Voaklander D, Eurich D. Predicting 30-day risk from benzodiazepine/Z-drug dispensations in older adults using administrative data: A prognostic machine learning approach. Int J Med Inform 2023; 178:105177. [PMID: 37591010 DOI: 10.1016/j.ijmedinf.2023.105177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Revised: 02/11/2023] [Accepted: 08/06/2023] [Indexed: 08/19/2023]
Abstract
OBJECTIVE To develop a machine-learning (ML) model using administrative data to estimate risk of adverse outcomes within 30-days of a benzodiazepine (BZRA) dispensation in older adults for use by health departments/regulators. DESIGN, SETTING AND PARTICIPANTS This study was conducted in Alberta, Canada during 2018-2019 in Albertans 65 years of age and older. Those with any history of malignancy or palliative care were excluded. EXPOSURE Each BZRA dispensation from a community pharmacy served as the unit of analysis. MAIN OUTCOMES AND MEASURES ML algorithms were developed on 2018 administrative data to predict risk of any-cause hospitalization, emergency department visit or death within 30-days of a BZRA dispensation. Validation on 2019 administrative data was done using XGBoost to evaluate discrimination, calibration and other relevant metrics on ranked predictions. Daily and quarterly predictions were simulated on 2019 data. RESULTS 65,063 study participants were included which represented 633,333 BZRA dispensation during 2018-2019. The validation set had 314,615 dispensations linked to 55,928 all-cause outcomes representing a pre-test probability of 17.8%. C-statistic for the XGBoost model was 0.75. Measuring risk at the end of 2019, the top 0.1 percentile of predicted risk had a LR + of 40.31 translating to a post-test probability of 90%. Daily and quarterly classification simulations resulted in uninformative predictions with positive likelihood ratios less than 10 in all risk prediction categories. Previous history of admissions was ranked highest in variable importance. CONCLUSION Developing ML models using only administrative health data may not provide health regulators with sufficient informative predictions to use as decision aids for potential interventions, especially if considering daily or quarterly classifications of BZRA risks in older adults. ML models may be informative for this context if yearly classifications are preferred. Health regulators should have access to other types of data to improve ML prediction.
Collapse
Affiliation(s)
- Vishal Sharma
- 2-040 Li Ka Shing Center for Health Research Innovation, School of Public Health, University of Alberta, Edmonton, Alberta T6G 2E1, Canada
| | - Tanya Joon
- OKAKI Health Intelligence, Edmonton, Alberta, Canada
| | | | | | - Scot H Simpson
- Faculty of Pharmacy and Pharmaceutical Sciences, University of Alberta, Edmonton, Alberta T6G 2E1, Canada
| | - Don Voaklander
- School of Public Health, University of Alberta, Edmonton, Alberta T6G 2E1, Canada
| | - Dean Eurich
- 2-040 Li Ka Shing Center for Health Research Innovation, School of Public Health, University of Alberta, Edmonton, Alberta T6G 2E1, Canada.
| |
Collapse
|
44
|
Ferrández MC, Golla SSV, Eertink JJ, de Vries BM, Wiegers SE, Zwezerijnen GJC, Pieplenbosch S, Schilder L, Heymans MW, Zijlstra JM, Boellaard R. Sensitivity of an AI method for [ 18F]FDG PET/CT outcome prediction of diffuse large B-cell lymphoma patients to image reconstruction protocols. EJNMMI Res 2023; 13:88. [PMID: 37758869 PMCID: PMC10533444 DOI: 10.1186/s13550-023-01036-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Accepted: 09/19/2023] [Indexed: 09/29/2023] Open
Abstract
BACKGROUND Convolutional neural networks (CNNs), applied to baseline [18F]-FDG PET/CT maximum intensity projections (MIPs), show potential for treatment outcome prediction in diffuse large B-cell lymphoma (DLBCL). The aim of this study is to investigate the robustness of CNN predictions to different image reconstruction protocols. Baseline [18F]FDG PET/CT scans were collected from 20 DLBCL patients. EARL1, EARL2 and high-resolution (HR) protocols were applied per scan, generating three images with different image qualities. Image-based transformation was applied by blurring EARL2 and HR images to generate EARL1 compliant images using a Gaussian filter of 5 and 7 mm, respectively. MIPs were generated for each of the reconstructions, before and after image transformation. An in-house developed CNN predicted the probability of tumor progression within 2 years for each MIP. The difference in probabilities per patient was then calculated between both EARL2 and HR with respect to EARL1 (delta probabilities or ΔP). We compared these to the probabilities obtained after aligning the data with ComBat using the difference in median and interquartile range (IQR). RESULTS CNN probabilities were found to be sensitive to different reconstruction protocols (EARL2 ΔP: median = 0.09, interquartile range (IQR) = [0.06, 0.10] and HR ΔP: median = 0.1, IQR = [0.08, 0.16]). Moreover, higher resolution images (EARL2 and HR) led to higher probability values. After image-based and ComBat transformation, an improved agreement of CNN probabilities among reconstructions was found for all patients. This agreement was slightly better after image-based transformation (transformed EARL2 ΔP: median = 0.022, IQR = [0.01, 0.02] and transformed HR ΔP: median = 0.029, IQR = [0.01, 0.03]). CONCLUSION Our CNN-based outcome predictions are affected by the applied reconstruction protocols, yet in a predictable manner. Image-based harmonization is a suitable approach to harmonize CNN predictions across image reconstruction protocols.
Collapse
Affiliation(s)
- Maria C Ferrández
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands.
| | - Sandeep S V Golla
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Jakoba J Eertink
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Department of Hematology, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Bart M de Vries
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Sanne E Wiegers
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Gerben J C Zwezerijnen
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Simone Pieplenbosch
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Department of Hematology, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Louise Schilder
- Department of Internal Medicine, Amstelland Hospital, Amstelveen, The Netherlands
| | - Martijn W Heymans
- Department of Epidemiology and Data Science, Amsterdam Public Health Research Institute, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Amsterdam Public Health Research Institute, Methodology, Amsterdam, The Netherlands
| | - Josée M Zijlstra
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Department of Hematology, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Ronald Boellaard
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| |
Collapse
|
45
|
Frascarelli C, Bonizzi G, Musico CR, Mane E, Cassi C, Guerini Rocco E, Farina A, Scarpa A, Lawlor R, Reggiani Bonetti L, Caramaschi S, Eccher A, Marletta S, Fusco N. Revolutionizing Cancer Research: The Impact of Artificial Intelligence in Digital Biobanking. J Pers Med 2023; 13:1390. [PMID: 37763157 PMCID: PMC10532470 DOI: 10.3390/jpm13091390] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 09/05/2023] [Accepted: 09/12/2023] [Indexed: 09/29/2023] Open
Abstract
BACKGROUND Biobanks are vital research infrastructures aiming to collect, process, store, and distribute biological specimens along with associated data in an organized and governed manner. Exploiting diverse datasets produced by the biobanks and the downstream research from various sources and integrating bioinformatics and "omics" data has proven instrumental in advancing research such as cancer research. Biobanks offer different types of biological samples matched with rich datasets comprising clinicopathologic information. As digital pathology and artificial intelligence (AI) have entered the precision medicine arena, biobanks are progressively transitioning from mere biorepositories to integrated computational databanks. Consequently, the application of AI and machine learning on these biobank datasets holds huge potential to profoundly impact cancer research. METHODS In this paper, we explore how AI and machine learning can respond to the digital evolution of biobanks with flexibility, solutions, and effective services. We look at the different data that ranges from specimen-related data, including digital images, patient health records and downstream genetic/genomic data and resulting "Big Data" and the analytic approaches used for analysis. RESULTS These cutting-edge technologies can address the challenges faced by translational and clinical research, enhancing their capabilities in data management, analysis, and interpretation. By leveraging AI, biobanks can unlock valuable insights from their vast repositories, enabling the identification of novel biomarkers, prediction of treatment responses, and ultimately facilitating the development of personalized cancer therapies. CONCLUSIONS The integration of biobanking with AI has the potential not only to expand the current understanding of cancer biology but also to pave the way for more precise, patient-centric healthcare strategies.
Collapse
Affiliation(s)
- Chiara Frascarelli
- Division of Pathology, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy; (C.F.); (E.M.); (E.G.R.); (N.F.)
- Department of Oncology and Hemato-Oncology, University of Milan, 20122 Milan, Italy
| | - Giuseppina Bonizzi
- Biobank for Translational and Digital Medicine, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy; (G.B.); (C.R.M.); (C.C.)
| | - Camilla Rosella Musico
- Biobank for Translational and Digital Medicine, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy; (G.B.); (C.R.M.); (C.C.)
| | - Eltjona Mane
- Division of Pathology, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy; (C.F.); (E.M.); (E.G.R.); (N.F.)
| | - Cristina Cassi
- Biobank for Translational and Digital Medicine, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy; (G.B.); (C.R.M.); (C.C.)
| | - Elena Guerini Rocco
- Division of Pathology, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy; (C.F.); (E.M.); (E.G.R.); (N.F.)
- Department of Oncology and Hemato-Oncology, University of Milan, 20122 Milan, Italy
| | - Annarosa Farina
- Central Information Systems and Technology Directorate, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy;
| | - Aldo Scarpa
- Department of Diagnostics and Public Health, Section of Pathology, University of Verona, 37134 Verona, Italy; (A.S.); (S.M.)
| | - Rita Lawlor
- ARC-Net Research Centre and Department of Diagnostics and Public Health, University of Verona, 37134 Verona, Italy;
| | - Luca Reggiani Bonetti
- Section of Pathology, Department of Medical and Surgical Sciences for Children and Adults, University of Modena and Reggio Emilia, University Hospital of Modena, 41121 Modena, Italy; (L.R.B.); (S.C.)
| | - Stefania Caramaschi
- Section of Pathology, Department of Medical and Surgical Sciences for Children and Adults, University of Modena and Reggio Emilia, University Hospital of Modena, 41121 Modena, Italy; (L.R.B.); (S.C.)
| | - Albino Eccher
- Section of Pathology, Department of Medical and Surgical Sciences for Children and Adults, University of Modena and Reggio Emilia, University Hospital of Modena, 41121 Modena, Italy; (L.R.B.); (S.C.)
| | - Stefano Marletta
- Department of Diagnostics and Public Health, Section of Pathology, University of Verona, 37134 Verona, Italy; (A.S.); (S.M.)
- Division of Pathology, Humanitas Cancer Center, 95045 Catania, Italy
| | - Nicola Fusco
- Division of Pathology, IEO, European Institute of Oncology IRCCS, 20139 Milan, Italy; (C.F.); (E.M.); (E.G.R.); (N.F.)
- Department of Oncology and Hemato-Oncology, University of Milan, 20122 Milan, Italy
| |
Collapse
|
46
|
Ramgopal S, Kapes J, Alpern ER, Carroll MS, Heffernan M, Simon NJE, Florin TA, Macy ML. Perceptions of Artificial Intelligence-Assisted Care for Children With a Respiratory Complaint. Hosp Pediatr 2023; 13:802-810. [PMID: 37593809 DOI: 10.1542/hpeds.2022-007066] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/19/2023]
Abstract
OBJECTIVES To evaluate caregiver opinions on the use of artificial intelligence (AI)-assisted medical decision-making for children with a respiratory complaint in the emergency department (ED). METHODS We surveyed a sample of caregivers of children presenting to a pediatric ED with a respiratory complaint. We assessed caregiver opinions with respect to AI, defined as "specialized computer programs" that "help make decisions about the best way to care for children." We performed multivariable logistic regression to identify factors associated with discomfort with AI-assisted decision-making. RESULTS Of 279 caregivers who were approached, 254 (91.0%) participated. Most indicated they would want to know if AI was being used for their child's health care (93.5%) and were extremely or somewhat comfortable with the use of AI in deciding the need for blood (87.9%) and viral testing (87.6%), interpreting chest radiography (84.6%), and determining need for hospitalization (78.9%). In multivariable analysis, caregiver age of 30 to 37 years (adjusted odds ratio [aOR] 3.67, 95% confidence interval [CI] 1.43-9.38; relative to 18-29 years) and a diagnosis of bronchospasm (aOR 5.77, 95% CI 1.24-30.28 relative to asthma) were associated with greater discomfort with AI. Caregivers with children being admitted to the hospital (aOR 0.23, 95% CI 0.09-0.50) had less discomfort with AI. CONCLUSIONS Caregivers were receptive toward the use of AI-assisted decision-making. Some subgroups (caregivers aged 30-37 years with children discharged from the ED) demonstrated greater discomfort with AI. Engaging with these subgroups should be considered when developing AI applications for acute care.
Collapse
Affiliation(s)
- Sriram Ramgopal
- Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, Illinois
- Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Jack Kapes
- Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Elizabeth R Alpern
- Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Michael S Carroll
- Data Analytics and Reporting
- Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Marie Heffernan
- Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, Illinois
- Mary Ann & J. Milburn Smith Child Health Outcomes, Research, and Evaluation Center, Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois
| | - Norma-Jean E Simon
- Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, Illinois
- Mary Ann & J. Milburn Smith Child Health Outcomes, Research, and Evaluation Center, Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois
| | - Todd A Florin
- Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, Illinois
- Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Michelle L Macy
- Division of Emergency Medicine, Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, Illinois
- Department of Pediatrics, Ann & Robert H. Lurie Children's Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, Illinois
- Mary Ann & J. Milburn Smith Child Health Outcomes, Research, and Evaluation Center, Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, Illinois
| |
Collapse
|
47
|
Tseng YJ, Chen CJ, Chang CW. lab: an R package for generating analysis-ready data from laboratory records. PeerJ Comput Sci 2023; 9:e1528. [PMID: 37705643 PMCID: PMC10495959 DOI: 10.7717/peerj-cs.1528] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2023] [Accepted: 07/20/2023] [Indexed: 09/15/2023]
Abstract
Background Electronic health records (EHRs) play a crucial role in healthcare decision-making by giving physicians insights into disease progression and suitable treatment options. Within EHRs, laboratory test results are frequently utilized for predicting disease progression. However, processing laboratory test results often poses challenges due to variations in units and formats. In addition, leveraging the temporal information in EHRs can improve outcomes, prognoses, and diagnosis predication. Nevertheless, the irregular frequency of the data in these records necessitates data preprocessing, which can add complexity to time-series analyses. Methods To address these challenges, we developed an open-source R package that facilitates the extraction of temporal information from laboratory records. The proposed lab package generates analysis-ready time series data by segmenting the data into time-series windows and imputing missing values. Moreover, users can map local laboratory codes to the Logical Observation Identifier Names and Codes (LOINC), an international standard. This mapping allows users to incorporate additional information, such as reference ranges and related diseases. Moreover, the reference ranges provided by LOINC enable us to categorize results into normal or abnormal. Finally, the analysis-ready time series data can be further summarized using descriptive statistics and utilized to develop models using machine learning technologies. Results Using the lab package, we analyzed data from MIMIC-III, focusing on newborns with patent ductus arteriosus (PDA). We extracted time-series laboratory records and compared the differences in test results between patients with and without 30-day in-hospital mortality. We then identified significant variations in several laboratory test results 7 days after PDA diagnosis. Leveraging the time series-analysis-ready data, we trained a prediction model with the long short-term memory algorithm, achieving an area under the receiver operating characteristic curve of 0.83 for predicting 30-day in-hospital mortality in model training. These findings demonstrate the lab package's effectiveness in analyzing disease progression. Conclusions The proposed lab package simplifies and expedites the workflow involved in laboratory records extraction. This tool is particularly valuable in assisting clinical data analysts in overcoming the obstacles associated with heterogeneous and sparse laboratory records.
Collapse
Affiliation(s)
- Yi-Ju Tseng
- Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
- Computational Health Informatics Program, Boston Children’s Hospital, Boston, MA, United States of America
| | - Chun Ju Chen
- Department of Information Management, National Taiwan University, Taipei, Taiwan
| | - Chia Wei Chang
- Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
| |
Collapse
|
48
|
Sharma V, Kulkarni V, Joon T, Eurich DT, Simpson SH, Voaklander D, Wright B, Samanani S. Predicting falls-related admissions in older adults in Alberta, Canada: a machine-learning falls prevention tool developed using population administrative health data. BMJ Open 2023; 13:e071321. [PMID: 37607796 PMCID: PMC10445355 DOI: 10.1136/bmjopen-2022-071321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Accepted: 07/26/2023] [Indexed: 08/24/2023] Open
Abstract
OBJECTIVE To construct a machine-learning (ML) model for health systems with organised falls prevention programmes to identify older adults at risk for fall-related admissions. DESIGN This prognostic study used population-level administrative health data to develop an ML prediction model. SETTING This study took place in Alberta, Canada during 2018-2019. PARTICIPANTS Albertans aged 65 and older with at least one prior admission. Those with palliative conditions or emigrated out of Alberta were excluded. EXPOSURE Unit of analysis was the individual person. MAIN OUTCOMES/MEASURES We identified fall-related admissions. A CatBoost model was developed on 2018 data to predict risk of fall-related emergency department visits or hospitalisations. Temporal validation was done using 2019 data to evaluate model performance. We reported discrimination, calibration and other relevant metrics measured at the end of 2019 on both ranked predictions and predicted probability thresholds. A cost-savings simulation was performed using 2019 data. RESULTS Final number of study participants was 224 445. The validation set had 203 584 participants with 19 389 fall-related events (9.5% pretest probability) and an ML model c-statistic of 0.70. The highest ranked predictions had post-test probabilities ranging from 40% to 50%. Net benefit analysis presented mixed results with some net benefit using the ML model in the 6%-30% range. The top 50 percentile of predicted risks represented nearly $C60 million in health system costs related to falls. Intervening on the top 25 or 50 percentiles of predicted risk could realise substantial (up to $C16 million) savings. CONCLUSION ML prediction models based on population-level administrative data can assist health systems with fall prevention programmes identify older adults at risk of fall-related admissions and reduce costs. ML predictions based on ranked predictions or probability thresholds could guide subsequent interventions to mitigate fall risks. Increased access to diverse forms of data could improve ML performance and further reduce costs.
Collapse
Affiliation(s)
- Vishal Sharma
- School of Public Health, University of Alberta, Edmonton, Alberta, Canada
| | | | | | - Dean T Eurich
- School of Public Health, University of Alberta, Edmonton, Alberta, Canada
| | - Scot H Simpson
- Faculty of Pharmacy and Pharmaceutical Sciences, University of Alberta, Edmonton, Alberta, Canada
| | - Don Voaklander
- School of Public Health, University of Alberta, Edmonton, Alberta, Canada
| | - Bruce Wright
- Island Medical Program, University of Victoria, Victoria, British Columbia, Canada
| | | |
Collapse
|
49
|
Ferrández MC, Golla SSV, Eertink JJ, de Vries BM, Lugtenburg PJ, Wiegers SE, Zwezerijnen GJC, Pieplenbosch S, Kurch L, Hüttmann A, Hanoun C, Dührsen U, de Vet HCW, Zijlstra JM, Boellaard R. An artificial intelligence method using FDG PET to predict treatment outcome in diffuse large B cell lymphoma patients. Sci Rep 2023; 13:13111. [PMID: 37573446 PMCID: PMC10423266 DOI: 10.1038/s41598-023-40218-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Accepted: 08/07/2023] [Indexed: 08/14/2023] Open
Abstract
Convolutional neural networks (CNNs) may improve response prediction in diffuse large B-cell lymphoma (DLBCL). The aim of this study was to investigate the feasibility of a CNN using maximum intensity projection (MIP) images from 18F-fluorodeoxyglucose (18F-FDG) positron emission tomography (PET) baseline scans to predict the probability of time-to-progression (TTP) within 2 years and compare it with the International Prognostic Index (IPI), i.e. a clinically used score. 296 DLBCL 18F-FDG PET/CT baseline scans collected from a prospective clinical trial (HOVON-84) were analysed. Cross-validation was performed using coronal and sagittal MIPs. An external dataset (340 DLBCL patients) was used to validate the model. Association between the probabilities, metabolic tumour volume and Dmaxbulk was assessed. Probabilities for PET scans with synthetically removed tumors were also assessed. The CNN provided a 2-year TTP prediction with an area under the curve (AUC) of 0.74, outperforming the IPI-based model (AUC = 0.68). Furthermore, high probabilities (> 0.6) of the original MIPs were considerably decreased after removing the tumours (< 0.4, generally). These findings suggest that MIP-based CNNs are able to predict treatment outcome in DLBCL.
Collapse
Affiliation(s)
- Maria C Ferrández
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, The Netherlands.
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands.
| | - Sandeep S V Golla
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Jakoba J Eertink
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Department of Hematology, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Bart M de Vries
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Pieternella J Lugtenburg
- Department of Hematology, Erasmus MC Cancer Institute, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Sanne E Wiegers
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Gerben J C Zwezerijnen
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Simone Pieplenbosch
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Department of Hematology, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Lars Kurch
- Department of Nuclear Medicine, Clinic and Polyclinic for Nuclear Medicine, University of Leipzig, Leipzig, Germany
| | - Andreas Hüttmann
- Department of Hematology, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
| | - Christine Hanoun
- Department of Hematology, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
| | - Ulrich Dührsen
- Department of Hematology, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
| | - Henrica C W de Vet
- Department of Epidemiology and Data Science, Amsterdam Public Health Research Institute, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Department of Methodology, Amsterdam Public Health Research Institute, Methodology, Amsterdam, The Netherlands
| | - Josée M Zijlstra
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Department of Hematology, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Ronald Boellaard
- Cancer Center Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| |
Collapse
|
50
|
Graf L, Tesch F, Gräßer F, Harst L, Siegels D, Schmitt J, Abraham S. Acceptance of a digital therapy recommender system for psoriasis. BMC Med Inform Decis Mak 2023; 23:150. [PMID: 37542251 PMCID: PMC10401871 DOI: 10.1186/s12911-023-02246-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2022] [Accepted: 07/20/2023] [Indexed: 08/06/2023] Open
Abstract
BACKGROUND About 2% of the German population are affected by psoriasis. A growing number of cost-intensive systemic treatments are available. Surveys have shown high proportions of patients with moderate to severe psoriasis are not adequately treated despite a high disease burden. Digital therapy recommendation systems (TRS) may help implement guideline-based treatment. However, little is known about the acceptance of such clinical decision support systems (CDSSs). Therefore, the aim of the study was to access the acceptance of a prototypical TRS demonstrator. METHODS Three scenarios (potential test patients with psoriasis but different sociodemographic and clinical characteristics, previous treatments, desire to have children, and multiple comorbidities) were designed in the demonstrator. The TRS demonstrator and test patients were presented to a random sample of 76 dermatologists attending a national dermatology conference in a cross-sectional face-to-face survey with case vignettes. The dermatologist were asked to rate the demonstrator by system usability scale (SUS), whether they would use it for certain patients populations and barriers of usage. Reasons for potential usage of the TRS demonstrator were tested via a Poisson regression with robust standard errors. RESULTS Acceptance of the TRS was highest for patients eligible for systemic therapy (82%). 50% of participants accepted the system for patients with additional comorbidities and 43% for patients with special subtypes of psoriasis. Dermatologists in the outpatient sector or with many patients per week were less willing to use the TRS for patients with special psoriasis-subtypes. Dermatologists rated the demonstrator as acceptable with an mean SUS of 76.8. Participants whose SUS was 10 points above average were 27% more likely to use TRS for special psoriasis-subtypes. The main barrier in using the TRS was time demand (47.4%). Participants who perceived time as an obstacle were 22.3% less willing to use TRS with systemic therapy patients. 27.6% of physicians stated that they did not understand exactly how the recommendation was generated by the TRS, with no effect on the preparedness to use the system. CONCLUSION The considerably high acceptance and the preparedness to use the psoriasis CDSS suggests that a TRS appears to be implementable in routine healthcare and may improve clinical care. Main barrier is the additional time demand posed on dermatologists in a busy clinical setting. Therefore, it will be a major challenge to identify a limited set of variables that still allows a valid recommendation with precise prediction of the patient-individual benefits and harms.
Collapse
Affiliation(s)
- Lisa Graf
- Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Faculty of Medicine Carl Gustav Carus, TU Dresden, Dresden, Germany
| | - Falko Tesch
- Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Faculty of Medicine Carl Gustav Carus, TU Dresden, Dresden, Germany
| | - Felix Gräßer
- Institute of Biomedical Engineering, TU Dresden, Dresden, Germany
| | - Lorenz Harst
- Center of Evidence-Based Healthcare, Branch Office at the Medical Campus Chemnitz, University Hospital Carl Gustav Carus and Faculty of Medicine Carl Gustav Carus, TU Dresden, Dresden, Germany
| | - Doreen Siegels
- Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Faculty of Medicine Carl Gustav Carus, TU Dresden, Dresden, Germany
| | - Jochen Schmitt
- Center for Evidence-Based Healthcare, University Hospital Carl Gustav Carus and Faculty of Medicine Carl Gustav Carus, TU Dresden, Dresden, Germany
| | - Susanne Abraham
- Department of Dermatology, Faculty of Medicine Carl Gustav Carus, University Hospital Carl Gustav Carus, TU Dresden, 01307, Dresden, Germany.
| |
Collapse
|