1
|
Hosseini MS, Bejnordi BE, Trinh VQH, Chan L, Hasan D, Li X, Yang S, Kim T, Zhang H, Wu T, Chinniah K, Maghsoudlou S, Zhang R, Zhu J, Khaki S, Buin A, Chaji F, Salehi A, Nguyen BN, Samaras D, Plataniotis KN. Computational pathology: A survey review and the way forward. J Pathol Inform 2024; 15:100357. [PMID: 38420608 PMCID: PMC10900832 DOI: 10.1016/j.jpi.2023.100357] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 12/21/2023] [Accepted: 12/23/2023] [Indexed: 03/02/2024] Open
Abstract
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
Collapse
Affiliation(s)
- Mahdi S Hosseini
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | | | - Vincent Quoc-Huy Trinh
- Institute for Research in Immunology and Cancer of the University of Montreal, Montreal, QC H3T 1J4, Canada
| | - Lyndon Chan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Danial Hasan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Xingwen Li
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Stephen Yang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Taehyo Kim
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Haochen Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Theodore Wu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Kajanan Chinniah
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Sina Maghsoudlou
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ryan Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Jiadai Zhu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Samir Khaki
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Andrei Buin
- Huron Digitial Pathology, St. Jacobs, ON N0B 2N0, Canada
| | - Fatemeh Chaji
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ala Salehi
- Department of Electrical and Computer Engineering, University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Bich Ngoc Nguyen
- University of Montreal Hospital Center, Montreal, QC H2X 0C2, Canada
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY 11794, United States
| | - Konstantinos N Plataniotis
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| |
Collapse
|
2
|
Chen S, Yu J, Chamouni S, Wang Y, Li Y. Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions. BMC Med 2024; 22:354. [PMID: 39218895 PMCID: PMC11367811 DOI: 10.1186/s12916-024-03566-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/22/2024] [Accepted: 08/19/2024] [Indexed: 09/04/2024] Open
Abstract
The integration of machine learning (ML) and artificial intelligence (AI) techniques in life-course epidemiology offers remarkable opportunities to advance our understanding of the complex interplay between biological, social, and environmental factors that shape health trajectories across the lifespan. This perspective summarizes the current applications, discusses future potential and challenges, and provides recommendations for harnessing ML and AI technologies to develop innovative public health solutions. ML and AI have been increasingly applied in epidemiological studies, demonstrating their ability to handle large, complex datasets, identify intricate patterns and associations, integrate multiple and multimodal data types, improve predictive accuracy, and enhance causal inference methods. In life-course epidemiology, these techniques can help identify sensitive periods and critical windows for intervention, model complex interactions between risk factors, predict individual and population-level disease risk trajectories, and strengthen causal inference in observational studies. By leveraging the five principles of life-course research proposed by Elder and Shanahan-lifespan development, agency, time and place, timing, and linked lives-we discuss a framework for applying ML and AI to uncover novel insights and inform targeted interventions. However, the successful integration of these technologies faces challenges related to data quality, model interpretability, bias, privacy, and equity. To fully realize the potential of ML and AI in life-course epidemiology, fostering interdisciplinary collaborations, developing standardized guidelines, advocating for their integration in public health decision-making, prioritizing fairness, and investing in training and capacity building are essential. By responsibly harnessing the power of ML and AI, we can take significant steps towards creating healthier and more equitable futures across the life course.
Collapse
Affiliation(s)
- Shanquan Chen
- Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, Keppel Street, London, WC1E 7HT, UK.
| | - Jiazhou Yu
- Department of Medicine and Therapeutics, The Chinese University of Hong Kong, Hong Kong SAR, China
| | - Sarah Chamouni
- Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, Keppel Street, London, WC1E 7HT, UK
| | - Yuqi Wang
- Department of Computer Science, University College London, London, WC1E 6BT, UK
| | - Yunfei Li
- Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, 171 64, Sweden.
| |
Collapse
|
3
|
Miljković N, Milenić N, Popović NB, Sodnik J. Data augmentation for generating synthetic electrogastrogram time series. Med Biol Eng Comput 2024; 62:2879-2891. [PMID: 38705957 PMCID: PMC11330405 DOI: 10.1007/s11517-024-03112-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2023] [Accepted: 04/25/2024] [Indexed: 05/07/2024]
Abstract
To address an emerging need for large number of diverse datasets for rigor evaluation of signal processing techniques, we developed and evaluated a new method for generating synthetic electrogastrogram time series. We used electrogastrography (EGG) data from an open database to set model parameters and statistical tests to evaluate synthesized data. Additionally, we illustrated method customization for generating artificial EGG time series alterations caused by the simulator sickness. Proposed data augmentation method generates synthetic EGG data with specified duration, sampling frequency, recording state (postprandial or fasting state), overall noise and breathing artifact injection, and pauses in the gastric rhythm (arrhythmia occurrence) with statistically significant difference between postprandial and fasting states in > 70% cases while not accounting for individual differences. Features obtained from the synthetic EGG signal resembling simulator sickness occurrence displayed expected trends. The code for generation of synthetic EGG time series is not only freely available and can be further customized to assess signal processing algorithms but also may be used to increase data diversity for training artificial intelligence (AI) algorithms. The proposed approach is customized for EGG data synthesis but can be easily utilized for other biosignals with similar nature such as electroencephalogram.
Collapse
Affiliation(s)
- Nadica Miljković
- University of Belgrade-School of Electrical Engineering, Bulevar Kralja Aleksandra 73, 11000, Belgrade, Serbia.
- Faculty of Electrical Engineering, University of Ljubljana, Tržaška Cesta 25, 1000, Ljubljana, Slovenia.
| | - Nikola Milenić
- University of Belgrade-School of Electrical Engineering, Bulevar Kralja Aleksandra 73, 11000, Belgrade, Serbia
| | - Nenad B Popović
- University of Belgrade-School of Electrical Engineering, Bulevar Kralja Aleksandra 73, 11000, Belgrade, Serbia
| | - Jaka Sodnik
- Faculty of Electrical Engineering, University of Ljubljana, Tržaška Cesta 25, 1000, Ljubljana, Slovenia
| |
Collapse
|
4
|
Bluethgen C, Chambon P, Delbrouck JB, van der Sluijs R, Połacin M, Zambrano Chaves JM, Abraham TM, Purohit S, Langlotz CP, Chaudhari AS. A vision-language foundation model for the generation of realistic chest X-ray images. Nat Biomed Eng 2024:10.1038/s41551-024-01246-y. [PMID: 39187663 DOI: 10.1038/s41551-024-01246-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Accepted: 07/28/2024] [Indexed: 08/28/2024]
Abstract
The paucity of high-quality medical imaging datasets could be mitigated by machine learning models that generate compositionally diverse images that faithfully represent medical concepts and pathologies. However, large vision-language models are trained on natural images, and the diversity distribution of the generated images substantially differs from that of medical images. Moreover, medical language involves specific and semantically rich vocabulary. Here we describe a domain-adaptation strategy for large vision-language models that overcomes distributional shifts. Specifically, by leveraging publicly available datasets of chest X-ray images and the corresponding radiology reports, we adapted a latent diffusion model pre-trained on pairs of natural images and text descriptors to generate diverse and visually plausible synthetic chest X-ray images (as confirmed by board-certified radiologists) whose appearance can be controlled with free-form medical text prompts. The domain-adaptation strategy for the text-conditioned synthesis of medical images can be used to augment training datasets and is a viable alternative to the sharing of real medical images for model training and fine-tuning.
Collapse
Affiliation(s)
- Christian Bluethgen
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA.
- Department of Radiology, Stanford University, Palo Alto, CA, USA.
- Diagnostic and Interventional Radiology, University Hospital Zurich, University of Zurich, Zurich, Switzerland.
| | - Pierre Chambon
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Palo Alto, CA, USA
| | - Jean-Benoit Delbrouck
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Palo Alto, CA, USA
| | - Rogier van der Sluijs
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Palo Alto, CA, USA
| | - Małgorzata Połacin
- Department of Radiology, Stanford University, Palo Alto, CA, USA
- Diagnostic and Interventional Radiology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Juan Manuel Zambrano Chaves
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA
- Department of Biomedical Data Science, Stanford University, Palo Alto, CA, USA
| | | | | | - Curtis P Langlotz
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Palo Alto, CA, USA
- Department of Biomedical Data Science, Stanford University, Palo Alto, CA, USA
| | - Akshay S Chaudhari
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Palo Alto, CA, USA
- Department of Biomedical Data Science, Stanford University, Palo Alto, CA, USA
| |
Collapse
|
5
|
Peng W, Bosschieter T, Ouyang J, Paul R, Sullivan EV, Pfefferbaum A, Adeli E, Zhao Q, Pohl KM. Metadata-conditioned generative models to synthesize anatomically-plausible 3D brain MRIs. Med Image Anal 2024; 98:103325. [PMID: 39208560 DOI: 10.1016/j.media.2024.103325] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2023] [Revised: 08/06/2024] [Accepted: 08/20/2024] [Indexed: 09/04/2024]
Abstract
Recent advances in generative models have paved the way for enhanced generation of natural and medical images, including synthetic brain MRIs. However, the mainstay of current AI research focuses on optimizing synthetic MRIs with respect to visual quality (such as signal-to-noise ratio) while lacking insights into their relevance to neuroscience. To generate high-quality T1-weighted MRIs relevant for neuroscience discovery, we present a two-stage Diffusion Probabilistic Model (called BrainSynth) to synthesize high-resolution MRIs conditionally-dependent on metadata (such as age and sex). We then propose a novel procedure to assess the quality of BrainSynth according to how well its synthetic MRIs capture macrostructural properties of brain regions and how accurately they encode the effects of age and sex. Results indicate that more than half of the brain regions in our synthetic MRIs are anatomically plausible, i.e., the effect size between real and synthetic MRIs is small relative to biological factors such as age and sex. Moreover, the anatomical plausibility varies across cortical regions according to their geometric complexity. As is, the MRIs generated by BrainSynth significantly improve the training of a predictive model to identify accelerated aging effects in an independent study. These results indicate that our model accurately capture the brain's anatomical information and thus could enrich the data of underrepresented samples in a study. The code of BrainSynth will be released as part of the MONAI project at https://github.com/Project-MONAI/GenerativeModels.
Collapse
Affiliation(s)
- Wei Peng
- Department of Psychiatry & Behavioral Sciences, Stanford University, Stanford, CA 94305, United States of America
| | - Tomas Bosschieter
- Institute for Computational and Mathematical Engineering, Stanford University, Stanford, CA 94305, United States of America
| | - Jiahong Ouyang
- Department of Electrical Engineering, Stanford University, Stanford, CA 94305, United States of America
| | - Robert Paul
- Missouri Institute of Mental Health, University of Missouri, St. Louis, MO 63121, United States of America
| | - Edith V Sullivan
- Department of Psychiatry & Behavioral Sciences, Stanford University, Stanford, CA 94305, United States of America
| | - Adolf Pfefferbaum
- Center for Health Sciences, SRI International, Menlo Park, CA 94025, United States of America
| | - Ehsan Adeli
- Department of Psychiatry & Behavioral Sciences, Stanford University, Stanford, CA 94305, United States of America; Department of Computer Science, Stanford University, Stanford, CA 94305, United States of America
| | - Qingyu Zhao
- Department of Radiology, Weill Cornell Medicine, New York, NY 10065, United States of America.
| | - Kilian M Pohl
- Department of Psychiatry & Behavioral Sciences, Stanford University, Stanford, CA 94305, United States of America; Department of Electrical Engineering, Stanford University, Stanford, CA 94305, United States of America.
| |
Collapse
|
6
|
Achterberg JL, Haas MR, Spruit MR. On the evaluation of synthetic longitudinal electronic health records. BMC Med Res Methodol 2024; 24:181. [PMID: 39143466 PMCID: PMC11323671 DOI: 10.1186/s12874-024-02304-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2024] [Accepted: 08/07/2024] [Indexed: 08/16/2024] Open
Abstract
BACKGROUND Synthetic Electronic Health Records (EHRs) are becoming increasingly popular as a privacy enhancing technology. However, for longitudinal EHRs specifically, little research has been done into how to properly evaluate synthetically generated samples. In this article, we provide a discussion on existing methods and recommendations when evaluating the quality of synthetic longitudinal EHRs. METHODS We recommend to assess synthetic EHR quality through similarity to real EHRs in low-dimensional projections, accuracy of a classifier discriminating synthetic from real samples, performance of synthetic versus real trained algorithms in clinical tasks, and privacy risk through risk of attribute inference. For each metric we discuss strengths and weaknesses, next to showing how it can be applied on a longitudinal dataset. RESULTS To support the discussion on evaluation metrics, we apply discussed metrics on a dataset of synthetic EHRs generated from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) repository. CONCLUSIONS The discussion on evaluation metrics provide guidance for researchers on how to use and interpret different metrics when evaluating the quality of synthetic longitudinal EHRs.
Collapse
Affiliation(s)
- Jim L Achterberg
- Public Health and Primary Care, Health Campus The Hague, Leiden University Medical Center, Albinusdreef 2, Leiden, South-Holland, 2333ZA, Netherlands.
| | - Marcel R Haas
- Public Health and Primary Care, Health Campus The Hague, Leiden University Medical Center, Albinusdreef 2, Leiden, South-Holland, 2333ZA, Netherlands
| | - Marco R Spruit
- Public Health and Primary Care, Health Campus The Hague, Leiden University Medical Center, Albinusdreef 2, Leiden, South-Holland, 2333ZA, Netherlands
- Leiden Institute of Advanced Computer Science, Leiden University, Einsteinweg 55, Leiden, South-Holland, 2333CC, Netherlands
| |
Collapse
|
7
|
Viswanathan VS, Parmar V, Madabhushi A. Towards equitable AI in oncology. Nat Rev Clin Oncol 2024; 21:628-637. [PMID: 38849530 DOI: 10.1038/s41571-024-00909-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/21/2024] [Indexed: 06/09/2024]
Abstract
Artificial intelligence (AI) stands at the threshold of revolutionizing clinical oncology, with considerable potential to improve early cancer detection and risk assessment, and to enable more accurate personalized treatment recommendations. However, a notable imbalance exists in the distribution of the benefits of AI, which disproportionately favour those living in specific geographical locations and in specific populations. In this Perspective, we discuss the need to foster the development of equitable AI tools that are both accurate in and accessible to a diverse range of patient populations, including those in low-income to middle-income countries. We also discuss some of the challenges and potential solutions in attaining equitable AI, including addressing the historically limited representation of diverse populations in existing clinical datasets and the use of inadequate clinical validation methods. Additionally, we focus on extant sources of inequity including the type of model approach (such as deep learning, and feature engineering-based methods), the implications of dataset curation strategies, the need for rigorous validation across a variety of populations and settings, and the risk of introducing contextual bias that comes with developing tools predominantly in high-income countries.
Collapse
Affiliation(s)
| | - Vani Parmar
- Department of Breast Surgical Oncology, Punyashlok Ahilyadevi Holkar Head & Neck Cancer Institute of India, Mumbai, India
| | - Anant Madabhushi
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA, USA.
- Atlanta Veterans Administration Medical Center, Atlanta, GA, USA.
| |
Collapse
|
8
|
Tong Y, Hu Z, Wang H, Huang J, Zhan Y, Chai W, Deng Y, Yuan Y, Shen K, Wang Y, Chen X, Yu J. Anti-HER2 therapy response assessment for guiding treatment (de-)escalation in early HER2-positive breast cancer using a novel deep learning radiomics model. Eur Radiol 2024; 34:5477-5486. [PMID: 38329503 PMCID: PMC11255056 DOI: 10.1007/s00330-024-10609-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2023] [Revised: 11/24/2023] [Accepted: 01/01/2024] [Indexed: 02/09/2024]
Abstract
OBJECTIVES Anti-HER2 targeted therapy significantly reduces risk of relapse in HER2 + breast cancer. New measures are needed for a precise risk stratification to guide (de-)escalation of anti-HER2 strategy. METHODS A total of 726 HER2 + cases who received no/single/dual anti-HER2 targeted therapies were split into three respective cohorts. A deep learning model (DeepTEPP) based on preoperative breast magnetic resonance (MR) was developed. Patients were scored and categorized into low-, moderate-, and high-risk groups. Recurrence-free survival (RFS) was compared in patients with different risk groups according to the anti-HER2 treatment they received, to validate the value of DeepTEPP in predicting treatment efficacy and guiding anti-HER2 strategy. RESULTS DeepTEPP was capable of risk stratification and guiding anti-HER2 treatment strategy: DeepTEPP-Low patients (60.5%) did not derive significant RFS benefit from trastuzumab (p = 0.144), proposing an anti-HER2 de-escalation. DeepTEPP-Moderate patients (19.8%) significantly benefited from trastuzumab (p = 0.048), but did not obtain additional improvements from pertuzumab (p = 0.125). DeepTEPP-High patients (19.7%) significantly benefited from dual HER2 blockade (p = 0.045), suggesting an anti-HER2 escalation. CONCLUSIONS DeepTEPP represents a pioneering MR-based deep learning model that enables the non-invasive prediction of adjuvant anti-HER2 effectiveness, thereby providing valuable guidance for anti-HER2 (de-)escalation strategies. DeepTEPP provides an important reference for choosing the appropriate individualized treatment in HER2 + breast cancer patients, warranting prospective validation. CLINICAL RELEVANCE STATEMENT We built an MR-based deep learning model DeepTEPP, which enables the non-invasive prediction of adjuvant anti-HER2 effectiveness, thus guiding anti-HER2 (de-)escalation strategies in early HER2-positive breast cancer patients. KEY POINTS • DeepTEPP is able to predict anti-HER2 effectiveness and to guide treatment (de-)escalation. • DeepTEPP demonstrated an impressive prognostic efficacy for recurrence-free survival and overall survival. • To our knowledge, this is one of the very few, also the largest study to test the efficacy of a deep learning model extracted from breast MR images on HER2-positive breast cancer survival and anti-HER2 therapy effectiveness prediction.
Collapse
Affiliation(s)
- Yiwei Tong
- Department of General Surgery, Comprehensive Breast Health Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, 197 Ruijin Er Road, Shanghai, 200025, China
| | - Zhaoyu Hu
- School of Information Science and Technology, Fudan University, No. 220, Handan Road, Yangpu District, Shanghai, 200433, China
| | - Haoyu Wang
- Department of General Surgery, Comprehensive Breast Health Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, 197 Ruijin Er Road, Shanghai, 200025, China
| | - Jiahui Huang
- Department of General Surgery, Comprehensive Breast Health Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, 197 Ruijin Er Road, Shanghai, 200025, China
| | - Ying Zhan
- Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Weimin Chai
- Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Yinhui Deng
- School of Information Science and Technology, Fudan University, No. 220, Handan Road, Yangpu District, Shanghai, 200433, China
| | - Ying Yuan
- Department of Radiology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Kunwei Shen
- Department of General Surgery, Comprehensive Breast Health Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, 197 Ruijin Er Road, Shanghai, 200025, China
| | - Yuanyuan Wang
- School of Information Science and Technology, Fudan University, No. 220, Handan Road, Yangpu District, Shanghai, 200433, China
| | - Xiaosong Chen
- Department of General Surgery, Comprehensive Breast Health Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, 197 Ruijin Er Road, Shanghai, 200025, China.
| | - Jinhua Yu
- School of Information Science and Technology, Fudan University, No. 220, Handan Road, Yangpu District, Shanghai, 200433, China.
| |
Collapse
|
9
|
Ganesan P, Feng R, Deb B, Tjong FVY, Rogers AJ, Ruipérez-Campillo S, Somani S, Clopton P, Baykaner T, Rodrigo M, Zou J, Haddad F, Zaharia M, Narayan SM. Novel Domain Knowledge-Encoding Algorithm Enables Label-Efficient Deep Learning for Cardiac CT Segmentation to Guide Atrial Fibrillation Treatment in a Pilot Dataset. Diagnostics (Basel) 2024; 14:1538. [PMID: 39061675 PMCID: PMC11276420 DOI: 10.3390/diagnostics14141538] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2024] [Revised: 07/07/2024] [Accepted: 07/10/2024] [Indexed: 07/28/2024] Open
Abstract
Background: Segmenting computed tomography (CT) is crucial in various clinical applications, such as tailoring personalized cardiac ablation for managing cardiac arrhythmias. Automating segmentation through machine learning (ML) is hindered by the necessity for large, labeled training data, which can be challenging to obtain. This article proposes a novel approach for automated, robust labeling using domain knowledge to achieve high-performance segmentation by ML from a small training set. The approach, the domain knowledge-encoding (DOKEN) algorithm, reduces the reliance on large training datasets by encoding cardiac geometry while automatically labeling the training set. The method was validated in a hold-out dataset of CT results from an atrial fibrillation (AF) ablation study. Methods: The DOKEN algorithm parses left atrial (LA) structures, extracts "anatomical knowledge" by leveraging digital LA models (available publicly), and then applies this knowledge to achieve high ML segmentation performance with a small number of training samples. The DOKEN-labeled training set was used to train a nnU-Net deep neural network (DNN) model for segmenting cardiac CT in N = 20 patients. Subsequently, the method was tested in a hold-out set with N = 100 patients (five times larger than training set) who underwent AF ablation. Results: The DOKEN algorithm integrated with the nn-Unet model achieved high segmentation performance with few training samples, with a training to test ratio of 1:5. The Dice score of the DOKEN-enhanced model was 96.7% (IQR: 95.3% to 97.7%), with a median error in surface distance of boundaries of 1.51 mm (IQR: 0.72 to 3.12) and a mean centroid-boundary distance of 1.16 mm (95% CI: -4.57 to 6.89), similar to expert results (r = 0.99; p < 0.001). In digital hearts, the novel DOKEN approach segmented the LA structures with a mean difference for the centroid-boundary distances of -0.27 mm (95% CI: -3.87 to 3.33; r = 0.99; p < 0.0001). Conclusions: The proposed novel domain knowledge-encoding algorithm was able to perform the segmentation of six substructures of the LA, reducing the need for large training data sets. The combination of domain knowledge encoding and a machine learning approach could reduce the dependence of ML on large training datasets and could potentially be applied to AF ablation procedures and extended in the future to other imaging, 3D printing, and data science applications.
Collapse
Affiliation(s)
- Prasanth Ganesan
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Ruibin Feng
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Brototo Deb
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Fleur V. Y. Tjong
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
- Heart Center, Department of Clinical and Experimental Cardiology, Amsterdam UMC, University of Amsterdam, 1105 AZ Amsterdam, The Netherlands
| | - Albert J. Rogers
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Samuel Ruipérez-Campillo
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
- Department of Computer Science, ETH Zurich, 8092 Zurich, Switzerland
| | - Sulaiman Somani
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Paul Clopton
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Tina Baykaner
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Miguel Rodrigo
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
- CoMMLab, Universitat de València, 46100 Valencia, Spain
| | - James Zou
- Department of Biomedical Data Science, Stanford University, Stanford, CA 94305, USA
| | - Francois Haddad
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| | - Matei Zaharia
- Department of Computer Science, University of California Berkeley, Berkeley, CA 94720, USA
| | - Sanjiv M. Narayan
- Department of Medicine and Stanford Cardiovascular Institute (CVI), Stanford University, Stanford, CA 94305, USA; (P.G.); (R.F.)
| |
Collapse
|
10
|
Chintapalli SS, Wang R, Yang Z, Tassopoulou V, Yu F, Bashyam V, Erus G, Chaudhari P, Shou H, Davatzikos C. NeuroSynth: MRI-Derived Neuroanatomical Generative Models and Associated Dataset of 18,000 Samples. ARXIV 2024:arXiv:2407.12897v1. [PMID: 39070036 PMCID: PMC11275685] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Subscribe] [Scholar Register] [Indexed: 07/30/2024]
Abstract
Availability of large and diverse medical datasets is often challenged by privacy and data sharing restrictions. For successful application of machine learning techniques for disease diagnosis, prognosis, and precision medicine, large amounts of data are necessary for model building and optimization. To help overcome such limitations in the context of brain MRI, we present NeuroSynth: a collection of generative models of normative regional volumetric features derived from structural brain imaging. NeuroSynth models are trained on real brain imaging regional volumetric measures from the iSTAGING consortium, which encompasses over 40,000 MRI scans across 13 studies, incorporating covariates such as age, sex, and race. Leveraging NeuroSynth, we produce and offer 18,000 synthetic samples spanning the adult lifespan (ages 22-90 years), alongside the model's capability to generate unlimited data. Experimental results indicate that samples generated from NeuroSynth agree with the distributions obtained from real data. Most importantly, the generated normative data significantly enhance the accuracy of downstream machine learning models on tasks such as disease classification. Data and models are available at: https://huggingface.co/spaces/rongguangw/neuro-synth.
Collapse
Affiliation(s)
- Sai Spandana Chintapalli
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Rongguang Wang
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Zhijian Yang
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Vasiliki Tassopoulou
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Fanyang Yu
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Vishnu Bashyam
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Guray Erus
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Pratik Chaudhari
- Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Haochang Shou
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
- Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Christos Davatzikos
- Center for AI and Data Science for Integrated Diagnostics (AI2D), Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
| |
Collapse
|
11
|
Ntampakis N, Argyriou V, Diamantaras K, Goulianas K, Sarigiannidis P, Siniosoglou I. Introducing SPINE: A Holistic Approach to Synthetic Pulmonary Imaging Evaluation Through End-to-End Data and Model Management. IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY 2024; 5:576-588. [PMID: 39157061 PMCID: PMC11329215 DOI: 10.1109/ojemb.2024.3426910] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Revised: 05/25/2024] [Accepted: 07/08/2024] [Indexed: 08/20/2024] Open
Abstract
In the evolving field of medical imaging and machine learning (ML), this paper introduces a novel framework for evaluating synthetic pulmonary imaging aiming to assess synthetic data quality and applicability. Our study concentrates on synthetic X-ray chest images, crucial for diagnosing respiratory diseases. We employ SPINE (Synthetic Pulmonary Imaging Evaluation) framework, a threefold synthetic images evaluation method including expert domain assessment, statistical data analysis and adversarial evaluation. In order to replicate and validate our methodology, we followed an End-to-End data and model management process which begins with a dataset of Normal and Pneumonia chest X-rays, generating synthetic images using Generative Adversarial Networks (GANs) and training a baseline classifier, essential in the adversarial evaluation axis, testing synthetic images against real data assessing their predictive value. The critical outcome of our approach is the post-market analysis of synthetic images. This innovative method evaluates synthetic images using clinical, statistical, and scientific criteria independently from traditional generation performance metrics. This independent evaluation provides deep insights into the clinical and research effectiveness of the synthetic data. By ensuring these images mirror real data's statistical properties and maintain clinical accuracy, our framework establishes a new standard for the ethical and reliable use of synthetic data in medical imaging and research.
Collapse
Affiliation(s)
- Nikolaos Ntampakis
- Department of Information & Electronic EngineeringInternational Hellenic University57001SindosGreece
- MetaMind Innovations50100KozaniGreece
| | | | - Konstantinos Diamantaras
- Department of Information & Electronic EngineeringInternational Hellenic University57001SindosGreece
| | - Konstantinos Goulianas
- Department of Information & Electronic EngineeringInternational Hellenic University57001SindosGreece
| | | | - Ilias Siniosoglou
- MetaMind Innovations50100KozaniGreece
- University of Western Macedonia50100KozaniGreece
| |
Collapse
|
12
|
Oikonomou EK, Khera R. Artificial intelligence-enhanced patient evaluation: bridging art and science. Eur Heart J 2024:ehae415. [PMID: 38976371 DOI: 10.1093/eurheartj/ehae415] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/11/2024] [Revised: 04/23/2024] [Accepted: 06/18/2024] [Indexed: 07/10/2024] Open
Abstract
The advent of digital health and artificial intelligence (AI) has promised to revolutionize clinical care, but real-world patient evaluation has yet to witness transformative changes. As history taking and physical examination continue to rely on long-established practices, a growing pipeline of AI-enhanced digital tools may soon augment the traditional clinical encounter into a data-driven process. This article presents an evidence-backed vision of how promising AI applications may enhance traditional practices, streamlining tedious tasks while elevating diverse data sources, including AI-enabled stethoscopes, cameras, and wearable sensors, to platforms for personalized medicine and efficient care delivery. Through the lens of traditional patient evaluation, we illustrate how digital technologies may soon be interwoven into routine clinical workflows, introducing a novel paradigm of longitudinal monitoring. Finally, we provide a skeptic's view on the practical, ethical, and regulatory challenges that limit the uptake of such technologies.
Collapse
Affiliation(s)
- Evangelos K Oikonomou
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, 333 Cedar Street, PO Box 208017, New Haven, 06520-8017 CT, USA
| | - Rohan Khera
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, 333 Cedar Street, PO Box 208017, New Haven, 06520-8017 CT, USA
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, 195 Church St, 6th Floor, New Haven, CT 06510, USA
- Section of Biomedical Informatics and Data Science, Yale School of Medicine, 100 College Street, New Haven, 06511 CT, USA
- Section of Health Informatics, Department of Biostatistics, Yale School of Public Health, 60 College Street, New Haven, 06510 CT, USA
| |
Collapse
|
13
|
Osorio P, Jimenez-Perez G, Montalt-Tordera J, Hooge J, Duran-Ballester G, Singh S, Radbruch M, Bach U, Schroeder S, Siudak K, Vienenkoetter J, Lawrenz B, Mohammadi S. Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology. Diagnostics (Basel) 2024; 14:1442. [PMID: 39001331 PMCID: PMC11241396 DOI: 10.3390/diagnostics14131442] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2024] [Revised: 06/19/2024] [Accepted: 06/26/2024] [Indexed: 07/16/2024] Open
Abstract
Artificial Intelligence (AI)-based image analysis has immense potential to support diagnostic histopathology, including cancer diagnostics. However, developing supervised AI methods requires large-scale annotated datasets. A potentially powerful solution is to augment training data with synthetic data. Latent diffusion models, which can generate high-quality, diverse synthetic images, are promising. However, the most common implementations rely on detailed textual descriptions, which are not generally available in this domain. This work proposes a method that constructs structured textual prompts from automatically extracted image features. We experiment with the PCam dataset, composed of tissue patches only loosely annotated as healthy or cancerous. We show that including image-derived features in the prompt, as opposed to only healthy and cancerous labels, improves the Fréchet Inception Distance (FID) by 88.6. We also show that pathologists find it challenging to detect synthetic images, with a median sensitivity/specificity of 0.55/0.55. Finally, we show that synthetic data effectively train AI models.
Collapse
Affiliation(s)
- Pedro Osorio
- Decision Science & Advanced Analytics, Bayer AG, 13353 Berlin, Germany
| | | | | | - Jens Hooge
- Decision Science & Advanced Analytics, Bayer AG, 13353 Berlin, Germany
| | | | - Shivam Singh
- Decision Science & Advanced Analytics, Bayer AG, 13353 Berlin, Germany
| | - Moritz Radbruch
- Pathology and Clinical Pathology, Bayer AG, 13353 Berlin, Germany
| | - Ute Bach
- Pathology and Clinical Pathology, Bayer AG, 13353 Berlin, Germany
| | | | - Krystyna Siudak
- Pathology and Clinical Pathology, Bayer AG, 13353 Berlin, Germany
| | | | - Bettina Lawrenz
- Pathology and Clinical Pathology, Bayer AG, 13353 Berlin, Germany
| | - Sadegh Mohammadi
- Decision Science & Advanced Analytics, Bayer AG, 13353 Berlin, Germany
| |
Collapse
|
14
|
Wu S, Kurugol S, Kleinman PK, Ecklund K, Walters M, Connolly SA, Johnston P, Tsai A. Deep generative model of the distal tibial classic metaphyseal lesion in infants: assessment of synthetic images. RADIOLOGY ADVANCES 2024; 1:umae018. [PMID: 39171131 PMCID: PMC11335364 DOI: 10.1093/radadv/umae018] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/14/2024] [Revised: 06/04/2024] [Accepted: 06/25/2024] [Indexed: 08/23/2024]
Abstract
Background The classic metaphyseal lesion (CML) is a distinctive fracture highly specific to infant abuse. To increase the size and diversity of the training CML database for automated deep-learning detection of this fracture, we developed a mask conditional diffusion model (MaC-DM) to generate synthetic images with and without CMLs. Purpose To objectively and subjectively assess the synthetic radiographic images with and without CMLs generated by MaC-DM. Materials and Methods For retrospective testing, we randomly chose 100 real images (50 normals and 50 with CMLs; 39 infants, male = 22, female = 17; mean age = 4.1 months; SD = 3.1 months) from an existing distal tibia dataset (177 normal, 73 with CMLs), and generated 100 synthetic distal tibia images via MaC-DM (50 normals and 50 with CMLs). These test images were shown to 3 blinded radiologists. In the first session, radiologists determined if the images were normal or had CMLs. In the second session, they determined if the images were real or synthetic. We analyzed the radiologists' interpretations and employed t-distributed stochastic neighbor embedding technique to analyze the data distribution of the test images. Results When presented with the 200 images (100 synthetic, 100 with CMLs), radiologists reliably and accurately diagnosed CMLs (kappa = 0.90, 95% CI = [0.88-0.92]; accuracy = 92%, 95% CI = [89-97]). However, they were inaccurate in differentiating between real and synthetic images (kappa = 0.05, 95% CI = [0.03-0.07]; accuracy = 53%, 95% CI = [49-59]). The t-distributed stochastic neighbor embedding analysis showed substantial differences in the data distribution between normal images and those with CMLs (area under the curve = 0.996, 95% CI = [0.992-1.000], P < .01), but minor differences between real and synthetic images (area under the curve = 0.566, 95% CI = [0.486-0.647], P = .11). Conclusion Radiologists accurately diagnosed images with distal tibial CMLs but were unable to distinguish real from synthetically generated ones, indicating that our generative model could synthesize realistic images. Thus, MaC-DM holds promise as an effective strategy for data augmentation in training machine-learning models for diagnosis of distal tibial CMLs.
Collapse
Affiliation(s)
- Shaoju Wu
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Sila Kurugol
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Paul K Kleinman
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Kirsten Ecklund
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Michele Walters
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Susan A Connolly
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Patrick Johnston
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Andy Tsai
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| |
Collapse
|
15
|
Stanekova V, Inglis JM, Lam L, Lam A, Smith W, Shakib S, Bacchi S. Improving the performance of machine learning penicillin adverse drug reaction classification with synthetic data and transfer learning. Intern Med J 2024; 54:1183-1189. [PMID: 38482918 DOI: 10.1111/imj.16360] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2023] [Accepted: 02/19/2024] [Indexed: 07/18/2024]
Abstract
BACKGROUND Machine learning may assist with the identification of potentially inappropriate penicillin allergy labels. Strategies to improve the performance of existing models for this task include the use of additional training data, synthetic data and transfer learning. AIMS The aims of this study were to investigate the use of additional training data and novel machine learning strategies, namely synthetic data and transfer learning, to improve the performance of penicillin adverse drug reaction (ADR) machine learning classification. METHODS Machine learning natural language processing was applied to free-text penicillin ADR data extracted from a public health system electronic health record (EHR). The models were developed by training on various labelled data sets. ADR entries were split into training and testing data sets and used to develop and test a variety of machine learning models. The effect of training on additional data and synthetic data versus the use of transfer learning was analysed. RESULTS Following the application of these techniques, the area under the receiver operator curve of best-performing models for the classification of penicillin allergy (vs intolerance) and high-risk allergy (vs low-risk allergy) improved to 0.984 (using the artificial neural network model) and 0.995 (with the transfer learning approach) respectively. CONCLUSIONS Machine learning models demonstrate high levels of accuracy in the classification and risk stratification of penicillin ADR labels using the reaction documented in the EHR. The model can be further optimised by incorporating additional training data and using transfer learning. Practical applications include automating case detection for penicillin allergy delabelling programmes.
Collapse
Affiliation(s)
- Viera Stanekova
- Royal Adelaide Hospital, Adelaide, South Australia, Australia
- University of Adelaide, Adelaide, South Australia, Australia
| | - Joshua M Inglis
- Royal Adelaide Hospital, Adelaide, South Australia, Australia
- University of Adelaide, Adelaide, South Australia, Australia
| | - Lydia Lam
- Royal Adelaide Hospital, Adelaide, South Australia, Australia
- University of Adelaide, Adelaide, South Australia, Australia
| | - Antoinette Lam
- Royal Adelaide Hospital, Adelaide, South Australia, Australia
- University of Adelaide, Adelaide, South Australia, Australia
| | - William Smith
- Royal Adelaide Hospital, Adelaide, South Australia, Australia
- University of Adelaide, Adelaide, South Australia, Australia
| | - Sepehr Shakib
- Royal Adelaide Hospital, Adelaide, South Australia, Australia
- University of Adelaide, Adelaide, South Australia, Australia
| | - Stephen Bacchi
- Royal Adelaide Hospital, Adelaide, South Australia, Australia
- University of Adelaide, Adelaide, South Australia, Australia
| |
Collapse
|
16
|
Sonmez SC, Sevgi M, Antaki F, Huemer J, Keane PA. Generative artificial intelligence in ophthalmology: current innovations, future applications and challenges. Br J Ophthalmol 2024:bjo-2024-325458. [PMID: 38925907 DOI: 10.1136/bjo-2024-325458] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Accepted: 06/03/2024] [Indexed: 06/28/2024]
Abstract
The rapid advancements in generative artificial intelligence are set to significantly influence the medical sector, particularly ophthalmology. Generative adversarial networks and diffusion models enable the creation of synthetic images, aiding the development of deep learning models tailored for specific imaging tasks. Additionally, the advent of multimodal foundational models, capable of generating images, text and videos, presents a broad spectrum of applications within ophthalmology. These range from enhancing diagnostic accuracy to improving patient education and training healthcare professionals. Despite the promising potential, this area of technology is still in its infancy, and there are several challenges to be addressed, including data bias, safety concerns and the practical implementation of these technologies in clinical settings.
Collapse
Affiliation(s)
| | - Mertcan Sevgi
- Institute of Ophthalmology, University College London, London, UK
- Moorfields Eye Hospital, NIHR Moorfields Biomedical Research Centre, London, UK
| | - Fares Antaki
- Institute of Ophthalmology, University College London, London, UK
- Moorfields Eye Hospital, NIHR Moorfields Biomedical Research Centre, London, UK
- The CHUM School of Artificial Intelligence in Healthcare, Montreal, Quebec, Canada
| | - Josef Huemer
- Moorfields Eye Hospital, NIHR Moorfields Biomedical Research Centre, London, UK
- Department of Ophthalmology and Optometry, Kepler University Hospital, Linz, Austria
| | - Pearse A Keane
- Institute of Ophthalmology, University College London, London, UK
- Moorfields Eye Hospital, NIHR Moorfields Biomedical Research Centre, London, UK
| |
Collapse
|
17
|
Wen D, Soltan A, Trucco E, Matin RN. From data to diagnosis: skin cancer image datasets for artificial intelligence. Clin Exp Dermatol 2024; 49:675-685. [PMID: 38549552 DOI: 10.1093/ced/llae112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Revised: 02/11/2024] [Accepted: 03/25/2024] [Indexed: 06/26/2024]
Abstract
Artificial intelligence (AI) solutions for skin cancer diagnosis continue to gain momentum, edging closer towards broad clinical use. These AI models, particularly deep-learning architectures, require large digital image datasets for development. This review provides an overview of the datasets used to develop AI algorithms and highlights the importance of dataset transparency for the evaluation of algorithm generalizability across varying populations and settings. Current challenges for curation of clinically valuable datasets are detailed, which include dataset shifts arising from demographic variations and differences in data collection methodologies, along with inconsistencies in labelling. These shifts can lead to differential algorithm performance, compromise of clinical utility, and the propagation of discriminatory biases when developed algorithms are implemented in mismatched populations. Limited representation of rare skin cancers and minoritized groups in existing datasets are highlighted, which can further skew algorithm performance. Strategies to address these challenges are presented, which include improving transparency, representation and interoperability. Federated learning and generative methods, which may improve dataset size and diversity without compromising privacy, are also examined. Lastly, we discuss model-level techniques that may address biases entrained through the use of datasets derived from routine clinical care. As the role of AI in skin cancer diagnosis becomes more prominent, ensuring the robustness of underlying datasets is increasingly important.
Collapse
Affiliation(s)
- David Wen
- Department of Dermatology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- Oxford University Clinical Academic Graduate School, University of Oxford, Oxford, UK
| | - Andrew Soltan
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
- Oxford Cancer and Haematology Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- Department of Oncology, University of Oxford, Oxford, UK
| | - Emanuele Trucco
- VAMPIRE Project, Computing, School of Science and Engineering, University of Dundee, Dundee, UK
| | - Rubeta N Matin
- Department of Dermatology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- Artificial Intelligence Working Party Group, British Association of Dermatologists, London, UK
| |
Collapse
|
18
|
Gim N, Wu Y, Blazes M, Lee CS, Wang RK, Lee AY. A Clinician's Guide to Sharing Data for AI in Ophthalmology. Invest Ophthalmol Vis Sci 2024; 65:21. [PMID: 38864811 PMCID: PMC11174091 DOI: 10.1167/iovs.65.6.21] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Accepted: 05/17/2024] [Indexed: 06/13/2024] Open
Abstract
Data is the cornerstone of using AI models, because their performance directly depends on the diversity, quantity, and quality of the data used for training. Using AI presents unique potential, particularly in medical applications that involve rich data such as ophthalmology, encompassing a variety of imaging methods, medical records, and eye-tracking data. However, sharing medical data comes with challenges because of regulatory issues and privacy concerns. This review explores traditional and nontraditional data sharing methods in medicine, focusing on previous works in ophthalmology. Traditional methods involve direct data transfer, whereas newer approaches prioritize security and privacy by sharing derived datasets, creating secure research environments, or using model-to-data strategies. We examine each method's mechanisms, variations, recent applications in ophthalmology, and their respective advantages and disadvantages. By empowering medical researchers with insights into data sharing methods and considerations, this review aims to assist informed decision-making while upholding ethical standards and patient privacy in medical AI development.
Collapse
Affiliation(s)
- Nayoon Gim
- Department of Ophthalmology, University of Washington, Seattle, WA, United States
- The Roger and Angie Karalis Retina Center, Seattle, Washington, United States
- Department of Bioengineering, University of Washington, Seattle, WA, United States
| | - Yue Wu
- Department of Ophthalmology, University of Washington, Seattle, WA, United States
- The Roger and Angie Karalis Retina Center, Seattle, Washington, United States
| | - Marian Blazes
- Department of Ophthalmology, University of Washington, Seattle, WA, United States
- The Roger and Angie Karalis Retina Center, Seattle, Washington, United States
| | - Cecilia S. Lee
- Department of Ophthalmology, University of Washington, Seattle, WA, United States
- The Roger and Angie Karalis Retina Center, Seattle, Washington, United States
| | - Ruikang K. Wang
- Department of Ophthalmology, University of Washington, Seattle, WA, United States
- Department of Bioengineering, University of Washington, Seattle, WA, United States
| | - Aaron Y. Lee
- Department of Ophthalmology, University of Washington, Seattle, WA, United States
- The Roger and Angie Karalis Retina Center, Seattle, Washington, United States
| |
Collapse
|
19
|
Kikuchi T, Hanaoka S, Nakao T, Takenaga T, Nomura Y, Mori H, Yoshikawa T. Synthesis of Hybrid Data Consisting of Chest Radiographs and Tabular Clinical Records Using Dual Generative Models for COVID-19 Positive Cases. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:1217-1227. [PMID: 38351224 PMCID: PMC11169290 DOI: 10.1007/s10278-024-01015-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Revised: 12/21/2023] [Accepted: 12/22/2023] [Indexed: 06/13/2024]
Abstract
To generate synthetic medical data incorporating image-tabular hybrid data by merging an image encoding/decoding model with a table-compatible generative model and assess their utility. We used 1342 cases from the Stony Brook University Covid-19-positive cases, comprising chest X-ray radiographs (CXRs) and tabular clinical data as a private dataset (pDS). We generated a synthetic dataset (sDS) through the following steps: (I) dimensionally reducing CXRs in the pDS using a pretrained encoder of the auto-encoding generative adversarial networks (αGAN) and integrating them with the correspondent tabular clinical data; (II) training the conditional tabular GAN (CTGAN) on this combined data to generate synthetic records, encompassing encoded image features and clinical data; and (III) reconstructing synthetic images from these encoded image features in the sDS using a pretrained decoder of the αGAN. The utility of sDS was assessed by the performance of the prediction models for patient outcomes (deceased or discharged). For the pDS test set, the area under the receiver operating characteristic (AUC) curve was calculated to compare the performance of prediction models trained separately with pDS, sDS, or a combination of both. We created an sDS comprising CXRs with a resolution of 256 × 256 pixels and tabular data containing 13 variables. The AUC for the outcome was 0.83 when the model was trained with the pDS, 0.74 with the sDS, and 0.87 when combining pDS and sDS for training. Our method is effective for generating synthetic records consisting of both images and tabular clinical data.
Collapse
Affiliation(s)
- Tomohiro Kikuchi
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan.
- Department of Radiology, School of Medicine, Jichi Medical University, 3311-1 Yakushiji, Shimotsuke, Tochigi, 329-0498, Japan.
| | - Shouhei Hanaoka
- Department of Radiology, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan
| | - Takahiro Nakao
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan
| | - Tomomi Takenaga
- Department of Radiology, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan
| | - Yukihiro Nomura
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan
- Center for Frontier Medical Engineering, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba, 263-8522, Japan
| | - Harushi Mori
- Department of Radiology, School of Medicine, Jichi Medical University, 3311-1 Yakushiji, Shimotsuke, Tochigi, 329-0498, Japan
| | - Takeharu Yoshikawa
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan
| |
Collapse
|
20
|
Niehues JM, Müller-Franzes G, Schirris Y, Wagner SJ, Jendrusch M, Kloor M, Pearson AT, Muti HS, Hewitt KJ, Veldhuizen GP, Zigutyte L, Truhn D, Kather JN. Using histopathology latent diffusion models as privacy-preserving dataset augmenters improves downstream classification performance. Comput Biol Med 2024; 175:108410. [PMID: 38678938 DOI: 10.1016/j.compbiomed.2024.108410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 03/23/2024] [Accepted: 04/02/2024] [Indexed: 05/01/2024]
Abstract
Latent diffusion models (LDMs) have emerged as a state-of-the-art image generation method, outperforming previous Generative Adversarial Networks (GANs) in terms of training stability and image quality. In computational pathology, generative models are valuable for data sharing and data augmentation. However, the impact of LDM-generated images on histopathology tasks compared to traditional GANs has not been systematically studied. We trained three LDMs and a styleGAN2 model on histology tiles from nine colorectal cancer (CRC) tissue classes. The LDMs include 1) a fine-tuned version of stable diffusion v1.4, 2) a Kullback-Leibler (KL)-autoencoder (KLF8-DM), and 3) a vector quantized (VQ)-autoencoder deploying LDM (VQF8-DM). We assessed image quality through expert ratings, dimensional reduction methods, distribution similarity measures, and their impact on training a multiclass tissue classifier. Additionally, we investigated image memorization in the KLF8-DM and styleGAN2 models. All models provided a high image quality, with the KLF8-DM achieving the best Frechet Inception Distance (FID) and expert rating scores for complex tissue classes. For simpler classes, the VQF8-DM and styleGAN2 models performed better. Image memorization was negligible for both styleGAN2 and KLF8-DM models. Classifiers trained on a mix of KLF8-DM generated and real images achieved a 4% improvement in overall classification accuracy, highlighting the usefulness of these images for dataset augmentation. Our systematic study of generative methods showed that KLF8-DM produces the highest quality images with negligible image memorization. The higher classifier performance in the generatively augmented dataset suggests that this augmentation technique can be employed to enhance histopathology classifiers for various tasks.
Collapse
Affiliation(s)
- Jan M Niehues
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Yoni Schirris
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Netherlands Cancer Institute, 1066 CX, Amsterdam, the Netherlands; University of Amsterdam, 1012 WP, Amsterdam, the Netherlands
| | - Sophia Janine Wagner
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
| | - Michael Jendrusch
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
| | - Matthias Kloor
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
| | | | - Hannah Sophie Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Katherine J Hewitt
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Gregory P Veldhuizen
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Laura Zigutyte
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom; Department of Medicine I, University Hospital Dresden, Dresden, Germany; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
21
|
Asadi F, Angsuwatanakul T, O’Reilly JA. Evaluating synthetic neuroimaging data augmentation for automatic brain tumour segmentation with a deep fully-convolutional network. IBRO Neurosci Rep 2024; 16:57-66. [PMID: 39007088 PMCID: PMC11240293 DOI: 10.1016/j.ibneur.2023.12.002] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Accepted: 12/11/2023] [Indexed: 07/16/2024] Open
Abstract
Gliomas observed in medical images require expert neuro-radiologist evaluation for treatment planning and monitoring, motivating development of intelligent systems capable of automating aspects of tumour evaluation. Deep learning models for automatic image segmentation rely on the amount and quality of training data. In this study we developed a neuroimaging synthesis technique to augment data for training fully-convolutional networks (U-nets) to perform automatic glioma segmentation. We used StyleGAN2-ada to simultaneously generate fluid-attenuated inversion recovery (FLAIR) magnetic resonance images and corresponding glioma segmentation masks. Synthetic data were successively added to real training data (n = 2751) in fourteen rounds of 1000 and used to train U-nets that were evaluated on held-out validation (n = 590) and test sets (n = 588). U-nets were trained with and without geometric augmentation (translation, zoom and shear), and Dice coefficients were computed to evaluate segmentation performance. We also monitored the number of training iterations before stopping, total training time, and time per iteration to evaluate computational costs associated with training each U-net. Synthetic data augmentation yielded marginal improvements in Dice coefficients (validation set +0.0409, test set +0.0355), whereas geometric augmentation improved generalization (standard deviation between training, validation and test set performances of 0.01 with, and 0.04 without geometric augmentation). Based on the modest performance gains for automatic glioma segmentation we find it hard to justify the computational expense of developing a synthetic image generation pipeline. Future work may seek to optimize the efficiency of synthetic data generation for augmentation of neuroimaging data.
Collapse
Affiliation(s)
- Fawad Asadi
- College of Biomedical Engineering, Rangsit University, Pathum Thani 12000, Thailand
| | | | - Jamie A. O’Reilly
- School of Engineering, King Mongkut’s Institute of Technology Ladkrabang, Bangkok 10520, Thailand
| |
Collapse
|
22
|
Megerlin F, Pinilla E. European Health Data Space Act proposal silent on "quality and utility label" of synthetic patient data. Consequences? ANNALES PHARMACEUTIQUES FRANÇAISES 2024; 82:641-644. [PMID: 38485088 DOI: 10.1016/j.pharma.2024.03.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2024] [Accepted: 03/07/2024] [Indexed: 06/15/2024]
Affiliation(s)
- Francis Megerlin
- Université de Strasbourg, Strasbourg, France; Centre d'études internationales et européennes, université de Strasbourg, Strasbourg, France.
| | - Erwan Pinilla
- Institut des hautes études du ministère de l'Intérieur (2015), Paris, France; Centre d'études internationales et européennes, université de Strasbourg, Strasbourg, France
| |
Collapse
|
23
|
Danilatou V, Dimopoulos D, Kostoulas T, Douketis J. Machine Learning-Based Predictive Models for Patients with Venous Thromboembolism: A Systematic Review. Thromb Haemost 2024. [PMID: 38574756 DOI: 10.1055/a-2299-4758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/06/2024]
Abstract
BACKGROUND Venous thromboembolism (VTE) is a chronic disorder with a significant health and economic burden. Several VTE-specific clinical prediction models (CPMs) have been used to assist physicians in decision-making but have several limitations. This systematic review explores if machine learning (ML) can enhance CPMs by analyzing extensive patient data derived from electronic health records. We aimed to explore ML-CPMs' applications in VTE for risk stratification, outcome prediction, diagnosis, and treatment. METHODS Three databases were searched: PubMed, Google Scholar, and IEEE electronic library. Inclusion criteria focused on studies using structured data, excluding non-English publications, studies on non-humans, and certain data types such as natural language processing and image processing. Studies involving pregnant women, cancer patients, and children were also excluded. After excluding irrelevant studies, a total of 77 studies were included. RESULTS Most studies report that ML-CPMs outperformed traditional CPMs in terms of receiver operating area under the curve in the four clinical domains that were explored. However, the majority of the studies were retrospective, monocentric, and lacked detailed model architecture description and external validation, which are essential for quality audit. This review identified research gaps and highlighted challenges related to standardized reporting, reproducibility, and model comparison. CONCLUSION ML-CPMs show promise in improving risk assessment and individualized treatment recommendations in VTE. Apparently, there is an urgent need for standardized reporting and methodology for ML models, external validation, prospective and real-world data studies, as well as interventional studies to evaluate the impact of artificial intelligence in VTE.
Collapse
Affiliation(s)
- Vasiliki Danilatou
- School of Medicine, European University of Cyprus, Nicosia, Cyprus
- Healthcare Division, Sphynx Technology Solutions, Nicosia, Cyprus
| | - Dimitrios Dimopoulos
- School of Engineering, Department of Information and Communication Systems Engineering, University of the Aegean, North Aegean, Greece
| | - Theodoros Kostoulas
- School of Engineering, Department of Information and Communication Systems Engineering, University of the Aegean, North Aegean, Greece
| | - James Douketis
- Department of Medicine, McMaster University, Hamilton, Canada
- Department of Medicine, St. Joseph's Healthcare Hamilton, Ontario, Canada
| |
Collapse
|
24
|
Alajaji SA, Khoury ZH, Jessri M, Sciubba JJ, Sultan AS. An Update on the Use of Artificial Intelligence in Digital Pathology for Oral Epithelial Dysplasia Research. Head Neck Pathol 2024; 18:38. [PMID: 38727841 PMCID: PMC11087425 DOI: 10.1007/s12105-024-01643-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Accepted: 03/30/2024] [Indexed: 05/13/2024]
Abstract
INTRODUCTION Oral epithelial dysplasia (OED) is a precancerous histopathological finding which is considered the most important prognostic indicator for determining the risk of malignant transformation into oral squamous cell carcinoma (OSCC). The gold standard for diagnosis and grading of OED is through histopathological examination, which is subject to inter- and intra-observer variability, impacting accurate diagnosis and prognosis. The aim of this review article is to examine the current advances in digital pathology for artificial intelligence (AI) applications used for OED diagnosis. MATERIALS AND METHODS We included studies that used AI for diagnosis, grading, or prognosis of OED on histopathology images or intraoral clinical images. Studies utilizing imaging modalities other than routine light microscopy (e.g., scanning electron microscopy), or immunohistochemistry-stained histology slides, or immunofluorescence were excluded from the study. Studies not focusing on oral dysplasia grading and diagnosis, e.g., to discriminate OSCC from normal epithelial tissue were also excluded. RESULTS A total of 24 studies were included in this review. Nineteen studies utilized deep learning (DL) convolutional neural networks for histopathological OED analysis, and 4 used machine learning (ML) models. Studies were summarized by AI method, main study outcomes, predictive value for malignant transformation, strengths, and limitations. CONCLUSION ML/DL studies for OED grading and prediction of malignant transformation are emerging as promising adjunctive tools in the field of digital pathology. These adjunctive objective tools can ultimately aid the pathologist in more accurate diagnosis and prognosis prediction. However, further supportive studies that focus on generalization, explainable decisions, and prognosis prediction are needed.
Collapse
Affiliation(s)
- Shahd A Alajaji
- Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, 650 W. Baltimore Street, 7 Floor, Baltimore, MD, 21201, USA
- Department of Oral Medicine and Diagnostic Sciences, College of Dentistry, King Saud University, Riyadh, Saudi Arabia
- Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA
| | - Zaid H Khoury
- Department of Oral Diagnostic Sciences and Research, Meharry Medical College School of Dentistry, Nashville, TN, USA
| | - Maryam Jessri
- Oral Medicine and Pathology Department, School of Dentistry, University of Queensland, Herston, QLD, Australia
- Oral Medicine Department, Metro North Hospital and Health Services, Queensland Health, Brisbane, QLD, Australia
| | - James J Sciubba
- Department of Otolaryngology, Head & Neck Surgery, The Johns Hopkins University, Baltimore, MD, USA
| | - Ahmed S Sultan
- Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, 650 W. Baltimore Street, 7 Floor, Baltimore, MD, 21201, USA.
- Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA.
- University of Maryland Marlene and Stewart Greenebaum Comprehensive Cancer Center, Baltimore, MD, USA.
| |
Collapse
|
25
|
Viswanathan M, Yin L, Kurmi Y, Zu Z. Machine learning-based amide proton transfer imaging using partially synthetic training data. Magn Reson Med 2024; 91:1908-1922. [PMID: 38098340 PMCID: PMC10955622 DOI: 10.1002/mrm.29970] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2023] [Revised: 10/30/2023] [Accepted: 11/26/2023] [Indexed: 12/20/2023]
Abstract
PURPOSE Machine learning (ML) has been increasingly used to quantify CEST effect. ML models are typically trained using either measured data or fully simulated data. However, training with measured data often lacks sufficient training data, whereas training with fully simulated data may introduce bias because of limited simulations pools. This study introduces a new platform that combines simulated and measured components to generate partially synthetic CEST data, and to evaluate its feasibility for training ML models to predict amide proton transfer (APT) effect. METHODS Partially synthetic CEST signals were created using an inverse summation of APT effects from simulations and the other components from measurements. Training data were generated by varying APT simulation parameters and applying scaling factors to adjust the measured components, achieving a balance between simulation flexibility and fidelity. First, tissue-mimicking CEST signals along with ground truth information were created using multiple-pool model simulations to validate this method. Second, an ML model was trained individually on partially synthetic data, in vivo data, and fully simulated data, to predict APT effect in rat brains bearing 9 L tumors. RESULTS Experiments on tissue-mimicking data suggest that the ML method using the partially synthetic data is accurate in predicting APT. In vivo experiments suggest that our method provides more accurate and robust prediction than the training using in vivo data and fully synthetic data. CONCLUSION Partially synthetic CEST data can address the challenges in conventional ML methods.
Collapse
Affiliation(s)
- Malvika Viswanathan
- Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, US
- Department of Biomedical Engineering, Vanderbilt University, Nashville, US
| | - Leqi Yin
- School of Engineering, Vanderbilt University, Nashville, US
| | - Yashwant Kurmi
- Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, US
- Department of Radiology and Radiological Sciences, Vanderbilt University Medical Center, Nashville, US
| | - Zhongliang Zu
- Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, US
- Department of Biomedical Engineering, Vanderbilt University, Nashville, US
- Department of Radiology and Radiological Sciences, Vanderbilt University Medical Center, Nashville, US
| |
Collapse
|
26
|
Wang L, Wan Z, Ni C, Song Q, Li Y, Clayton EW, Malin BA, Yin Z. A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.04.26.24306390. [PMID: 38712148 PMCID: PMC11071576 DOI: 10.1101/2024.04.26.24306390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2024]
Abstract
Background The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators. Objective This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare. Methods We utilized PubMed, ACM, and IEEE digital libraries as primary sources for this review. We followed the guidance of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRIMSA) to screen and select peer-reviewed research articles that (1) were related to both healthcare applications and conversational LLMs and (2) were published before September 1st, 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns. Results Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research. Conclusions Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare.
Collapse
Affiliation(s)
- Leyao Wang
- Department of Computer Science, Vanderbilt University, Nashville, TN, USA, 37212
| | - Zhiyu Wan
- Department of Biomedical Informatics, Vanderbilt University Medical Center, TN, USA, 37203
| | - Congning Ni
- Department of Computer Science, Vanderbilt University, Nashville, TN, USA, 37212
| | - Qingyuan Song
- Department of Computer Science, Vanderbilt University, Nashville, TN, USA, 37212
| | - Yang Li
- Department of Computer Science, Vanderbilt University, Nashville, TN, USA, 37212
| | - Ellen Wright Clayton
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee, USA, 37203
- Center for Biomedical Ethics and Society, Vanderbilt University Medical Center, Nashville, Tennessee, USA, 37203
| | - Bradley A. Malin
- Department of Computer Science, Vanderbilt University, Nashville, TN, USA, 37212
- Department of Biomedical Informatics, Vanderbilt University Medical Center, TN, USA, 37203
- Department of Biostatistics, Vanderbilt University Medical Center, TN, USA, 37203
| | - Zhijun Yin
- Department of Computer Science, Vanderbilt University, Nashville, TN, USA, 37212
- Department of Biomedical Informatics, Vanderbilt University Medical Center, TN, USA, 37203
| |
Collapse
|
27
|
Warner E, Lee J, Hsu W, Syeda-Mahmood T, Kahn Jr. CE, Gevaert O, Rao A. Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects. Int J Comput Vis 2024; 132:3753-3769. [PMID: 39211895 PMCID: PMC11349845 DOI: 10.1007/s11263-024-02032-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Accepted: 02/09/2024] [Indexed: 09/04/2024]
Abstract
Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of "big data" in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.
Collapse
Affiliation(s)
- Elisa Warner
- Department of Computational Medicine and Bioinformatics, University of Michigan Ann Arbor, 100 Washtenaw Ave., Ann Arbor, MI 48109 USA
| | - Joonsang Lee
- Department of Computational Medicine and Bioinformatics, University of Michigan Ann Arbor, 100 Washtenaw Ave., Ann Arbor, MI 48109 USA
| | - William Hsu
- Department of Medical and Imaging Informatics, University of California Los Angeles, 924 Westwood Blvd Ste 420, Los Angeles, CA 90024 USA
| | | | - Charles E. Kahn Jr.
- Department of Radiology, University of Pennsylvania, 3400 Spruce St., Philadelphia, PA 19104 USA
| | - Olivier Gevaert
- Center for Biomedical Informatics Research, Stanford, 1265 Welch Road, Stanford, CA 94305 USA
| | - Arvind Rao
- Department of Computational Medicine and Bioinformatics, University of Michigan Ann Arbor, 100 Washtenaw Ave., Ann Arbor, MI 48109 USA
| |
Collapse
|
28
|
Yan C, Zhang Z, Nyemba S, Li Z. Generating Synthetic Electronic Health Record Data Using Generative Adversarial Networks: Tutorial. JMIR AI 2024; 3:e52615. [PMID: 38875595 PMCID: PMC11074891 DOI: 10.2196/52615] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/10/2023] [Revised: 01/24/2024] [Accepted: 03/07/2024] [Indexed: 06/16/2024]
Abstract
Synthetic electronic health record (EHR) data generation has been increasingly recognized as an important solution to expand the accessibility and maximize the value of private health data on a large scale. Recent advances in machine learning have facilitated more accurate modeling for complex and high-dimensional data, thereby greatly enhancing the data quality of synthetic EHR data. Among various approaches, generative adversarial networks (GANs) have become the main technical path in the literature due to their ability to capture the statistical characteristics of real data. However, there is a scarcity of detailed guidance within the domain regarding the development procedures of synthetic EHR data. The objective of this tutorial is to present a transparent and reproducible process for generating structured synthetic EHR data using a publicly accessible EHR data set as an example. We cover the topics of GAN architecture, EHR data types and representation, data preprocessing, GAN training, synthetic data generation and postprocessing, and data quality evaluation. We conclude this tutorial by discussing multiple important issues and future opportunities in this domain. The source code of the entire process has been made publicly available.
Collapse
Affiliation(s)
- Chao Yan
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Ziqi Zhang
- Department of Computer Science, Vanderbilt University, Nashville, TN, United States
| | - Steve Nyemba
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Zhuohang Li
- Department of Computer Science, Vanderbilt University, Nashville, TN, United States
| |
Collapse
|
29
|
Perets O, Stagno E, Yehuda EB, McNichol M, Anthony Celi L, Rappoport N, Dorotic M. Inherent Bias in Electronic Health Records: A Scoping Review of Sources of Bias. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.04.09.24305594. [PMID: 38680842 PMCID: PMC11046491 DOI: 10.1101/2024.04.09.24305594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/01/2024]
Abstract
Objectives 1.1Biases inherent in electronic health records (EHRs), and therefore in medical artificial intelligence (AI) models may significantly exacerbate health inequities and challenge the adoption of ethical and responsible AI in healthcare. Biases arise from multiple sources, some of which are not as documented in the literature. Biases are encoded in how the data has been collected and labeled, by implicit and unconscious biases of clinicians, or by the tools used for data processing. These biases and their encoding in healthcare records undermine the reliability of such data and bias clinical judgments and medical outcomes. Moreover, when healthcare records are used to build data-driven solutions, the biases are further exacerbated, resulting in systems that perpetuate biases and induce healthcare disparities. This literature scoping review aims to categorize the main sources of biases inherent in EHRs. Methods 1.2We queried PubMed and Web of Science on January 19th, 2023, for peer-reviewed sources in English, published between 2016 and 2023, using the PRISMA approach to stepwise scoping of the literature. To select the papers that empirically analyze bias in EHR, from the initial yield of 430 papers, 27 duplicates were removed, and 403 studies were screened for eligibility. 196 articles were removed after the title and abstract screening, and 96 articles were excluded after the full-text review resulting in a final selection of 116 articles. Results 1.3Systematic categorizations of diverse sources of bias are scarce in the literature, while the effects of separate studies are often convoluted and methodologically contestable. Our categorization of published empirical evidence identified the six main sources of bias: a) bias arising from past clinical trials; b) data-related biases arising from missing, incomplete information or poor labeling of data; human-related bias induced by c) implicit clinician bias, d) referral and admission bias; e) diagnosis or risk disparities bias and finally, (f) biases in machinery and algorithms. Conclusions 1.4Machine learning and data-driven solutions can potentially transform healthcare delivery, but not without limitations. The core inputs in the systems (data and human factors) currently contain several sources of bias that are poorly documented and analyzed for remedies. The current evidence heavily focuses on data-related biases, while other sources are less often analyzed or anecdotal. However, these different sources of biases add to one another exponentially. Therefore, to understand the issues holistically we need to explore these diverse sources of bias. While racial biases in EHR have been often documented, other sources of biases have been less frequently investigated and documented (e.g. gender-related biases, sexual orientation discrimination, socially induced biases, and implicit, often unconscious, human-related cognitive biases). Moreover, some existing studies lack causal evidence, illustrating the different prevalences of disease across groups, which does not per se prove the causality. Our review shows that data-, human- and machine biases are prevalent in healthcare and they significantly impact healthcare outcomes and judgments and exacerbate disparities and differential treatment. Understanding how diverse biases affect AI systems and recommendations is critical. We suggest that researchers and medical personnel should develop safeguards and adopt data-driven solutions with a "bias-in-mind" approach. More empirical evidence is needed to tease out the effects of different sources of bias on health outcomes.
Collapse
|
30
|
Barakat A, Munro G, Heegaard AM. Finding new analgesics: Computational pharmacology faces drug discovery challenges. Biochem Pharmacol 2024; 222:116091. [PMID: 38412924 DOI: 10.1016/j.bcp.2024.116091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2023] [Revised: 01/10/2024] [Accepted: 02/23/2024] [Indexed: 02/29/2024]
Abstract
Despite the worldwide prevalence and huge burden of pain, pain is an undertreated phenomenon. Currently used analgesics have several limitations regarding their efficacy and safety. The discovery of analgesics possessing a novel mechanism of action has faced multiple challenges, including a limited understanding of biological processes underpinning pain and analgesia and poor animal-to-human translation. Computational pharmacology is currently employed to face these challenges. In this review, we discuss the theory, methods, and applications of computational pharmacology in pain research. Computational pharmacology encompasses a wide variety of theoretical concepts and practical methodological approaches, with the overall aim of gaining biological insight through data acquisition and analysis. Data are acquired from patients or animal models with pain or analgesic treatment, at different levels of biological organization (molecular, cellular, physiological, and behavioral). Distinct methodological algorithms can then be used to analyze and integrate data. This helps to facilitate the identification of biological molecules and processes associated with pain phenotype, build quantitative models of pain signaling, and extract translatable features between humans and animals. However, computational pharmacology has several limitations, and its predictions can provide false positive and negative findings. Therefore, computational predictions are required to be validated experimentally before drawing solid conclusions. In this review, we discuss several case study examples of combining and integrating computational tools with experimental pain research tools to meet drug discovery challenges.
Collapse
Affiliation(s)
- Ahmed Barakat
- Department of Drug Design and Pharmacology, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark; Department of Pharmacology and Toxicology, Faculty of Pharmacy, Assiut University, Assiut, Egypt.
| | | | - Anne-Marie Heegaard
- Department of Drug Design and Pharmacology, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| |
Collapse
|
31
|
Ferreira A, Li J, Pomykala KL, Kleesiek J, Alves V, Egger J. GAN-based generation of realistic 3D volumetric data: A systematic review and taxonomy. Med Image Anal 2024; 93:103100. [PMID: 38340545 DOI: 10.1016/j.media.2024.103100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 11/20/2023] [Accepted: 01/30/2024] [Indexed: 02/12/2024]
Abstract
With the massive proliferation of data-driven algorithms, such as deep learning-based approaches, the availability of high-quality data is of great interest. Volumetric data is very important in medicine, as it ranges from disease diagnoses to therapy monitoring. When the dataset is sufficient, models can be trained to help doctors with these tasks. Unfortunately, there are scenarios where large amounts of data is unavailable. For example, rare diseases and privacy issues can lead to restricted data availability. In non-medical fields, the high cost of obtaining enough high-quality data can also be a concern. A solution to these problems can be the generation of realistic synthetic data using Generative Adversarial Networks (GANs). The existence of these mechanisms is a good asset, especially in healthcare, as the data must be of good quality, realistic, and without privacy issues. Therefore, most of the publications on volumetric GANs are within the medical domain. In this review, we provide a summary of works that generate realistic volumetric synthetic data using GANs. We therefore outline GAN-based methods in these areas with common architectures, loss functions and evaluation metrics, including their advantages and disadvantages. We present a novel taxonomy, evaluations, challenges, and research opportunities to provide a holistic overview of the current state of volumetric GANs.
Collapse
Affiliation(s)
- André Ferreira
- Center Algoritmi/LASI, University of Minho, Braga, 4710-057, Portugal; Computer Algorithms for Medicine Laboratory, Graz, Austria; Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany; Institute of Medical Informatics, University Hospital RWTH Aachen, 52074 Aachen, Germany.
| | - Jianning Li
- Computer Algorithms for Medicine Laboratory, Graz, Austria; Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, Essen, 45147, Germany.
| | - Kelsey L Pomykala
- Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany.
| | - Jens Kleesiek
- Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, Essen, 45147, Germany; German Cancer Consortium (DKTK), Partner Site Essen, Hufelandstraße 55, Essen, 45147, Germany; TU Dortmund University, Department of Physics, Otto-Hahn-Straße 4, 44227 Dortmund, Germany.
| | - Victor Alves
- Center Algoritmi/LASI, University of Minho, Braga, 4710-057, Portugal.
| | - Jan Egger
- Computer Algorithms for Medicine Laboratory, Graz, Austria; Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, Essen, 45147, Germany; Institute of Computer Graphics and Vision, Graz University of Technology, Inffeldgasse 16, Graz, 801, Austria.
| |
Collapse
|
32
|
Ktena I, Wiles O, Albuquerque I, Rebuffi SA, Tanno R, Roy AG, Azizi S, Belgrave D, Kohli P, Cemgil T, Karthikesalingam A, Gowal S. Generative models improve fairness of medical classifiers under distribution shifts. Nat Med 2024; 30:1166-1173. [PMID: 38600282 PMCID: PMC11031395 DOI: 10.1038/s41591-024-02838-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Accepted: 01/26/2024] [Indexed: 04/12/2024]
Abstract
Domain generalization is a ubiquitous challenge for machine learning in healthcare. Model performance in real-world conditions might be lower than expected because of discrepancies between the data encountered during deployment and development. Underrepresentation of some groups or conditions during model development is a common cause of this phenomenon. This challenge is often not readily addressed by targeted data acquisition and 'labeling' by expert clinicians, which can be prohibitively expensive or practically impossible because of the rarity of conditions or the available clinical expertise. We hypothesize that advances in generative artificial intelligence can help mitigate this unmet need in a steerable fashion, enriching our training dataset with synthetic examples that address shortfalls of underrepresented conditions or subgroups. We show that diffusion models can automatically learn realistic augmentations from data in a label-efficient manner. We demonstrate that learned augmentations make models more robust and statistically fair in-distribution and out of distribution. To evaluate the generality of our approach, we studied three distinct medical imaging contexts of varying difficulty: (1) histopathology, (2) chest X-ray and (3) dermatology images. Complementing real samples with synthetic ones improved the robustness of models in all three medical tasks and increased fairness by improving the accuracy of clinical diagnosis within underrepresented groups, especially out of distribution.
Collapse
|
33
|
Jiang M, Wang S, Song Z, Song L, Wang Y, Zhu C, Zheng Q. Cross 2SynNet: cross-device-cross-modal synthesis of routine brain MRI sequences from CT with brain lesion. MAGMA (NEW YORK, N.Y.) 2024; 37:241-256. [PMID: 38315352 DOI: 10.1007/s10334-023-01145-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 11/28/2023] [Accepted: 12/27/2023] [Indexed: 02/07/2024]
Abstract
OBJECTIVES CT and MR are often needed to determine the location and extent of brain lesions collectively to improve diagnosis. However, patients with acute brain diseases cannot complete the MRI examination within a short time. The aim of the study is to devise a cross-device and cross-modal medical image synthesis (MIS) method Cross2SynNet for synthesizing routine brain MRI sequences of T1WI, T2WI, FLAIR, and DWI from CT with stroke and brain tumors. MATERIALS AND METHODS For the retrospective study, the participants covered four different diseases of cerebral ischemic stroke (CIS-cohort), cerebral hemorrhage (CH-cohort), meningioma (M-cohort), glioma (G-cohort). The MIS model Cross2SynNet was established on the basic architecture of conditional generative adversarial network (CGAN), of which, the fully convolutional Transformer (FCT) module was adopted into generator to capture the short- and long-range dependencies between healthy and pathological tissues, and the edge loss function was to minimize the difference in gradient magnitude between synthetic image and ground truth. Three metrics of mean square error (MSE), peak signal-to-noise ratio (PSNR), and structure similarity index measure (SSIM) were used for evaluation. RESULTS A total of 230 participants (mean patient age, 59.77 years ± 13.63 [standard deviation]; 163 men [71%] and 67 women [29%]) were included, including CIS-cohort (95 participants between Dec 2019 and Feb 2022), CH-cohort (69 participants between Jan 2020 and Dec 2021), M-cohort (40 participants between Sep 2018 and Dec 2021), and G-cohort (26 participants between Sep 2019 and Dec 2021). The Cross2SynNet achieved averaged values of MSE = 0.008, PSNR = 21.728, and SSIM = 0.758 when synthesizing MRIs from CT, outperforming the CycleGAN, pix2pix, RegGAN, Pix2PixHD, and ResViT. The Cross2SynNet could synthesize the brain lesion on pseudo DWI even if the CT image did not exhibit clear signal in the acute ischemic stroke patients. CONCLUSIONS Cross2SynNet could achieve routine brain MRI synthesis of T1WI, T2WI, FLAIR, and DWI from CT with promising performance given the brain lesion of stroke and brain tumor.
Collapse
Affiliation(s)
- Minbo Jiang
- School of Computer and Control Engineering, Yantai University, No 30, Qingquan Road, Laishan District, Yantai, 264005, Shandong, China
| | - Shuai Wang
- Department of Radiology, Binzhou Medical University Hospital, Binzhou, 256603, China
| | - Zhiwei Song
- School of Computer and Control Engineering, Yantai University, No 30, Qingquan Road, Laishan District, Yantai, 264005, Shandong, China
| | - Limei Song
- School of Medical Imaging, Weifang Medical University, Weifang, 261000, China
| | - Yi Wang
- School of Computer and Control Engineering, Yantai University, No 30, Qingquan Road, Laishan District, Yantai, 264005, Shandong, China
| | - Chuanzhen Zhu
- School of Computer and Control Engineering, Yantai University, No 30, Qingquan Road, Laishan District, Yantai, 264005, Shandong, China
| | - Qiang Zheng
- School of Computer and Control Engineering, Yantai University, No 30, Qingquan Road, Laishan District, Yantai, 264005, Shandong, China.
| |
Collapse
|
34
|
Carrillo-Perez F, Pizurica M, Zheng Y, Nandi TN, Madduri R, Shen J, Gevaert O. Generation of synthetic whole-slide image tiles of tumours from RNA-sequencing data via cascaded diffusion models. Nat Biomed Eng 2024:10.1038/s41551-024-01193-8. [PMID: 38514775 DOI: 10.1038/s41551-024-01193-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2022] [Accepted: 02/29/2024] [Indexed: 03/23/2024]
Abstract
Training machine-learning models with synthetically generated data can alleviate the problem of data scarcity when acquiring diverse and sufficiently large datasets is costly and challenging. Here we show that cascaded diffusion models can be used to synthesize realistic whole-slide image tiles from latent representations of RNA-sequencing data from human tumours. Alterations in gene expression affected the composition of cell types in the generated synthetic image tiles, which accurately preserved the distribution of cell types and maintained the cell fraction observed in bulk RNA-sequencing data, as we show for lung adenocarcinoma, kidney renal papillary cell carcinoma, cervical squamous cell carcinoma, colon adenocarcinoma and glioblastoma. Machine-learning models pretrained with the generated synthetic data performed better than models trained from scratch. Synthetic data may accelerate the development of machine-learning models in scarce-data settings and allow for the imputation of missing data modalities.
Collapse
Affiliation(s)
- Francisco Carrillo-Perez
- Stanford Center for Biomedical Informatics Research (BMIR), Stanford University, School of Medicine, Stanford, CA, USA
| | - Marija Pizurica
- Stanford Center for Biomedical Informatics Research (BMIR), Stanford University, School of Medicine, Stanford, CA, USA
- Internet technology and Data science Lab (IDLab), Ghent University, Ghent, Belgium
| | - Yuanning Zheng
- Stanford Center for Biomedical Informatics Research (BMIR), Stanford University, School of Medicine, Stanford, CA, USA
| | - Tarak Nath Nandi
- Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA
| | - Ravi Madduri
- Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA
| | - Jeanne Shen
- Department of Pathology, Stanford University, School of Medicine, Palo Alto, CA, USA
| | - Olivier Gevaert
- Stanford Center for Biomedical Informatics Research (BMIR), Stanford University, School of Medicine, Stanford, CA, USA.
- Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, CA, USA.
| |
Collapse
|
35
|
Quistberg DA. Potential of artificial intelligence in injury prevention research and practice. Inj Prev 2024; 30:89-91. [PMID: 38307714 PMCID: PMC11003389 DOI: 10.1136/ip-2023-045203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 01/18/2024] [Indexed: 02/04/2024]
Abstract
There is increasing interest and use of artificial Intelligence algorithms and methods in biomedical research and practice, particularly as the technology has made significant advances in the past decade and become more accessible to more disciplines. This editorial briefly reviews this technology and its potential for injury prevention research and practice, proposing ways that it can be used to advance the discipline, as well as the potential pitfalls, concerns and biases that accompany it.
Collapse
Affiliation(s)
- D Alex Quistberg
- Urban Health Collaborative, Drexel University, Philadelphia, Pennsylvania, USA
- Environmental & Occupational Health, Drexel University, Philadelphia, Pennsylvania, USA
| |
Collapse
|
36
|
Eckardt JN, Hahn W, Röllig C, Stasik S, Platzbecker U, Müller-Tidow C, Serve H, Baldus CD, Schliemann C, Schäfer-Eckart K, Hanoun M, Kaufmann M, Burchert A, Thiede C, Schetelig J, Sedlmayr M, Bornhäuser M, Wolfien M, Middeke JM. Mimicking clinical trials with synthetic acute myeloid leukemia patients using generative artificial intelligence. NPJ Digit Med 2024; 7:76. [PMID: 38509224 PMCID: PMC10954666 DOI: 10.1038/s41746-024-01076-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2023] [Accepted: 03/07/2024] [Indexed: 03/22/2024] Open
Abstract
Clinical research relies on high-quality patient data, however, obtaining big data sets is costly and access to existing data is often hindered by privacy and regulatory concerns. Synthetic data generation holds the promise of effectively bypassing these boundaries allowing for simplified data accessibility and the prospect of synthetic control cohorts. We employed two different methodologies of generative artificial intelligence - CTAB-GAN+ and normalizing flows (NFlow) - to synthesize patient data derived from 1606 patients with acute myeloid leukemia, a heterogeneous hematological malignancy, that were treated within four multicenter clinical trials. Both generative models accurately captured distributions of demographic, laboratory, molecular and cytogenetic variables, as well as patient outcomes yielding high performance scores regarding fidelity and usability of both synthetic cohorts (n = 1606 each). Survival analysis demonstrated close resemblance of survival curves between original and synthetic cohorts. Inter-variable relationships were preserved in univariable outcome analysis enabling explorative analysis in our synthetic data. Additionally, training sample privacy is safeguarded mitigating possible patient re-identification, which we quantified using Hamming distances. We provide not only a proof-of-concept for synthetic data generation in multimodal clinical data for rare diseases, but also full public access to synthetic data sets to foster further research.
Collapse
Affiliation(s)
- Jan-Niklas Eckardt
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Else Kröner Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
| | - Waldemar Hahn
- Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig, Leipzig, Germany
- Institute for Medical Informatics and Biometry, Technical University Dresden, Dresden, Germany
| | - Christoph Röllig
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Sebastian Stasik
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Uwe Platzbecker
- Medical Clinic and Policlinic I Hematology and Cell Therapy, University Hospital, Leipzig, Germany
| | | | - Hubert Serve
- Department of Medicine 2, Hematology and Oncology, Goethe University Frankfurt, Frankfurt, Germany
| | - Claudia D Baldus
- Department of Hematology and Oncology, University Hospital Schleswig Holstein, Kiel, Germany
| | | | - Kerstin Schäfer-Eckart
- Department of Internal Medicine V, Paracelsus Medizinische Privatuniversität and University Hospital Nürnberg, Nürnberg, Germany
| | - Maher Hanoun
- Department of Hematology, University Hospital Essen, Essen, Germany
| | - Martin Kaufmann
- Department of Hematology, Oncology and Palliative Care, Robert-Bosch-Hospital, Stuttgart, Germany
| | - Andreas Burchert
- Department of Hematology, Oncology and Immunology, Philipps-University-Marburg, Marburg, Germany
| | - Christian Thiede
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Johannes Schetelig
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Martin Sedlmayr
- Institute for Medical Informatics and Biometry, Technical University Dresden, Dresden, Germany
| | - Martin Bornhäuser
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- German Consortium for Translational Cancer Research DKFZ, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Dresden, Germany
| | - Markus Wolfien
- Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig, Leipzig, Germany
- Institute for Medical Informatics and Biometry, Technical University Dresden, Dresden, Germany
| | - Jan Moritz Middeke
- Department of Internal Medicine I, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Else Kröner Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| |
Collapse
|
37
|
Chen Y, Esmaeilzadeh P. Generative AI in Medical Practice: In-Depth Exploration of Privacy and Security Challenges. J Med Internet Res 2024; 26:e53008. [PMID: 38457208 PMCID: PMC10960211 DOI: 10.2196/53008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2023] [Revised: 12/12/2023] [Accepted: 01/31/2024] [Indexed: 03/09/2024] Open
Abstract
As advances in artificial intelligence (AI) continue to transform and revolutionize the field of medicine, understanding the potential uses of generative AI in health care becomes increasingly important. Generative AI, including models such as generative adversarial networks and large language models, shows promise in transforming medical diagnostics, research, treatment planning, and patient care. However, these data-intensive systems pose new threats to protected health information. This Viewpoint paper aims to explore various categories of generative AI in health care, including medical diagnostics, drug discovery, virtual health assistants, medical research, and clinical decision support, while identifying security and privacy threats within each phase of the life cycle of such systems (ie, data collection, model development, and implementation phases). The objectives of this study were to analyze the current state of generative AI in health care, identify opportunities and privacy and security challenges posed by integrating these technologies into existing health care infrastructure, and propose strategies for mitigating security and privacy risks. This study highlights the importance of addressing the security and privacy threats associated with generative AI in health care to ensure the safe and effective use of these systems. The findings of this study can inform the development of future generative AI systems in health care and help health care organizations better understand the potential benefits and risks associated with these systems. By examining the use cases and benefits of generative AI across diverse domains within health care, this paper contributes to theoretical discussions surrounding AI ethics, security vulnerabilities, and data privacy regulations. In addition, this study provides practical insights for stakeholders looking to adopt generative AI solutions within their organizations.
Collapse
Affiliation(s)
- Yan Chen
- Department of Information Systems and Business Analytics, College of Business, Florida International University, Miami, FL, United States
| | - Pouyan Esmaeilzadeh
- Department of Information Systems and Business Analytics, College of Business, Florida International University, Miami, FL, United States
| |
Collapse
|
38
|
Umer F, Adnan N. Generative artificial intelligence: synthetic datasets in dentistry. BDJ Open 2024; 10:13. [PMID: 38429258 PMCID: PMC10907705 DOI: 10.1038/s41405-024-00198-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Revised: 02/15/2024] [Accepted: 02/16/2024] [Indexed: 03/03/2024] Open
Abstract
INTRODUCTION Artificial Intelligence (AI) algorithms, particularly Deep Learning (DL) models are known to be data intensive. This has increased the demand for digital data in all domains of healthcare, including dentistry. The main hindrance in the progress of AI is access to diverse datasets which train DL models ensuring optimal performance, comparable to subject experts. However, administration of these traditionally acquired datasets is challenging due to privacy regulations and the extensive manual annotation required by subject experts. Biases such as ethical, socioeconomic and class imbalances are also incorporated during the curation of these datasets, limiting their overall generalizability. These challenges prevent their accrual at a larger scale for training DL models. METHODS Generative AI techniques can be useful in the production of Synthetic Datasets (SDs) that can overcome issues affecting traditionally acquired datasets. Variational autoencoders, generative adversarial networks and diffusion models have been used to generate SDs. The following text is a review of these generative AI techniques and their operations. It discusses the chances of SDs and challenges with potential solutions which will improve the understanding of healthcare professionals working in AI research. CONCLUSION Synthetic data customized to the need of researchers can be produced to train robust AI models. These models, having been trained on such a diverse dataset will be applicable for dissemination across countries. However, there is a need for the limitations associated with SDs to be better understood, and attempts made to overcome those concerns prior to their widespread use.
Collapse
Affiliation(s)
- Fahad Umer
- Operative Dentistry and Endodontics, Department of Surgery, Aga Khan University Hospital, Karachi, Pakistan
| | - Niha Adnan
- Operative Dentistry and Endodontics, Department of Surgery, Aga Khan University Hospital, Karachi, Pakistan.
| |
Collapse
|
39
|
Qiao M, Wang S, Qiu H, de Marvao A, O’Regan DP, Rueckert D, Bai W. CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1259-1269. [PMID: 37948142 PMCID: PMC7615911 DOI: 10.1109/tmi.2023.3331982] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/12/2023]
Abstract
Two key questions in cardiac image analysis are to assess the anatomy and motion of the heart from images; and to understand how they are associated with non-imaging clinical factors such as gender, age and diseases. While the first question can often be addressed by image segmentation and motion tracking algorithms, our capability to model and answer the second question is still limited. In this work, we propose a novel conditional generative model to describe the 4D spatio-temporal anatomy of the heart and its interaction with non-imaging clinical factors. The clinical factors are integrated as the conditions of the generative modelling, which allows us to investigate how these factors influence the cardiac anatomy. We evaluate the model performance in mainly two tasks, anatomical sequence completion and sequence generation. The model achieves high performance in anatomical sequence completion, comparable to or outperforming other state-of-the-art generative models. In terms of sequence generation, given clinical conditions, the model can generate realistic synthetic 4D sequential anatomies that share similar distributions with the real data. The code and the trained generative model are available at https://github.com/MengyunQ/CHeart.
Collapse
Affiliation(s)
- Mengyun Qiao
- Department of Computing, Department of Brain Sciences and Data Science Institute, Imperial College London, London, SW7 2AZ, United Kingdom
| | - Shuo Wang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University and Shanghai Key Laboratory of MICCAI, Shanghai, China
| | - Huaqi Qiu
- Biomedical Image Analysis Group (BioMedIA), Department of Computing, Imperial College London
| | - Antonio de Marvao
- MRC Laboratory of Medical Sciences, Imperial College London, London W12 0HS, United Kingdom; Department of Women and Children’s Health, and British Heart Foundation Centre of Research Excellence, School of Cardiovascular and Metabolic Medicine and Sciences, King’s College London, London, United Kingdom
| | - Declan P. O’Regan
- MRC Laboratory of Medical Sciences, Imperial College London, London W12 0HS, United Kingdom
| | - Daniel Rueckert
- Biomedical Image Analysis Group (BioMedIA), Department of Computing, Imperial College London; Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Wenjia Bai
- Department of Computing, Department of Brain Sciences and Data Science Institute, Imperial College London, London, SW7 2AZ, United Kingdom
| |
Collapse
|
40
|
Darvish M, Kist AM. A Generative Method for a Laryngeal Biosignal. J Voice 2024:S0892-1997(24)00019-5. [PMID: 38395653 DOI: 10.1016/j.jvoice.2024.01.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Revised: 01/26/2024] [Accepted: 01/26/2024] [Indexed: 02/25/2024]
Abstract
The Glottal Area Waveform (GAW) is an important component in quantitative clinical voice assessment, providing valuable insights into vocal fold function. In this study, we introduce a novel method employing Variational Autoencoders (VAEs) to generate synthetic GAWs. Our approach enables the creation of synthetic GAWs that closely replicate real-world data, offering a versatile tool for researchers and clinicians. We elucidate the process of manipulating the VAE latent space using the Glottal Opening Vector (GlOVe). The GlOVe allows precise control over the synthetic closure and opening of the vocal folds. By utilizing the GlOVe, we generate synthetic laryngeal biosignals. These biosignals accurately reflect vocal fold behavior, allowing for the emulation of realistic glottal opening changes. This manipulation extends to the introduction of arbitrary oscillations in the vocal folds, closely resembling real vocal fold oscillations. The range of factor coefficient values enables the generation of diverse biosignals with varying frequencies and amplitudes. Our results demonstrate that this approach yields highly accurate laryngeal biosignals, with the Normalized Mean Absolute Error values for various frequencies ranging from 9.6 ⋅ 10-3 to 1.20 ⋅ 10-2 for different experimented frequencies, alongside a remarkable training effectiveness, reflected in reductions of up to approximately 89.52% in key loss components. This proposed method may have implications for downstream speech synthesis and phonetics research, offering the potential for advanced and natural-sounding speech technologies.
Collapse
Affiliation(s)
- Mahdi Darvish
- Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Andreas M Kist
- Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
| |
Collapse
|
41
|
Thakur A, Zhu T, Abrol V, Armstrong J, Wang Y, Clifton DA. Data encoding for healthcare data democratization and information leakage prevention. Nat Commun 2024; 15:1582. [PMID: 38383571 PMCID: PMC10882022 DOI: 10.1038/s41467-024-45777-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Accepted: 02/05/2024] [Indexed: 02/23/2024] Open
Abstract
The lack of data democratization and information leakage from trained models hinder the development and acceptance of robust deep learning-based healthcare solutions. This paper argues that irreversible data encoding can provide an effective solution to achieve data democratization without violating the privacy constraints imposed on healthcare data and clinical models. An ideal encoding framework transforms the data into a new space where it is imperceptible to a manual or computational inspection. However, encoded data should preserve the semantics of the original data such that deep learning models can be trained effectively. This paper hypothesizes the characteristics of the desired encoding framework and then exploits random projections and random quantum encoding to realize this framework for dense and longitudinal or time-series data. Experimental evaluation highlights that models trained on encoded time-series data effectively uphold the information bottleneck principle and hence, exhibit lesser information leakage from trained models.
Collapse
Affiliation(s)
- Anshul Thakur
- Department of Engineering Science, University of Oxford, OX3 7DQ, Oxfordshire, UK.
| | - Tingting Zhu
- Department of Engineering Science, University of Oxford, OX3 7DQ, Oxfordshire, UK
| | | | - Jacob Armstrong
- Department of Engineering Science, University of Oxford, OX3 7DQ, Oxfordshire, UK
| | - Yujiang Wang
- Department of Engineering Science, University of Oxford, OX3 7DQ, Oxfordshire, UK.
- Oxford Suzhou Centre for Advanced Research, Suzhou, China.
| | - David A Clifton
- Department of Engineering Science, University of Oxford, OX3 7DQ, Oxfordshire, UK
- Oxford Suzhou Centre for Advanced Research, Suzhou, China
| |
Collapse
|
42
|
Soltan AAS, Thakur A, Yang J, Chauhan A, D'Cruz LG, Dickson P, Soltan MA, Thickett DR, Eyre DW, Zhu T, Clifton DA. A scalable federated learning solution for secondary care using low-cost microcomputing: privacy-preserving development and evaluation of a COVID-19 screening test in UK hospitals. Lancet Digit Health 2024; 6:e93-e104. [PMID: 38278619 DOI: 10.1016/s2589-7500(23)00226-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Revised: 10/17/2023] [Accepted: 10/30/2023] [Indexed: 01/28/2024]
Abstract
BACKGROUND Multicentre training could reduce biases in medical artificial intelligence (AI); however, ethical, legal, and technical considerations can constrain the ability of hospitals to share data. Federated learning enables institutions to participate in algorithm development while retaining custody of their data but uptake in hospitals has been limited, possibly as deployment requires specialist software and technical expertise at each site. We previously developed an artificial intelligence-driven screening test for COVID-19 in emergency departments, known as CURIAL-Lab, which uses vital signs and blood tests that are routinely available within 1 h of a patient's arrival. Here we aimed to federate our COVID-19 screening test by developing an easy-to-use embedded system-which we introduce as full-stack federated learning-to train and evaluate machine learning models across four UK hospital groups without centralising patient data. METHODS We supplied a Raspberry Pi 4 Model B preloaded with our federated learning software pipeline to four National Health Service (NHS) hospital groups in the UK: Oxford University Hospitals NHS Foundation Trust (OUH; through the locally linked research University, University of Oxford), University Hospitals Birmingham NHS Foundation Trust (UHB), Bedfordshire Hospitals NHS Foundation Trust (BH), and Portsmouth Hospitals University NHS Trust (PUH). OUH, PUH, and UHB participated in federated training, training a deep neural network and logistic regressor over 150 rounds to form and calibrate a global model to predict COVID-19 status, using clinical data from patients admitted before the pandemic (COVID-19-negative) and testing positive for COVID-19 during the first wave of the pandemic. We conducted a federated evaluation of the global model for admissions during the second wave of the pandemic at OUH, PUH, and externally at BH. For OUH and PUH, we additionally performed local fine-tuning of the global model using the sites' individual training data, forming a site-tuned model, and evaluated the resultant model for admissions during the second wave of the pandemic. This study included data collected between Dec 1, 2018, and March 1, 2021; the exact date ranges used varied by site. The primary outcome was overall model performance, measured as the area under the receiver operating characteristic curve (AUROC). Removable micro secure digital (microSD) storage was destroyed on study completion. FINDINGS Clinical data from 130 941 patients (1772 COVID-19-positive), routinely collected across three hospital groups (OUH, PUH, and UHB), were included in federated training. The evaluation step included data from 32 986 patients (3549 COVID-19-positive) attending OUH, PUH, or BH during the second wave of the pandemic. Federated training of a global deep neural network classifier improved upon performance of models trained locally in terms of AUROC by a mean of 27·6% (SD 2·2): AUROC increased from 0·574 (95% CI 0·560-0·589) at OUH and 0·622 (0·608-0·637) at PUH using the locally trained models to 0·872 (0·862-0·882) at OUH and 0·876 (0·865-0·886) at PUH using the federated global model. Performance improvement was smaller for a logistic regression model, with a mean increase in AUROC of 13·9% (0·5%). During federated external evaluation at BH, AUROC for the global deep neural network model was 0·917 (0·893-0·942), with 89·7% sensitivity (83·6-93·6) and 76·6% specificity (73·9-79·1). Site-specific tuning of the global model did not significantly improve performance (change in AUROC <0·01). INTERPRETATION We developed an embedded system for federated learning, using microcomputing to optimise for ease of deployment. We deployed full-stack federated learning across four UK hospital groups to develop a COVID-19 screening test without centralising patient data. Federation improved model performance, and the resultant global models were generalisable. Full-stack federated learning could enable hospitals to contribute to AI development at low cost and without specialist technical expertise at each site. FUNDING The Wellcome Trust, University of Oxford Medical and Life Sciences Translational Fund.
Collapse
Affiliation(s)
- Andrew A S Soltan
- Oxford University Hospitals NHS Foundation Trust, Oxford, UK; Department of Oncology, University of Oxford, Oxford, UK; Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK; Big Data Institute, Nuffield Department of Population Health, University of Oxford, Oxford, UK; Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK.
| | - Anshul Thakur
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
| | - Jenny Yang
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
| | - Anoop Chauhan
- Portsmouth Hospitals University NHS Trust, Portsmouth, UK
| | - Leon G D'Cruz
- Portsmouth Hospitals University NHS Trust, Portsmouth, UK
| | | | - Marina A Soltan
- The Queen Elizabeth Hospital, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK; Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
| | - David R Thickett
- The Queen Elizabeth Hospital, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK; Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
| | - David W Eyre
- Oxford University Hospitals NHS Foundation Trust, Oxford, UK; Big Data Institute, Nuffield Department of Population Health, University of Oxford, Oxford, UK; NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance, University of Oxford and Public Health England, Oxford, UK; NIHR Oxford Biomedical Research Centre, Oxford, UK
| | - Tingting Zhu
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
| | - David A Clifton
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK; NIHR Oxford Biomedical Research Centre, Oxford, UK; Oxford-Suzhou Centre for Advanced Research, Suzhou, China
| |
Collapse
|
43
|
Prasser F, Riedel N, Wolter S, Corr D, Ludwig M. [Artificial intelligence and secure use of health data in the KI-FDZ project: anonymization, synthetization, and secure processing of real-world data]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz 2024; 67:171-179. [PMID: 38175194 PMCID: PMC10834625 DOI: 10.1007/s00103-023-03823-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Accepted: 12/08/2023] [Indexed: 01/05/2024]
Abstract
The increasing digitization of the healthcare system is leading to a growing volume of health data. Leveraging this data beyond its initial collection purpose for secondary use can provide valuable insights into diagnostics, treatment processes, and the quality of care. The Health Data Lab (HDL) will provide infrastructure for this purpose. Both the protection of patient privacy and optimal analytical capabilities are of central importance in this context, and artificial intelligence (AI) provides two opportunities. First, it enables the analysis of large volumes of data with flexible models, which means that hidden correlations and patterns can be discovered. Second, synthetic - that is, artificial - data generated by AI can protect privacy.This paper describes the KI-FDZ project, which aims to investigate innovative technologies that can support the secure provision of health data for secondary research purposes. A multi-layered approach is investigated in which data-level measures can be combined in different ways with processing in secure environments. To this end, anonymization and synthetization methods, among others, are evaluated based on two concrete application examples. Moreover, it is examined how the creation of machine learning pipelines and the execution of AI algorithms can be supported in secure processing environments. Preliminary results indicate that this approach can achieve a high level of protection while maintaining data validity. The approach investigated in the project can be an important building block in the secure secondary use of health data.
Collapse
Affiliation(s)
- Fabian Prasser
- Center für Health Data Science, Berlin Institute of Health der Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Deutschland.
| | - Nico Riedel
- Forschungsdatenzentrum Gesundheit, Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM), Bonn, Deutschland
| | - Steven Wolter
- Forschungsdatenzentrum Gesundheit, Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM), Bonn, Deutschland
| | - Dörte Corr
- Fraunhofer-Institut für Digitale Medizin MEVIS, Bremen, Deutschland
| | - Marion Ludwig
- InGef - Institut für angewandte Gesundheitsforschung Berlin GmbH, Berlin, Deutschland
| |
Collapse
|
44
|
de-la-Torre R, Oña ED, Victores JG, Jardón A. SpasticSim: a synthetic data generation method for upper limb spasticity modelling in neurorehabilitation. Sci Rep 2024; 14:1646. [PMID: 38238475 PMCID: PMC10796340 DOI: 10.1038/s41598-024-51993-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 01/11/2024] [Indexed: 01/22/2024] Open
Abstract
In neurorehabilitation, assessment of functional problems is essential to define optimal rehabilitation treatments. Usually, this assessment process requires distinguishing between impaired and non-impaired behavior of limbs. One of the common muscle motor disorders affecting limbs is spasticity, which is complicated to quantify objectively due to the complex nature of motor control. Thus, the lack of heterogeneous samples of patients constituting an acceptable amount of data is an obstacle which is relevant to understanding the behavior of spasticity and, consequently, quantifying it. In this article, we use the 3D creation suite Blender combined with the MBLab add-on to generate synthetic samples of human body models, aiming to be as sufficiently representative as possible to real human samples. Exporting these samples to OpenSim and performing four specific upper limb movements, we analyze the muscle behavior by simulating the six degrees of spasticity contemplated by the Modified Ashworth Scale (MAS). The complete dataset of patients and movements is open-source and available for future research. This approach advocates the potential to generate synthetic data for testing and validating musculoskeletal models.
Collapse
Affiliation(s)
- Rubén de-la-Torre
- Department of Systems Engineering and Automation, Universidad Carlos III de Madrid, Avda. de la Universidad 30, Leganés, 28911, Madrid, Spain
| | - Edwin Daniel Oña
- Department of Systems Engineering and Automation, Universidad Carlos III de Madrid, Avda. de la Universidad 30, Leganés, 28911, Madrid, Spain.
| | - Juan G Victores
- Department of Systems Engineering and Automation, Universidad Carlos III de Madrid, Avda. de la Universidad 30, Leganés, 28911, Madrid, Spain
| | - Alberto Jardón
- Department of Systems Engineering and Automation, Universidad Carlos III de Madrid, Avda. de la Universidad 30, Leganés, 28911, Madrid, Spain
| |
Collapse
|
45
|
Qian N, Gao X, Lang X, Deng H, Bratu TM, Chen Q, Stapleton P, Yan B, Min W. Rapid single-particle chemical imaging of nanoplastics by SRS microscopy. Proc Natl Acad Sci U S A 2024; 121:e2300582121. [PMID: 38190543 PMCID: PMC10801917 DOI: 10.1073/pnas.2300582121] [Citation(s) in RCA: 40] [Impact Index Per Article: 40.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2023] [Accepted: 10/24/2023] [Indexed: 01/10/2024] Open
Abstract
Plastics are now omnipresent in our daily lives. The existence of microplastics (1 µm to 5 mm in length) and possibly even nanoplastics (<1 μm) has recently raised health concerns. In particular, nanoplastics are believed to be more toxic since their smaller size renders them much more amenable, compared to microplastics, to enter the human body. However, detecting nanoplastics imposes tremendous analytical challenges on both the nano-level sensitivity and the plastic-identifying specificity, leading to a knowledge gap in this mysterious nanoworld surrounding us. To address these challenges, we developed a hyperspectral stimulated Raman scattering (SRS) imaging platform with an automated plastic identification algorithm that allows micro-nano plastic analysis at the single-particle level with high chemical specificity and throughput. We first validated the sensitivity enhancement of the narrow band of SRS to enable high-speed single nanoplastic detection below 100 nm. We then devised a data-driven spectral matching algorithm to address spectral identification challenges imposed by sensitive narrow-band hyperspectral imaging and achieve robust determination of common plastic polymers. With the established technique, we studied the micro-nano plastics from bottled water as a model system. We successfully detected and identified nanoplastics from major plastic types. Micro-nano plastics concentrations were estimated to be about 2.4 ± 1.3 × 105 particles per liter of bottled water, about 90% of which are nanoplastics. This is orders of magnitude more than the microplastic abundance reported previously in bottled water. High-throughput single-particle counting revealed extraordinary particle heterogeneity and nonorthogonality between plastic composition and morphologies; the resulting multidimensional profiling sheds light on the science of nanoplastics.
Collapse
Affiliation(s)
- Naixin Qian
- Department of Chemistry, Columbia University, New York, NY10027
| | - Xin Gao
- Department of Chemistry, Columbia University, New York, NY10027
| | - Xiaoqi Lang
- Department of Chemistry, Columbia University, New York, NY10027
| | - Huiping Deng
- Lamont-Doherty Earth Observatory of Columbia University, Palisades, NY10964
| | | | - Qixuan Chen
- Department of Biostatistics, Columbia University Mailman School of Public Health, New York, NY10032
| | - Phoebe Stapleton
- Department of Pharmacology and Toxicology, Ernest Mario School of Pharmacy, Environmental and Occupational Health Sciences Institute, Rutgers University, New Brunswick, NJ08854
| | - Beizhan Yan
- Lamont-Doherty Earth Observatory of Columbia University, Palisades, NY10964
| | - Wei Min
- Department of Chemistry, Columbia University, New York, NY10027
- Department of Biomedical Engineering, Columbia University, New York, NY10027
| |
Collapse
|
46
|
Guevara M, Chen S, Thomas S, Chaunzwa TL, Franco I, Kann BH, Moningi S, Qian JM, Goldstein M, Harper S, Aerts HJWL, Catalano PJ, Savova GK, Mak RH, Bitterman DS. Large language models to identify social determinants of health in electronic health records. NPJ Digit Med 2024; 7:6. [PMID: 38200151 PMCID: PMC10781957 DOI: 10.1038/s41746-023-00970-0] [Citation(s) in RCA: 15] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 11/15/2023] [Indexed: 01/12/2024] Open
Abstract
Social determinants of health (SDoH) play a critical role in patient outcomes, yet their documentation is often missing or incomplete in the structured data of electronic health records (EHRs). Large language models (LLMs) could enable high-throughput extraction of SDoH from the EHR to support research and clinical care. However, class imbalance and data limitations present challenges for this sparsely documented yet critical information. Here, we investigated the optimal methods for using LLMs to extract six SDoH categories from narrative text in the EHR: employment, housing, transportation, parental status, relationship, and social support. The best-performing models were fine-tuned Flan-T5 XL for any SDoH mentions (macro-F1 0.71), and Flan-T5 XXL for adverse SDoH mentions (macro-F1 0.70). Adding LLM-generated synthetic data to training varied across models and architecture, but improved the performance of smaller Flan-T5 models (delta F1 + 0.12 to +0.23). Our best-fine-tuned models outperformed zero- and few-shot performance of ChatGPT-family models in the zero- and few-shot setting, except GPT4 with 10-shot prompting for adverse SDoH. Fine-tuned models were less likely than ChatGPT to change their prediction when race/ethnicity and gender descriptors were added to the text, suggesting less algorithmic bias (p < 0.05). Our models identified 93.8% of patients with adverse SDoH, while ICD-10 codes captured 2.0%. These results demonstrate the potential of LLMs in improving real-world evidence on SDoH and assisting in identifying patients who could benefit from resource support.
Collapse
Affiliation(s)
- Marco Guevara
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | - Shan Chen
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | - Spencer Thomas
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
- Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Tafadzwa L Chaunzwa
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | - Idalid Franco
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | - Benjamin H Kann
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | - Shalini Moningi
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | - Jack M Qian
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | | | - Susan Harper
- Adult Resource Office, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Hugo J W L Aerts
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
- Radiology and Nuclear Medicine, GROW & CARIM, Maastricht University, Maastricht, The Netherlands
| | - Paul J Catalano
- Department of Data Science, Dana-Farber Cancer Institute and Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, MA, USA
| | - Guergana K Savova
- Computational Health Informatics Program, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Raymond H Mak
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA
| | - Danielle S Bitterman
- Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA.
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, MA, USA.
| |
Collapse
|
47
|
Kim H, Jang WS, Sim WS, Kim HS, Choi JE, Baek ES, Park YR, Shin SJ. Synthetic Data Improve Survival Status Prediction Models in Early-Onset Colorectal Cancer. JCO Clin Cancer Inform 2024; 8:e2300201. [PMID: 38271642 PMCID: PMC10830088 DOI: 10.1200/cci.23.00201] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2023] [Revised: 11/19/2023] [Accepted: 12/07/2023] [Indexed: 01/27/2024] Open
Abstract
PURPOSE In artificial intelligence-based modeling, working with a limited number of patient groups is challenging. This retrospective study aimed to evaluate whether applying synthetic data generation methods to the clinical data of small patient groups can enhance the performance of prediction models. MATERIALS AND METHODS A data set collected by the Cancer Registry Library Project from the Yonsei Cancer Center (YCC), Severance Hospital, between January 2008 and October 2020 was reviewed. Patients with colorectal cancer younger than 50 years who started their initial treatment at YCC were included. A Bayesian network-based synthesizing model was used to generate a synthetic data set, combined with the differential privacy (DP) method. RESULTS A synthetic population of 5,005 was generated from a data set of 1,253 patients with 93 clinical features. The Hellinger distance and correlation difference metric were below 0.3 and 0.5, respectively, indicating no statistical difference. The overall survival by disease stage did not differ between the synthetic and original populations. Training with the synthetic data and validating with the original data showed the highest performances of 0.850, 0.836, and 0.790 for the Decision Tree, Random Forest, and XGBoost models, respectively. Comparison of synthetic data sets with different epsilon parameters from the original data sets showed improved performance >0.1%. For extremely small data sets, models using synthetic data outperformed those using only original data sets. The reidentification risk measures demonstrated that the epsilons between 0.1 and 100 fell below the baseline, indicating a preserved privacy state. CONCLUSION The synthetic data generation approach enhances predictive modeling performance by maintaining statistical and clinical integrity, and simultaneously reduces privacy risks through the application of DP techniques.
Collapse
Affiliation(s)
- Hyunwook Kim
- Division of Medical Oncology, Department of Internal Medicine, Yonsei Cancer Center, Severance Hospital, Yonsei University College of Medicine, Seoul, South Korea
| | - Won Seok Jang
- Miner School of Computer & Information Sciences, University of Massachusetts Lowell, Lowell, MA
| | - Woo Seob Sim
- Medical Informatics Collaboration Unit, Department of Research Affairs, Yonsei University College of Medicine, Seoul, South Korea
| | - Han Sang Kim
- Division of Medical Oncology, Department of Internal Medicine, Yonsei Cancer Center, Severance Hospital, Yonsei University College of Medicine, Seoul, South Korea
| | - Jeong Eun Choi
- Office of Data Services at Division of Digital Health, Yonsei University Health System, Seoul, South Korea
| | - Eun Sil Baek
- Songdang Institute for Cancer Research, Yonsei University College of Medicine, Seoul, South Korea
| | - Yu Rang Park
- Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seoul, South Korea
| | - Sang Joon Shin
- Division of Medical Oncology, Department of Internal Medicine, Yonsei Cancer Center, Severance Hospital, Yonsei University College of Medicine, Seoul, South Korea
| |
Collapse
|
48
|
Choi G, Cha WC, Lee SU, Shin SY. Survey of Medical Applications of Federated Learning. Healthc Inform Res 2024; 30:3-15. [PMID: 38359845 PMCID: PMC10879826 DOI: 10.4258/hir.2024.30.1.3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Revised: 01/23/2024] [Accepted: 01/24/2024] [Indexed: 02/17/2024] Open
Abstract
OBJECTIVES Medical artificial intelligence (AI) has recently attracted considerable attention. However, training medical AI models is challenging due to privacy-protection regulations. Among the proposed solutions, federated learning (FL) stands out. FL involves transmitting only model parameters without sharing the original data, making it particularly suitable for the medical field, where data privacy is paramount. This study reviews the application of FL in the medical domain. METHODS We conducted a literature search using the keywords "federated learning" in combination with "medical," "healthcare," or "clinical" on Google Scholar and PubMed. After reviewing titles and abstracts, 58 papers were selected for analysis. These FL studies were categorized based on the types of data used, the target disease, the use of open datasets, the local model of FL, and the neural network model. We also examined issues related to heterogeneity and security. RESULTS In the investigated FL studies, the most commonly used data type was image data, and the most studied target diseases were cancer and COVID-19. The majority of studies utilized open datasets. Furthermore, 72% of the FL articles addressed heterogeneity issues, while 50% discussed security concerns. CONCLUSIONS FL in the medical domain appears to be in its early stages, with most research using open data and focusing on specific data types and diseases for performance verification purposes. Nonetheless, medical FL research is anticipated to be increasingly applied and to become a vital component of multi-institutional research.
Collapse
Affiliation(s)
- Geunho Choi
- Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul,
Korea
| | - Won Chul Cha
- Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul,
Korea
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul,
Korea
| | - Se Uk Lee
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul,
Korea
| | - Soo-Yong Shin
- Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul,
Korea
| |
Collapse
|
49
|
Rafiei A, Ghiasi Rad M, Sikora A, Kamaleswaran R. Improving mixed-integer temporal modeling by generating synthetic data using conditional generative adversarial networks: A case study of fluid overload prediction in the intensive care unit. Comput Biol Med 2024; 168:107749. [PMID: 38011778 DOI: 10.1016/j.compbiomed.2023.107749] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2023] [Revised: 10/29/2023] [Accepted: 11/20/2023] [Indexed: 11/29/2023]
Abstract
OBJECTIVE The challenge of mixed-integer temporal data, which is particularly prominent for medication use in the critically ill, limits the performance of predictive models. The purpose of this evaluation was to pilot test integrating synthetic data within an existing dataset of complex medication data to improve machine learning model prediction of fluid overload. MATERIALS AND METHODS This retrospective cohort study evaluated patients admitted to an ICU ≥ 72 h. Four machine learning algorithms to predict fluid overload after 48-72 h of ICU admission were developed using the original dataset. Then, two distinct synthetic data generation methodologies (synthetic minority over-sampling technique (SMOTE) and conditional tabular generative adversarial network (CTGAN)) were used to create synthetic data. Finally, a stacking ensemble technique designed to train a meta-learner was established. Models underwent training in three scenarios of varying qualities and quantities of datasets. RESULTS Training machine learning algorithms on the combined synthetic and original dataset overall increased the performance of the predictive models compared to training on the original dataset. The highest performing model was the meta-model trained on the combined dataset with 0.83 AUROC while it managed to significantly enhance the sensitivity across different training scenarios. DISCUSSION The integration of synthetically generated data is the first time such methods have been applied to ICU medication data and offers a promising solution to enhance the performance of machine learning models for fluid overload, which may be translated to other ICU outcomes. A meta-learner was able to make a trade-off between different performance metrics and improve the ability to identify the minority class.
Collapse
Affiliation(s)
- Alireza Rafiei
- Department of Computer Science and Informatics, Emory University, Ste. W302, 400 Dowman Dr., Atlanta, GA, 30322, USA.
| | - Milad Ghiasi Rad
- Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA.
| | - Andrea Sikora
- University of Georgia College of Pharmacy, Department of Clinical and Administrative Pharmacy, Augusta, GA, USA.
| | - Rishikesan Kamaleswaran
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, USA; Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA.
| |
Collapse
|
50
|
Ray TR, Kellogg RT, Fargen KM, Hui F, Vargas J. The perils and promises of generative artificial intelligence in neurointerventional surgery. J Neurointerv Surg 2023; 16:4-7. [PMID: 37438101 DOI: 10.1136/jnis-2023-020353] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/26/2023] [Indexed: 07/14/2023]
Abstract
Generative artificial intelligence (AI) holds great promise in neurointerventional surgery by providing clinicians with powerful tools for improving surgical precision, accuracy of diagnoses, and treatment planning. However, potential perils include biases or inaccuracies in the data used to train the algorithms, over-reliance on generative AI without human oversight, patient privacy concerns, and ethical implications of using AI in medical decision-making. Careful regulation and oversight are needed to ensure that the promises of generative AI in neurointerventional surgery are realized while minimizing its potential perils.[ChatGPT authored summary using the prompt "In one paragraph summarize the promises and perils of generative AI in neurointerventional surgery".].
Collapse
Affiliation(s)
- Tyler R Ray
- Department of Mechanical Engineering, University of Hawaii at Mānoa College of Engineering, Honolulu, Hawaii, USA
| | - Ryan T Kellogg
- Department of Neurosurgery, University of Virginia, Charlottesville, Virginia, USA
| | - Kyle M Fargen
- Department of Neurological Surgery and Radiology, Wake Forest University, Winston-Salem, North Carolina, USA
| | - Ferdinand Hui
- Neurointerventional Surgery, Queen's Medical Center Neuroscience Institute, Honolulu, Hawaii, USA
| | - Jan Vargas
- Division of Neurosurgery, Prisma Health Upstate, Greenville, South Carolina, USA
| |
Collapse
|