1
|
Cho H, Froelicher D, Dokmai N, Nandi A, Sadhuka S, Hong MM, Berger B. Privacy-Enhancing Technologies in Biomedical Data Science. Annu Rev Biomed Data Sci 2024; 7:317-343. [PMID: 39178425 PMCID: PMC11346580 DOI: 10.1146/annurev-biodatasci-120423-120107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/25/2024]
Abstract
The rapidly growing scale and variety of biomedical data repositories raise important privacy concerns. Conventional frameworks for collecting and sharing human subject data offer limited privacy protection, often necessitating the creation of data silos. Privacy-enhancing technologies (PETs) promise to safeguard these data and broaden their usage by providing means to share and analyze sensitive data while protecting privacy. Here, we review prominent PETs and illustrate their role in advancing biomedicine. We describe key use cases of PETs and their latest technical advances and highlight recent applications of PETs in a range of biomedical domains. We conclude by discussing outstanding challenges and social considerations that need to be addressed to facilitate a broader adoption of PETs in biomedical data science.
Collapse
Affiliation(s)
- Hyunghoon Cho
- Department of Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, Connecticut, USA;
| | - David Froelicher
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| | - Natnatee Dokmai
- Department of Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, Connecticut, USA;
| | - Anupama Nandi
- Department of Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, Connecticut, USA;
| | - Shuvom Sadhuka
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| | - Matthew M Hong
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
| | - Bonnie Berger
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA;
- Department of Mathematics, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| |
Collapse
|
2
|
Ntampakis N, Argyriou V, Diamantaras K, Goulianas K, Sarigiannidis P, Siniosoglou I. Introducing SPINE: A Holistic Approach to Synthetic Pulmonary Imaging Evaluation Through End-to-End Data and Model Management. IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY 2024; 5:576-588. [PMID: 39157061 PMCID: PMC11329215 DOI: 10.1109/ojemb.2024.3426910] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Revised: 05/25/2024] [Accepted: 07/08/2024] [Indexed: 08/20/2024] Open
Abstract
In the evolving field of medical imaging and machine learning (ML), this paper introduces a novel framework for evaluating synthetic pulmonary imaging aiming to assess synthetic data quality and applicability. Our study concentrates on synthetic X-ray chest images, crucial for diagnosing respiratory diseases. We employ SPINE (Synthetic Pulmonary Imaging Evaluation) framework, a threefold synthetic images evaluation method including expert domain assessment, statistical data analysis and adversarial evaluation. In order to replicate and validate our methodology, we followed an End-to-End data and model management process which begins with a dataset of Normal and Pneumonia chest X-rays, generating synthetic images using Generative Adversarial Networks (GANs) and training a baseline classifier, essential in the adversarial evaluation axis, testing synthetic images against real data assessing their predictive value. The critical outcome of our approach is the post-market analysis of synthetic images. This innovative method evaluates synthetic images using clinical, statistical, and scientific criteria independently from traditional generation performance metrics. This independent evaluation provides deep insights into the clinical and research effectiveness of the synthetic data. By ensuring these images mirror real data's statistical properties and maintain clinical accuracy, our framework establishes a new standard for the ethical and reliable use of synthetic data in medical imaging and research.
Collapse
Affiliation(s)
- Nikolaos Ntampakis
- Department of Information & Electronic EngineeringInternational Hellenic University57001SindosGreece
- MetaMind Innovations50100KozaniGreece
| | | | - Konstantinos Diamantaras
- Department of Information & Electronic EngineeringInternational Hellenic University57001SindosGreece
| | - Konstantinos Goulianas
- Department of Information & Electronic EngineeringInternational Hellenic University57001SindosGreece
| | | | - Ilias Siniosoglou
- MetaMind Innovations50100KozaniGreece
- University of Western Macedonia50100KozaniGreece
| |
Collapse
|
3
|
Heesen P, Elyes M, Domanski J, Schelling G, Könneker S, Fuchs B. Exploring Risk Factors for Predicting 30-Day Postoperative Morbidity in Musculoskeletal Tumor Surgery. J Clin Med 2024; 13:2681. [PMID: 38731210 PMCID: PMC11084328 DOI: 10.3390/jcm13092681] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2024] [Revised: 04/29/2024] [Accepted: 04/30/2024] [Indexed: 05/13/2024] Open
Abstract
Background: This study investigates the risk factors associated with postoperative complications in musculoskeletal tumor surgeries and evaluates the impact of benchmarking in enhancing surgical outcomes. Methods: Conducted at a tertiary referral center, this retrospective analysis included 196 patients who underwent surgeries for various musculoskeletal tumors, ranging from soft tissue to bone sarcomas. Patient and tumor characteristics, along with surgical interventions and outcomes, were comprehensively assessed using the Charlson Comorbidity Index and the Clavien-Dindo classification. Results: Key findings indicate that surgical reconstruction, ASA 3 status, bone tumor presence, and the need for multiple erythrocyte transfusions significantly increase postoperative morbidity. Notably, no significant correlation was found between the Charlson Comorbidity Index scores and the occurrence or severity of complications, challenging the utility of this index in predicting short-term surgical outcomes. Conclusions: This study highlights the importance of tailored surgical approaches and emphasizes rigorous preoperative assessments to mitigate risks and enhance patient care. Despite its insights, limitations include its retrospective nature and single-center scope, suggesting a need for broader, multicenter studies to generalize findings. Overall, our results underscore the necessity of integrating clinical assessments with benchmarking data to optimize outcomes in the complex field of musculoskeletal tumor surgery.
Collapse
Affiliation(s)
- Philip Heesen
- Department of Plastic & Reconstructive Surgery, University Hospital USZ, University of Zurich, 8000 Zurich, Switzerland
| | - Maria Elyes
- Department of Orthopedics & Trauma, Sarcoma Service, LUKS University Hospital, University of Lucerne, 6000 Lucerne, Switzerland
| | - Jan Domanski
- Department of Trauma Surgery and Orthopedics, Ortenau Klinikum, 77654 Offenburg, Germany
| | - Georg Schelling
- Department of Orthopedics & Trauma, Sarcoma Service, LUKS University Hospital, University of Lucerne, 6000 Lucerne, Switzerland
| | - Sören Könneker
- Department of Plastic & Reconstructive Surgery, University Hospital USZ, University of Zurich, 8000 Zurich, Switzerland
| | - Bruno Fuchs
- Department of Orthopedics & Trauma, Sarcoma Service, LUKS University Hospital, University of Lucerne, 6000 Lucerne, Switzerland
- Department of Orthopedics and Trauma, Sarcoma Service, Kantonsspital Winterthur, 8400 Winterthur, Switzerland
| |
Collapse
|
4
|
Vallevik VB, Babic A, Marshall SE, Elvatun S, Brøgger HMB, Alagaratnam S, Edwin B, Veeraragavan NR, Befring AK, Nygård JF. Can I trust my fake data - A comprehensive quality assessment framework for synthetic tabular data in healthcare. Int J Med Inform 2024; 185:105413. [PMID: 38493547 DOI: 10.1016/j.ijmedinf.2024.105413] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2023] [Revised: 02/17/2024] [Accepted: 03/11/2024] [Indexed: 03/19/2024]
Abstract
BACKGROUND Ensuring safe adoption of AI tools in healthcare hinges on access to sufficient data for training, testing and validation. Synthetic data has been suggested in response to privacy concerns and regulatory requirements and can be created by training a generator on real data to produce a dataset with similar statistical properties. Competing metrics with differing taxonomies for quality evaluation have been proposed, resulting in a complex landscape. Optimising quality entails balancing considerations that make the data fit for use, yet relevant dimensions are left out of existing frameworks. METHOD We performed a comprehensive literature review on the use of quality evaluation metrics on synthetic data within the scope of synthetic tabular healthcare data using deep generative methods. Based on this and the collective team experiences, we developed a conceptual framework for quality assurance. The applicability was benchmarked against a practical case from the Dutch National Cancer Registry. CONCLUSION We present a conceptual framework for quality assuranceof synthetic data for AI applications in healthcare that aligns diverging taxonomies, expands on common quality dimensions to include the dimensions of Fairness and Carbon footprint, and proposes stages necessary to support real-life applications. Building trust in synthetic data by increasing transparency and reducing the safety risk will accelerate the development and uptake of trustworthy AI tools for the benefit of patients. DISCUSSION Despite the growing emphasis on algorithmic fairness and carbon footprint, these metrics were scarce in the literature review. The overwhelming focus was on statistical similarity using distance metrics while sequential logic detection was scarce. A consensus-backed framework that includes all relevant quality dimensions can provide assurance for safe and responsible real-life applications of synthetic data. As the choice of appropriate metrics are highly context dependent, further research is needed on validation studies to guide metric choices and support the development of technical standards.
Collapse
Affiliation(s)
- Vibeke Binz Vallevik
- University of Oslo, Boks 1072 Blindern, NO-0316 Oslo, Norway; DNV AS, Veritasveien 1, 1322 Høvik, Norway.
| | | | | | - Severin Elvatun
- Cancer Registry of Norway, Ullernchausseen 64, 0379 Oslo, Norway
| | - Helga M B Brøgger
- DNV AS, Veritasveien 1, 1322 Høvik, Norway; Oslo University Hospital, Sognsvannsveien 20, 0372 Oslo, Norway
| | | | - Bjørn Edwin
- University of Oslo, Boks 1072 Blindern, NO-0316 Oslo, Norway; The Intervention Centre and Department of HPB Surgery, Oslo University Hospital and Institute of Clinical Medicine, Faculty of Medicine, University of Oslo, Oslo, Norway
| | | | | | - Jan F Nygård
- Cancer Registry of Norway, Ullernchausseen 64, 0379 Oslo, Norway; UiT - The Arctic University of Norway, Tromsø, Norway
| |
Collapse
|
5
|
Yan C, Zhang Z, Nyemba S, Li Z. Generating Synthetic Electronic Health Record Data Using Generative Adversarial Networks: Tutorial. JMIR AI 2024; 3:e52615. [PMID: 38875595 PMCID: PMC11074891 DOI: 10.2196/52615] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/10/2023] [Revised: 01/24/2024] [Accepted: 03/07/2024] [Indexed: 06/16/2024]
Abstract
Synthetic electronic health record (EHR) data generation has been increasingly recognized as an important solution to expand the accessibility and maximize the value of private health data on a large scale. Recent advances in machine learning have facilitated more accurate modeling for complex and high-dimensional data, thereby greatly enhancing the data quality of synthetic EHR data. Among various approaches, generative adversarial networks (GANs) have become the main technical path in the literature due to their ability to capture the statistical characteristics of real data. However, there is a scarcity of detailed guidance within the domain regarding the development procedures of synthetic EHR data. The objective of this tutorial is to present a transparent and reproducible process for generating structured synthetic EHR data using a publicly accessible EHR data set as an example. We cover the topics of GAN architecture, EHR data types and representation, data preprocessing, GAN training, synthetic data generation and postprocessing, and data quality evaluation. We conclude this tutorial by discussing multiple important issues and future opportunities in this domain. The source code of the entire process has been made publicly available.
Collapse
Affiliation(s)
- Chao Yan
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Ziqi Zhang
- Department of Computer Science, Vanderbilt University, Nashville, TN, United States
| | - Steve Nyemba
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Zhuohang Li
- Department of Computer Science, Vanderbilt University, Nashville, TN, United States
| |
Collapse
|
6
|
Pereira M, Kshirsagar M, Mukherjee S, Dodhia R, Lavista Ferres J, de Sousa R. Assessment of differentially private synthetic data for utility and fairness in end-to-end machine learning pipelines for tabular data. PLoS One 2024; 19:e0297271. [PMID: 38315667 PMCID: PMC10843030 DOI: 10.1371/journal.pone.0297271] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2023] [Accepted: 01/02/2024] [Indexed: 02/07/2024] Open
Abstract
Differentially private (DP) synthetic datasets are a solution for sharing data while preserving the privacy of individual data providers. Understanding the effects of utilizing DP synthetic data in end-to-end machine learning pipelines impacts areas such as health care and humanitarian action, where data is scarce and regulated by restrictive privacy laws. In this work, we investigate the extent to which synthetic data can replace real, tabular data in machine learning pipelines and identify the most effective synthetic data generation techniques for training and evaluating machine learning models. We systematically investigate the impacts of differentially private synthetic data on downstream classification tasks from the point of view of utility as well as fairness. Our analysis is comprehensive and includes representatives of the two main types of synthetic data generation algorithms: marginal-based and GAN-based. To the best of our knowledge, our work is the first that: (i) proposes a training and evaluation framework that does not assume that real data is available for testing the utility and fairness of machine learning models trained on synthetic data; (ii) presents the most extensive analysis of synthetic dataset generation algorithms in terms of utility and fairness when used for training machine learning models; and (iii) encompasses several different definitions of fairness. Our findings demonstrate that marginal-based synthetic data generators surpass GAN-based ones regarding model training utility for tabular data. Indeed, we show that models trained using data generated by marginal-based algorithms can exhibit similar utility to models trained using real data. Our analysis also reveals that the marginal-based synthetic data generated using AIM and MWEM PGM algorithms can train models that simultaneously achieve utility and fairness characteristics close to those obtained by models trained with real data.
Collapse
Affiliation(s)
- Mayana Pereira
- AI for Good Research Lab, Microsoft, Redmond, Washington, United States of America
- Department of Electrical Engineering, University of Brasilia, Brasilia, Brazil
| | - Meghana Kshirsagar
- AI for Good Research Lab, Microsoft, Redmond, Washington, United States of America
| | | | - Rahul Dodhia
- AI for Good Research Lab, Microsoft, Redmond, Washington, United States of America
| | - Juan Lavista Ferres
- AI for Good Research Lab, Microsoft, Redmond, Washington, United States of America
| | - Rafael de Sousa
- Department of Electrical Engineering, University of Brasilia, Brasilia, Brazil
| |
Collapse
|
7
|
Chang Q, Yan Z, Zhou M, Qu H, He X, Zhang H, Baskaran L, Al'Aref S, Li H, Zhang S, Metaxas DN. Mining multi-center heterogeneous medical data with distributed synthetic learning. Nat Commun 2023; 14:5510. [PMID: 37679325 PMCID: PMC10484909 DOI: 10.1038/s41467-023-40687-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Accepted: 08/03/2023] [Indexed: 09/09/2023] Open
Abstract
Overcoming barriers on the use of multi-center data for medical analytics is challenging due to privacy protection and data heterogeneity in the healthcare system. In this study, we propose the Distributed Synthetic Learning (DSL) architecture to learn across multiple medical centers and ensure the protection of sensitive personal information. DSL enables the building of a homogeneous dataset with entirely synthetic medical images via a form of GAN-based synthetic learning. The proposed DSL architecture has the following key functionalities: multi-modality learning, missing modality completion learning, and continual learning. We systematically evaluate the performance of DSL on different medical applications using cardiac computed tomography angiography (CTA), brain tumor MRI, and histopathology nuclei datasets. Extensive experiments demonstrate the superior performance of DSL as a high-quality synthetic medical image provider by the use of an ideal synthetic quality metric called Dist-FID. We show that DSL can be adapted to heterogeneous data and remarkably outperforms the real misaligned modalities segmentation model by 55% and the temporal datasets segmentation model by 8%.
Collapse
Affiliation(s)
- Qi Chang
- Department of Computer Science, Rutgers University, Piscataway, NJ, USA
| | | | - Mu Zhou
- SenseBrain Research, Princeton, NJ, USA
- Shanghai Artificial Intelligence Laboratory, Shanghai, China
| | - Hui Qu
- Department of Computer Science, Rutgers University, Piscataway, NJ, USA
| | - Xiaoxiao He
- Department of Computer Science, Rutgers University, Piscataway, NJ, USA
| | - Han Zhang
- Department of Computer Science, Rutgers University, Piscataway, NJ, USA
| | - Lohendran Baskaran
- Department of Cardiovascular Medicine, National Heart Centre Singapore, and Duke-National University Of Singapore, Singapore, Singapore
| | - Subhi Al'Aref
- Department of Medicine, Division of Cardiology, University of Arkansas for Medical Sciences, Little Rock, AR, USA
| | - Hongsheng Li
- Chinese University of Hong Kong, Hong Kong SAR, China.
- Centre for Perceptual and Interactive Intelligence (CPII), Hong Kong SAR, China.
| | - Shaoting Zhang
- Shanghai Artificial Intelligence Laboratory, Shanghai, China.
- Centre for Perceptual and Interactive Intelligence (CPII), Hong Kong SAR, China.
- SenseTime, Shanghai, China.
| | - Dimitris N Metaxas
- Department of Computer Science, Rutgers University, Piscataway, NJ, USA.
| |
Collapse
|
8
|
El Kababji S, Mitsakakis N, Fang X, Beltran-Bless AA, Pond G, Vandermeer L, Radhakrishnan D, Mosquera L, Paterson A, Shepherd L, Chen B, Barlow WE, Gralow J, Savard MF, Clemons M, El Emam K. Evaluating the Utility and Privacy of Synthetic Breast Cancer Clinical Trial Data Sets. JCO Clin Cancer Inform 2023; 7:e2300116. [PMID: 38011617 PMCID: PMC10703127 DOI: 10.1200/cci.23.00116] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Revised: 08/24/2023] [Accepted: 09/19/2023] [Indexed: 11/29/2023] Open
Abstract
PURPOSE There is strong interest from patients, researchers, the pharmaceutical industry, medical journal editors, funders of research, and regulators in sharing clinical trial data for secondary analysis. However, data access remains a challenge because of concerns about patient privacy. It has been argued that synthetic data generation (SDG) is an effective way to address these privacy concerns. There is a dearth of evidence supporting this on oncology clinical trial data sets, and on the utility of privacy-preserving synthetic data. The objective of the proposed study is to validate the utility and privacy risks of synthetic clinical trial data sets across multiple SDG techniques. METHODS We synthesized data sets from eight breast cancer clinical trial data sets using three types of generative models: sequential synthesis, conditional generative adversarial network, and variational autoencoder. Synthetic data utility was evaluated by replicating the published analyses on the synthetic data and assessing concordance of effect estimates and CIs between real and synthetic data. Privacy was evaluated by measuring attribution disclosure risk and membership disclosure risk. RESULTS Utility was highest using the sequential synthesis method where all results were replicable and the CI overlap most similar or higher for seven of eight data sets. Both types of privacy risks were low across all three types of generative models. DISCUSSION Synthetic data using sequential synthesis methods can act as a proxy for real clinical trial data sets, and simultaneously have low privacy risks. This type of generative model can be one way to enable broader sharing of clinical trial data.
Collapse
Affiliation(s)
| | | | - Xi Fang
- Replica Analytics Ltd, Ottawa, ON, Canada
| | - Ana-Alicia Beltran-Bless
- Ottawa Hospital Research Institute, Ottawa, ON, Canada
- Division of Medical Oncology, Department of Medicine, University of Ottawa, ON, Canada
| | - Greg Pond
- McMaster University, Hamilton, ON, Canada
| | | | - Dhenuka Radhakrishnan
- CHEO Research Institute, Ottawa, ON, Canada
- Department of Paediatrics, University of Ottawa, Ottawa, ON, Canada
| | - Lucy Mosquera
- CHEO Research Institute, Ottawa, ON, Canada
- Replica Analytics Ltd, Ottawa, ON, Canada
| | | | | | | | | | | | - Marie-France Savard
- Ottawa Hospital Research Institute, Ottawa, ON, Canada
- Division of Medical Oncology, Department of Medicine, University of Ottawa, ON, Canada
| | - Mark Clemons
- Ottawa Hospital Research Institute, Ottawa, ON, Canada
- Division of Medical Oncology, Department of Medicine, University of Ottawa, ON, Canada
| | - Khaled El Emam
- CHEO Research Institute, Ottawa, ON, Canada
- Replica Analytics Ltd, Ottawa, ON, Canada
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
9
|
Theodorou B, Xiao C, Sun J. Synthesize high-dimensional longitudinal electronic health records via hierarchical autoregressive language model. Nat Commun 2023; 14:5305. [PMID: 37652934 PMCID: PMC10471716 DOI: 10.1038/s41467-023-41093-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 08/23/2023] [Indexed: 09/02/2023] Open
Abstract
Synthetic electronic health records (EHRs) that are both realistic and privacy-preserving offer alternatives to real EHRs for machine learning (ML) and statistical analysis. However, generating high-fidelity EHR data in its original, high-dimensional form poses challenges for existing methods. We propose Hierarchical Autoregressive Language mOdel (HALO) for generating longitudinal, high-dimensional EHR, which preserve the statistical properties of real EHRs and can train accurate ML models without privacy concerns. HALO generates a probability density function over medical codes, clinical visits, and patient records, allowing for generating realistic EHR data without requiring variable selection or aggregation. Extensive experiments demonstrated that HALO can generate high-fidelity data with high-dimensional disease code probabilities closely mirroring (above 0.9 R2 correlation) real EHR data. HALO also enhances the accuracy of predictive modeling and enables downstream ML models to attain similar accuracy as models trained on genuine data.
Collapse
Affiliation(s)
- Brandon Theodorou
- University of Illinois at Urbana-Champaign, 201 North Goodwin Avenue, Urbana, IL, USA
- Medisyn Inc., Las Vegas, NV, USA
| | - Cao Xiao
- Medisyn Inc., Las Vegas, NV, USA
| | - Jimeng Sun
- University of Illinois at Urbana-Champaign, 201 North Goodwin Avenue, Urbana, IL, USA.
- Medisyn Inc., Las Vegas, NV, USA.
| |
Collapse
|
10
|
Kuo NIH, Garcia F, Sönnerborg A, Böhm M, Kaiser R, Zazzi M, Polizzotto M, Jorm L, Barbieri S. Generating synthetic clinical data that capture class imbalanced distributions with generative adversarial networks: Example using antiretroviral therapy for HIV. J Biomed Inform 2023; 144:104436. [PMID: 37451495 DOI: 10.1016/j.jbi.2023.104436] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 06/24/2023] [Accepted: 06/30/2023] [Indexed: 07/18/2023]
Abstract
OBJECTIVE Clinical data's confidential nature often limits the development of machine learning models in healthcare. Generative adversarial networks (GANs) can synthesise realistic datasets, but suffer from mode collapse, resulting in low diversity and bias towards majority demographics and common clinical practices. This work proposes an extension to the classic GAN framework that includes a variational autoencoder (VAE) and an external memory mechanism to overcome these limitations and generate synthetic data accurately describing imbalanced class distributions commonly found in clinical variables. METHODS The proposed method generated a synthetic dataset related to antiretroviral therapy for human immunodeficiency virus (ART for HIV). We evaluated it based on five metrics: (1) accurately representing imbalanced class distribution; (2) the realism of the individual variables; (3) the realism among variables; (4) patient disclosure risk; and (5) the utility of the generated dataset for developing downstream machine learning models. RESULTS The proposed method overcomes the issue of mode collapse and generates a synthetic dataset that accurately describes imbalanced class distributions commonly found in clinical variables. The generated data has a patient disclosure risk of 0.095%, lower than the 9% threshold stated by Health Canada and the European Medicines Agency, making it suitable for distribution to the research community with high security. The generated data also has high utility, indicating the potential of the proposed method to enable the development of downstream machine learning algorithms for healthcare applications using synthetic data. CONCLUSION Our proposed extension to the classic GAN framework, which includes a VAE and an external memory mechanism, represents a promising approach towards generating synthetic data that accurately describe imbalanced class distributions commonly found in clinical variables. This method overcomes the limitations of GANs and creates more realistic datasets with higher patient cohort diversity, facilitating the development of downstream machine learning algorithms for healthcare applications.
Collapse
Affiliation(s)
- Nicholas I-Hsien Kuo
- Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia.
| | - Federico Garcia
- Instituto de Investigación Ibs.Granada, Spain; Hospital Universitario San Cecilio, Spain; CIBER de Enfermedades Infecciosas, Spain
| | | | | | - Rolf Kaiser
- Uniklinik Köln, Universität zu Köln, Germany
| | | | | | - Louisa Jorm
- Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia
| | - Sebastiano Barbieri
- Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia
| |
Collapse
|
11
|
Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med 2023; 29:1930-1940. [PMID: 37460753 DOI: 10.1038/s41591-023-02448-8] [Citation(s) in RCA: 316] [Impact Index Per Article: 316.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2023] [Accepted: 06/08/2023] [Indexed: 08/17/2023]
Abstract
Large language models (LLMs) can respond to free-text queries without being specifically trained in the task in question, causing excitement and concern about their use in healthcare settings. ChatGPT is a generative artificial intelligence (AI) chatbot produced through sophisticated fine-tuning of an LLM, and other tools are emerging through similar developmental processes. Here we outline how LLM applications such as ChatGPT are developed, and we discuss how they are being leveraged in clinical settings. We consider the strengths and limitations of LLMs and their potential to improve the efficiency and effectiveness of clinical, educational and research work in medicine. LLM chatbots have already been deployed in a range of biomedical contexts, with impressive but mixed results. This review acts as a primer for interested clinicians, who will determine if and how LLM technology is used in healthcare for the benefit of patients and practitioners.
Collapse
Affiliation(s)
- Arun James Thirunavukarasu
- University of Cambridge School of Clinical Medicine, Cambridge, UK
- Corpus Christi College, University of Cambridge, Cambridge, UK
| | - Darren Shu Jeng Ting
- Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
- Birmingham and Midland Eye Centre, Birmingham, UK
- Academic Ophthalmology, School of Medicine, University of Nottingham, Nottingham, UK
| | - Kabilan Elangovan
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
| | - Laura Gutierrez
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
| | - Ting Fang Tan
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
- Department of Ophthalmology and Visual Sciences, Duke-National University of Singapore Medical School, Singapore, Singapore
| | - Daniel Shu Wei Ting
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore.
- Department of Ophthalmology and Visual Sciences, Duke-National University of Singapore Medical School, Singapore, Singapore.
- Byers Eye Institute, Stanford University, Palo Alto, CA, USA.
| |
Collapse
|
12
|
Li J, Cairns BJ, Li J, Zhu T. Generating synthetic mixed-type longitudinal electronic health records for artificial intelligent applications. NPJ Digit Med 2023; 6:98. [PMID: 37244963 DOI: 10.1038/s41746-023-00834-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2022] [Accepted: 05/05/2023] [Indexed: 05/29/2023] Open
Abstract
The recent availability of electronic health records (EHRs) have provided enormous opportunities to develop artificial intelligence (AI) algorithms. However, patient privacy has become a major concern that limits data sharing across hospital settings and subsequently hinders the advances in AI. Synthetic data, which benefits from the development and proliferation of generative models, has served as a promising substitute for real patient EHR data. However, the current generative models are limited as they only generate single type of clinical data for a synthetic patient, i.e., either continuous-valued or discrete-valued. To mimic the nature of clinical decision-making which encompasses various data types/sources, in this study, we propose a generative adversarial network (GAN) entitled EHR-M-GAN that simultaneously synthesizes mixed-type timeseries EHR data. EHR-M-GAN is capable of capturing the multidimensional, heterogeneous, and correlated temporal dynamics in patient trajectories. We have validated EHR-M-GAN on three publicly-available intensive care unit databases with records from a total of 141,488 unique patients, and performed privacy risk evaluation of the proposed model. EHR-M-GAN has demonstrated its superiority over state-of-the-art benchmarks for synthesizing clinical timeseries with high fidelity, while addressing the limitations regarding data types and dimensionality in the current generative models. Notably, prediction models for outcomes of intensive care performed significantly better when training data was augmented with the addition of EHR-M-GAN-generated timeseries. EHR-M-GAN may have use in developing AI algorithms in resource-limited settings, lowering the barrier for data acquisition while preserving patient privacy.
Collapse
Affiliation(s)
- Jin Li
- Engineering Research Center of EMR and Intelligent Expert System, Ministry of Education, College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, China
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - Benjamin J Cairns
- Clinical Trial Service Unit and Epidemiological Studies, Nuffield Department of Population Health, University of Oxford, Oxford, UK
| | - Jingsong Li
- Engineering Research Center of EMR and Intelligent Expert System, Ministry of Education, College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou, China.
- Research Center for Healthcare Data Science, Zhejiang Lab, Hangzhou, China.
| | - Tingting Zhu
- Department of Engineering Science, University of Oxford, Oxford, UK.
| |
Collapse
|
13
|
Theodorou B, Xiao C, Sun J. Synthesize Extremely High-dimensional Longitudinal Electronic Health Records via Hierarchical Autoregressive Language Model. RESEARCH SQUARE 2023:rs.3.rs-2644725. [PMID: 36945542 PMCID: PMC10029081 DOI: 10.21203/rs.3.rs-2644725/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2023]
Abstract
Synthetic electronic health records (EHRs) that are both realistic and preserve privacy can serve as an alternative to real EHRs for machine learning (ML) modeling and statistical analysis. However, generating high-fidelity and granular electronic health record (EHR) data in its original, highly-dimensional form poses challenges for existing methods due to the complexities inherent in high-dimensional data. In this paper, we propose Hierarchical Autoregressive Language mOdel (HALO) for generating longitudinal high-dimensional EHR, which preserve the statistical properties of real EHR and can be used to train accurate ML models without privacy concerns. Our HALO method, designed as a hierarchical autoregressive model, generates a probability density function of medical codes, clinical visits, and patient records, allowing for the generation of realistic EHR data in its original, unaggregated form without the need for variable selection or aggregation. Additionally, our model also produces high-quality continuous variables in a longitudinal and probabilistic manner. We conducted extensive experiments and demonstrate that HALO can generate high-fidelity EHR data with high-dimensional disease code probabilities ( d ≈ 10,000), disease code co-occurrence probabilities within a visit ( d ≈ 1,000,000), and conditional probabilities across consecutive visits ( d ≈ 5,000,000) and achieve above 0.9 R 2 correlation in comparison to real EHR data. In comparison to the leading baseline, HALO improves predictive modeling by over 17% in its predictive accuracy and perplexity on a hold-off test set of real EHR data. This performance then enables downstream ML models trained on its synthetic data to achieve comparable accuracy to models trained on real data (0.938 area under the ROC curve with HALO data vs. 0.943 with real data). Finally, using a combination of real and synthetic data enhances the accuracy of ML models beyond that achieved by using only real EHR data.
Collapse
|
14
|
Arora A, Arora A. Machine learning models trained on synthetic datasets of multiple sample sizes for the use of predicting blood pressure from clinical data in a national dataset. PLoS One 2023; 18:e0283094. [PMID: 36928534 PMCID: PMC10019654 DOI: 10.1371/journal.pone.0283094] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2023] [Accepted: 03/01/2023] [Indexed: 03/18/2023] Open
Abstract
INTRODUCTION The potential for synthetic data to act as a replacement for real data in research has attracted attention in recent months due to the prospect of increasing access to data and overcoming data privacy concerns when sharing data. The field of generative artificial intelligence and synthetic data is still early in its development, with a research gap evidencing that synthetic data can adequately be used to train algorithms that can be used on real data. This study compares the performance of a series machine learning models trained on real data and synthetic data, based on the National Diet and Nutrition Survey (NDNS). METHODS Features identified to be potentially of relevance by directed acyclic graphs were isolated from the NDNS dataset and used to construct synthetic datasets and impute missing data. Recursive feature elimination identified only four variables needed to predict mean arterial blood pressure: age, sex, weight and height. Bayesian generalised linear regression, random forest and neural network models were constructed based on these four variables to predict blood pressure. Models were trained on the real data training set (n = 2408), a synthetic data training set (n = 2408) and larger synthetic data training set (n = 4816) and a combination of the real and synthetic data training set (n = 4816). The same test set (n = 424) was used for each model. RESULTS Synthetic datasets demonstrated a high degree of fidelity with the real dataset. There was no significant difference between the performance of models trained on real, synthetic or combined datasets. Mean average error across all models and all training data ranged from 8.12 To 8.33. This indicates that synthetic data was capable of training equally accurate machine learning models as real data. DISCUSSION Further research is needed on a variety of datasets to confirm the utility of synthetic data to replace the use of potentially identifiable patient data. There is also further urgent research needed into evidencing that synthetic data can truly protect patient privacy against adversarial attempts to re-identify real individuals from the synthetic dataset.
Collapse
Affiliation(s)
- Anmol Arora
- School of Clinical Medicine, University of Cambridge, Cambridge, United Kingdom
- * E-mail:
| | - Ananya Arora
- School of Clinical Medicine, University of Cambridge, Cambridge, United Kingdom
| |
Collapse
|