1
|
Sattler F, Korjakow T, Rischke R, Samek W. FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:5531-5543. [PMID: 34851838 DOI: 10.1109/tnnls.2021.3129371] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Federated distillation (FD) is a popular novel algorithmic paradigm for Federated learning (FL), which achieves training performance competitive to prior parameter averaging-based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work, we propose FedAUX, an extension to FD, which, under the same set of assumptions, drastically improves the performance by deriving maximum utility from the unlabeled auxiliary data. FedAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a suitable model initialization for the distributed training. Second, (ε, δ) -differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks (CNNs) and transformer models demonstrate that our proposed method achieves remarkable performance improvements over state-of-the-art FL methods, without adding appreciable computation, communication, or privacy cost. For instance, when training ResNet8 on non-independent identically distributed (i.i.d.) subsets of CIFAR10, FedAUX raises the maximum achieved validation accuracy from 30.4% to 78.1%, further closing the gap to centralized training performance. Code is available at https://github.com/fedl-repo/fedaux.
Collapse
|
2
|
Oala L, Murchison AG, Balachandran P, Choudhary S, Fehr J, Leite AW, Goldschmidt PG, Johner C, Schörverth EDM, Nakasi R, Meyer M, Cabitza F, Baird P, Prabhu C, Weicken E, Liu X, Wenzel M, Vogler S, Akogo D, Alsalamah S, Kazim E, Koshiyama A, Piechottka S, Macpherson S, Shadforth I, Geierhofer R, Matek C, Krois J, Sanguinetti B, Arentz M, Bielik P, Calderon-Ramirez S, Abbood A, Langer N, Haufe S, Kherif F, Pujari S, Samek W, Wiegand T. Machine Learning for Health: Algorithm Auditing & Quality Control. J Med Syst 2021; 45:105. [PMID: 34729675 PMCID: PMC8562935 DOI: 10.1007/s10916-021-01783-y] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Accepted: 10/11/2021] [Indexed: 01/26/2023]
Abstract
Developers proposing new machine learning for health (ML4H) tools often pledge to match or even surpass the performance of existing tools, yet the reality is usually more complicated. Reliable deployment of ML4H to the real world is challenging as examples from diabetic retinopathy or Covid-19 screening show. We envision an integrated framework of algorithm auditing and quality control that provides a path towards the effective and reliable application of ML systems in healthcare. In this editorial, we give a summary of ongoing work towards that vision and announce a call for participation to the special issue Machine Learning for Health: Algorithm Auditing & Quality Control in this journal to advance the practice of ML4H auditing.
Collapse
Affiliation(s)
| | | | | | | | - Jana Fehr
- Hasso-Plattner-Institute of Digital Engineering, Potsdam, Germany
| | - Alixandro Werneck Leite
- Machine Learning Laboratory in Finance and Organizations, Universidade de Brasília, Brasília, Brazil
| | | | | | | | | | | | | | | | | | | | - Xiaoxuan Liu
- University Hospitals Birmingham NHS Foundation Trust & Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom
| | | | | | | | - Shada Alsalamah
- Information Systems Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia
- Digital Health and Innovation Department, Science Division, World Health Organization, Winterthur, Switzerland
| | - Emre Kazim
- University College London, London, United Kingdom
| | | | | | | | | | | | | | - Joachim Krois
- Oral Diagnostics Digital Health Health Services Research, Charité-Universitätsmedizin, Berlin, Germany
| | | | - Matthew Arentz
- Department of Global Health, University of Washington, Washington, USA
| | | | | | | | - Nicolas Langer
- Department of Psychology, University of Zurich, Zürich, Switzerland
| | | | - Ferath Kherif
- Laboratory for Research in Neuroimaging, Department of Clinical Neuroscience, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
| | - Sameer Pujari
- Digital Health and Innovation Department, Science Division, World Health Organization, Winterthur, Switzerland
| | | | | |
Collapse
|
3
|
Calderon-Ramirez S, Yang S, Moemeni A, Colreavy-Donnelly S, Elizondo DA, Oala L, Rodriguez-Capitan J, Jimenez-Navarro M, Lopez-Rubio E, Molina-Cabello MA. Improving Uncertainty Estimation With Semi-Supervised Deep Learning for COVID-19 Detection Using Chest X-Ray Images. IEEE ACCESS : PRACTICAL INNOVATIONS, OPEN SOLUTIONS 2021; 9:85442-85454. [PMID: 34812397 PMCID: PMC8545186 DOI: 10.1109/access.2021.3085418] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/14/2021] [Accepted: 05/24/2021] [Indexed: 05/02/2023]
Abstract
In this work we implement a COVID-19 infection detection system based on chest X-ray images with uncertainty estimation. Uncertainty estimation is vital for safe usage of computer aided diagnosis tools in medical applications. Model estimations with high uncertainty should be carefully analyzed by a trained radiologist. We aim to improve uncertainty estimations using unlabelled data through the MixMatch semi-supervised framework. We test popular uncertainty estimation approaches, comprising Softmax scores, Monte-Carlo dropout and deterministic uncertainty quantification. To compare the reliability of the uncertainty estimates, we propose the usage of the Jensen-Shannon distance between the uncertainty distributions of correct and incorrect estimations. This metric is statistically relevant, unlike most previously used metrics, which often ignore the distribution of the uncertainty estimations. Our test results show a significant improvement in uncertainty estimates when using unlabelled data. The best results are obtained with the use of the Monte Carlo dropout method.
Collapse
Affiliation(s)
- Saul Calderon-Ramirez
- School of Computer Science and InformaticsDe Montfort University Leicester LE1 9BH U.K
- Instituto Tecnologico de Costa Rica Cartago 30101 Costa Rica
| | - Shengxiang Yang
- School of Computer Science and InformaticsDe Montfort University Leicester LE1 9BH U.K
| | - Armaghan Moemeni
- School of Computer ScienceUniversity of Nottingham Nottingham NG8 1BB U.K
| | | | - David A Elizondo
- School of Computer Science and InformaticsDe Montfort University Leicester LE1 9BH U.K
| | - Luis Oala
- XAI GroupArtificial Intelligence DepartmentFraunhofer Heinrich Hertz Institute 10587 Berlin Germany
| | - Jorge Rodriguez-Capitan
- CIBERCVHospital Universitario Virgen de la Victoria 29010 Málaga Spain
- Instituto de Investigación Biomédica de Mñlaga (IBIMA) 29010 Málaga Spain
| | - Manuel Jimenez-Navarro
- CIBERCVHospital Universitario Virgen de la Victoria 29010 Málaga Spain
- Instituto de Investigación Biomédica de Mñlaga (IBIMA) 29010 Málaga Spain
| | - Ezequiel Lopez-Rubio
- Department of Computer Languages and Computer ScienceUniversity of Málaga 29071 Málaga Spain
- Instituto de Investigación Biomédica de Mñlaga (IBIMA) 29010 Málaga Spain
| | - Miguel A Molina-Cabello
- Department of Computer Languages and Computer ScienceUniversity of Málaga 29071 Málaga Spain
- Instituto de Investigación Biomédica de Mñlaga (IBIMA) 29010 Málaga Spain
| |
Collapse
|