1
|
Bonnet D, Hirtzlin T, Majumdar A, Dalgaty T, Esmanhotto E, Meli V, Castellani N, Martin S, Nodin JF, Bourgeois G, Portal JM, Querlioz D, Vianello E. Bringing uncertainty quantification to the extreme-edge with memristor-based Bayesian neural networks. Nat Commun 2023; 14:7530. [PMID: 37985669 PMCID: PMC10661910 DOI: 10.1038/s41467-023-43317-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2023] [Accepted: 11/07/2023] [Indexed: 11/22/2023] Open
Abstract
Safety-critical sensory applications, like medical diagnosis, demand accurate decisions from limited, noisy data. Bayesian neural networks excel at such tasks, offering predictive uncertainty assessment. However, because of their probabilistic nature, they are computationally intensive. An innovative solution utilizes memristors' inherent probabilistic nature to implement Bayesian neural networks. However, when using memristors, statistical effects follow the laws of device physics, whereas in Bayesian neural networks, those effects can take arbitrary shapes. This work overcome this difficulty by adopting a variational inference training augmented by a "technological loss", incorporating memristor physics. This technique enabled programming a Bayesian neural network on 75 crossbar arrays of 1,024 memristors, incorporating CMOS periphery for in-memory computing. The experimental neural network classified heartbeats with high accuracy, and estimated the certainty of its predictions. The results reveal orders-of-magnitude improvement in inference energy efficiency compared to a microcontroller or an embedded graphics processing unit performing the same task.
Collapse
Affiliation(s)
- Djohan Bonnet
- Université Grenoble Alpes, CEA, LETI, Grenoble, France.
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France.
| | | | - Atreya Majumdar
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
| | | | | | | | | | - Simon Martin
- Université Grenoble Alpes, CEA, LETI, Grenoble, France
| | | | | | - Jean-Michel Portal
- Aix-Marseille Université, CNRS, Institut Matériaux Microélectronique Nanosciences de Provence, Marseille, France
| | - Damien Querlioz
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France.
| | | |
Collapse
|
2
|
Bianchi S, Muñoz-Martin I, Covi E, Bricalli A, Piccolboni G, Regev A, Molas G, Nodin JF, Andrieu F, Ielmini D. A self-adaptive hardware with resistive switching synapses for experience-based neurocomputing. Nat Commun 2023; 14:1565. [PMID: 36944647 PMCID: PMC10030830 DOI: 10.1038/s41467-023-37097-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2021] [Accepted: 03/02/2023] [Indexed: 03/23/2023] Open
Abstract
Neurobiological systems continually interact with the surrounding environment to refine their behaviour toward the best possible reward. Achieving such learning by experience is one of the main challenges of artificial intelligence, but currently it is hindered by the lack of hardware capable of plastic adaptation. Here, we propose a bio-inspired recurrent neural network, mastered by a digital system on chip with resistive-switching synaptic arrays of memory devices, which exploits homeostatic Hebbian learning for improved efficiency. All the results are discussed experimentally and theoretically, proposing a conceptual framework for benchmarking the main outcomes in terms of accuracy and resilience. To test the proposed architecture for reinforcement learning tasks, we study the autonomous exploration of continually evolving environments and verify the results for the Mars rover navigation. We also show that, compared to conventional deep learning techniques, our in-memory hardware has the potential to achieve a significant boost in speed and power-saving.
Collapse
Affiliation(s)
- S Bianchi
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IUNET, Milano, 20133, Italy
- Infineon Technologies, Villach, Austria
| | - I Muñoz-Martin
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IUNET, Milano, 20133, Italy
- Infineon Technologies, Villach, Austria
| | - E Covi
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IUNET, Milano, 20133, Italy
- NaMLab gGmbH, Dresden, Germany
| | | | | | - A Regev
- Weebit Nano, Hod Hasharon, Israel
| | - G Molas
- Weebit Nano, Hod Hasharon, Israel
| | - J F Nodin
- Univ. Grenoble Alpes, CEA, Leti, F-38000, Grenoble, France
| | - F Andrieu
- Univ. Grenoble Alpes, CEA, Leti, F-38000, Grenoble, France
| | - D Ielmini
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IUNET, Milano, 20133, Italy.
| |
Collapse
|
3
|
Hennen T, Elias A, Nodin JF, Molas G, Waser R, Wouters DJ, Bedau D. A high throughput generative vector autoregression model for stochastic synapses. Front Neurosci 2022; 16:941753. [PMID: 36061591 PMCID: PMC9433991 DOI: 10.3389/fnins.2022.941753] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2022] [Accepted: 08/04/2022] [Indexed: 11/29/2022] Open
Abstract
By imitating the synaptic connectivity and plasticity of the brain, emerging electronic nanodevices offer new opportunities as the building blocks of neuromorphic systems. One challenge for large-scale simulations of computational architectures based on emerging devices is to accurately capture device response, hysteresis, noise, and the covariance structure in the temporal domain as well as between the different device parameters. We address this challenge with a high throughput generative model for synaptic arrays that is based on a recently available type of electrical measurement data for resistive memory cells. We map this real-world data onto a vector autoregressive stochastic process to accurately reproduce the device parameters and their cross-correlation structure. While closely matching the measured data, our model is still very fast; we provide parallelized implementations for both CPUs and GPUs and demonstrate array sizes above one billion cells and throughputs exceeding one hundred million weight updates per second, above the pixel rate of a 30 frames/s 4K video stream.
Collapse
Affiliation(s)
- Tyler Hennen
- Institut für Werkstoffe der Elektrotechnik 2 (IWE II), RWTH Aachen University, Aachen, Germany
| | - Alexander Elias
- Western Digital San Jose Research Center, San Jose, CA, United States
| | | | - Gabriel Molas
- CEA, LETI, Grenoble, France
- Weebit Nano Ltd., Grenoble, France
| | - Rainer Waser
- Institut für Werkstoffe der Elektrotechnik 2 (IWE II), RWTH Aachen University, Aachen, Germany
| | - Dirk J. Wouters
- Institut für Werkstoffe der Elektrotechnik 2 (IWE II), RWTH Aachen University, Aachen, Germany
| | - Daniel Bedau
- Western Digital San Jose Research Center, San Jose, CA, United States
| |
Collapse
|