1
|
Aguirre F, Sebastian A, Le Gallo M, Song W, Wang T, Yang JJ, Lu W, Chang MF, Ielmini D, Yang Y, Mehonic A, Kenyon A, Villena MA, Roldán JB, Wu Y, Hsu HH, Raghavan N, Suñé J, Miranda E, Eltawil A, Setti G, Smagulova K, Salama KN, Krestinskaya O, Yan X, Ang KW, Jain S, Li S, Alharbi O, Pazos S, Lanza M. Hardware implementation of memristor-based artificial neural networks. Nat Commun 2024; 15:1974. [PMID: 38438350 PMCID: PMC10912231 DOI: 10.1038/s41467-024-45670-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Accepted: 02/01/2024] [Indexed: 03/06/2024] Open
Abstract
Artificial Intelligence (AI) is currently experiencing a bloom driven by deep learning (DL) techniques, which rely on networks of connected simple computing units operating in parallel. The low communication bandwidth between memory and processing units in conventional von Neumann machines does not support the requirements of emerging applications that rely extensively on large sets of data. More recent computing paradigms, such as high parallelization and near-memory computing, help alleviate the data communication bottleneck to some extent, but paradigm- shifting concepts are required. Memristors, a novel beyond-complementary metal-oxide-semiconductor (CMOS) technology, are a promising choice for memory devices due to their unique intrinsic device-level properties, enabling both storing and computing with a small, massively-parallel footprint at low power. Theoretically, this directly translates to a major boost in energy efficiency and computational throughput, but various practical challenges remain. In this work we review the latest efforts for achieving hardware-based memristive artificial neural networks (ANNs), describing with detail the working principia of each block and the different design alternatives with their own advantages and disadvantages, as well as the tools required for accurate estimation of performance metrics. Ultimately, we aim to provide a comprehensive protocol of the materials and methods involved in memristive neural networks to those aiming to start working in this field and the experts looking for a holistic approach.
Collapse
Affiliation(s)
- Fernando Aguirre
- Physical Science and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
- Departament d'Enginyeria Electrònica, Universitat Autònoma de Barcelona (UAB), 08193, Barcelona, Spain
| | | | | | - Wenhao Song
- Department of Electrical and Computer Engineering, University of Southern California (USC), Los Angeles, CA, 90089, USA
| | - Tong Wang
- Department of Electrical and Computer Engineering, University of Southern California (USC), Los Angeles, CA, 90089, USA
| | - J Joshua Yang
- Department of Electrical and Computer Engineering, University of Southern California (USC), Los Angeles, CA, 90089, USA
| | - Wei Lu
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, 48109, USA
| | - Meng-Fan Chang
- Department of Electrical Engineering, National Tsing Hua University, Hsinchu, 30013, Taiwan
| | - Daniele Ielmini
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IUNET, Piazza L. da Vinci 32, 20133, Milano, Italy
| | - Yuchao Yang
- School of Electronic and Computer Engineering, Peking University, Shenzhen, China
| | - Adnan Mehonic
- Department of Electronic and Electrical Engineering, University College London (UCL), Torrington Place, WC1E 7JE, London, UK
| | - Anthony Kenyon
- Department of Electronic and Electrical Engineering, University College London (UCL), Torrington Place, WC1E 7JE, London, UK
| | - Marco A Villena
- Physical Science and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Juan B Roldán
- Departamento de Electrónica y Tecnología de Computadores, Facultad de Ciencias, Universidad de Granada, Avenida Fuentenueva s/n, 18071, Granada, Spain
| | - Yuting Wu
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, 48109, USA
| | - Hung-Hsi Hsu
- Department of Electrical Engineering, National Tsing Hua University, Hsinchu, 30013, Taiwan
| | - Nagarajan Raghavan
- Engineering Product Development (EPD) Pillar, Singapore University of Technology & Design, 8 Somapah Road, 487372, Singapore, Singapore
| | - Jordi Suñé
- Departament d'Enginyeria Electrònica, Universitat Autònoma de Barcelona (UAB), 08193, Barcelona, Spain
| | - Enrique Miranda
- Departament d'Enginyeria Electrònica, Universitat Autònoma de Barcelona (UAB), 08193, Barcelona, Spain
| | - Ahmed Eltawil
- Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Gianluca Setti
- Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Kamilya Smagulova
- Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Khaled N Salama
- Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Olga Krestinskaya
- Computer, Electrical and Mathematical Sciences and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Xiaobing Yan
- Key Laboratory of Brain-Like Neuromorphic Devices and Systems of Hebei Province, Hebei University, Baoding, 071002, China
| | - Kah-Wee Ang
- Department of Electrical and Computer Engineering, College of Design and Engineering, National University of Singapore (NUS), Singapore, Singapore
| | - Samarth Jain
- Department of Electrical and Computer Engineering, College of Design and Engineering, National University of Singapore (NUS), Singapore, Singapore
| | - Sifan Li
- Department of Electrical and Computer Engineering, College of Design and Engineering, National University of Singapore (NUS), Singapore, Singapore
| | - Osamah Alharbi
- Physical Science and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Sebastian Pazos
- Physical Science and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia
| | - Mario Lanza
- Physical Science and Engineering Division, King Abdullah University of Science and Technology (KAUST), Thuwal, 23955-6900, Saudi Arabia.
| |
Collapse
|
2
|
Jebali F, Majumdar A, Turck C, Harabi KE, Faye MC, Muhr E, Walder JP, Bilousov O, Michaud A, Vianello E, Hirtzlin T, Andrieu F, Bocquet M, Collin S, Querlioz D, Portal JM. Powering AI at the edge: A robust, memristor-based binarized neural network with near-memory computing and miniaturized solar cell. Nat Commun 2024; 15:741. [PMID: 38272896 PMCID: PMC10811339 DOI: 10.1038/s41467-024-44766-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Accepted: 01/04/2024] [Indexed: 01/27/2024] Open
Abstract
Memristor-based neural networks provide an exceptional energy-efficient platform for artificial intelligence (AI), presenting the possibility of self-powered operation when paired with energy harvesters. However, most memristor-based networks rely on analog in-memory computing, necessitating a stable and precise power supply, which is incompatible with the inherently unstable and unreliable energy harvesters. In this work, we fabricated a robust binarized neural network comprising 32,768 memristors, powered by a miniature wide-bandgap solar cell optimized for edge applications. Our circuit employs a resilient digital near-memory computing approach, featuring complementarily programmed memristors and logic-in-sense-amplifier. This design eliminates the need for compensation or calibration, operating effectively under diverse conditions. Under high illumination, the circuit achieves inference performance comparable to that of a lab bench power supply. In low illumination scenarios, it remains functional with slightly reduced accuracy, seamlessly transitioning to an approximate computing mode. Through image classification neural network simulations, we demonstrate that misclassified images under low illumination are primarily difficult-to-classify cases. Our approach lays the groundwork for self-powered AI and the creation of intelligent sensors for various applications in health, safety, and environment monitoring.
Collapse
Affiliation(s)
- Fadi Jebali
- Aix-Marseille Université, CNRS, Institut Matériaux Microélectronique Nanosciences de Provence, Marseille, France
| | - Atreya Majumdar
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
| | - Clément Turck
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
| | - Kamel-Eddine Harabi
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
| | - Mathieu-Coumba Faye
- Aix-Marseille Université, CNRS, Institut Matériaux Microélectronique Nanosciences de Provence, Marseille, France
- Université Grenoble Alpes, CEA, LETI, Grenoble, France
| | - Eloi Muhr
- Aix-Marseille Université, CNRS, Institut Matériaux Microélectronique Nanosciences de Provence, Marseille, France
| | - Jean-Pierre Walder
- Aix-Marseille Université, CNRS, Institut Matériaux Microélectronique Nanosciences de Provence, Marseille, France
| | | | - Amadéo Michaud
- Institut Photovoltaïque d'Ile-de-France (IPVF), Palaiseau, France
| | | | | | | | - Marc Bocquet
- Aix-Marseille Université, CNRS, Institut Matériaux Microélectronique Nanosciences de Provence, Marseille, France
| | - Stéphane Collin
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
- Institut Photovoltaïque d'Ile-de-France (IPVF), Palaiseau, France
| | - Damien Querlioz
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France.
| | - Jean-Michel Portal
- Aix-Marseille Université, CNRS, Institut Matériaux Microélectronique Nanosciences de Provence, Marseille, France.
| |
Collapse
|
3
|
Su Y, Seng KP, Ang LM, Smith J. Binary Neural Networks in FPGAs: Architectures, Tool Flows and Hardware Comparisons. SENSORS (BASEL, SWITZERLAND) 2023; 23:9254. [PMID: 38005640 PMCID: PMC10675041 DOI: 10.3390/s23229254] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/20/2023] [Revised: 10/14/2023] [Accepted: 11/13/2023] [Indexed: 11/26/2023]
Abstract
Binary neural networks (BNNs) are variations of artificial/deep neural network (ANN/DNN) architectures that constrain the real values of weights to the binary set of numbers {-1,1}. By using binary values, BNNs can convert matrix multiplications into bitwise operations, which accelerates both training and inference and reduces hardware complexity and model sizes for implementation. Compared to traditional deep learning architectures, BNNs are a good choice for implementation in resource-constrained devices like FPGAs and ASICs. However, BNNs have the disadvantage of reduced performance and accuracy because of the tradeoff due to binarization. Over the years, this has attracted the attention of the research community to overcome the performance gap of BNNs, and several architectures have been proposed. In this paper, we provide a comprehensive review of BNNs for implementation in FPGA hardware. The survey covers different aspects, such as BNN architectures and variants, design and tool flows for FPGAs, and various applications for BNNs. The final part of the paper gives some benchmark works and design tools for implementing BNNs in FPGAs based on established datasets used by the research community.
Collapse
Affiliation(s)
- Yuanxin Su
- School of AI and Advanced Computing, Xi’an Jiaotong Liverpool University, Suzhou 215000, China;
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool L69 3GJ, UK;
| | - Kah Phooi Seng
- School of AI and Advanced Computing, Xi’an Jiaotong Liverpool University, Suzhou 215000, China;
- School of Computer Science, Queensland University of Technology, Brisbane City, QLD 4000, Australia;
- School of Science Technology and Engineering, University of the Sunshine Coast, Sippy Downs, QLD 4556, Australia
| | - Li Minn Ang
- School of Computer Science, Queensland University of Technology, Brisbane City, QLD 4000, Australia;
| | - Jeremy Smith
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool L69 3GJ, UK;
| |
Collapse
|
4
|
Aguilera-Pedregosa C, Maldonado D, González MB, Moreno E, Jiménez-Molinos F, Campabadal F, Roldán JB. Thermal Characterization of Conductive Filaments in Unipolar Resistive Memories. MICROMACHINES 2023; 14:630. [PMID: 36985037 PMCID: PMC10057622 DOI: 10.3390/mi14030630] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/14/2023] [Revised: 02/07/2023] [Accepted: 03/06/2023] [Indexed: 06/18/2023]
Abstract
A methodology to estimate the device temperature in resistive random access memories (RRAMs) is presented. Unipolar devices, which are known to be highly influenced by thermal effects in their resistive switching operation, are employed to develop the technique. A 3D RRAM simulator is used to fit experimental data and obtain the maximum and average temperatures of the conductive filaments (CFs) that are responsible for the switching behavior. It is found that the experimental CFs temperature corresponds to the maximum simulated temperatures obtained at the narrowest sections of the CFs. These temperature values can be used to improve compact models for circuit simulation purposes.
Collapse
Affiliation(s)
- Cristina Aguilera-Pedregosa
- Departamento de Electrónica y Tecnología de Computadores, Facultad de Ciencias, Universidad de Granada, Avd. Fuentenueva s/n, 18071 Granada, Spain
| | - David Maldonado
- Departamento de Electrónica y Tecnología de Computadores, Facultad de Ciencias, Universidad de Granada, Avd. Fuentenueva s/n, 18071 Granada, Spain
| | - Mireia B. González
- Institut de Microelectrònica de Barcelona, IMB-CNM (CSIC), Carrer dels Til·lers s/n, Campus UAB, 08193 Bellaterra, Spain
| | - Enrique Moreno
- Departamento de Física y Matemáticas, Facultad de Ciencias, Universidad de Alcalá, Pl. de San Diego s/n, Alcalá de Henares, 28801 Madrid, Spain
| | - Francisco Jiménez-Molinos
- Departamento de Electrónica y Tecnología de Computadores, Facultad de Ciencias, Universidad de Granada, Avd. Fuentenueva s/n, 18071 Granada, Spain
| | - Francesca Campabadal
- Institut de Microelectrònica de Barcelona, IMB-CNM (CSIC), Carrer dels Til·lers s/n, Campus UAB, 08193 Bellaterra, Spain
| | - Juan B. Roldán
- Departamento de Electrónica y Tecnología de Computadores, Facultad de Ciencias, Universidad de Granada, Avd. Fuentenueva s/n, 18071 Granada, Spain
| |
Collapse
|
5
|
Buckley SM, Tait AN, McCaughan AN, Shastri BJ. Photonic online learning: a perspective. NANOPHOTONICS 2023; 12:833-845. [PMID: 36909290 PMCID: PMC9995662 DOI: 10.1515/nanoph-2022-0553] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/09/2022] [Revised: 10/31/2022] [Accepted: 12/03/2022] [Indexed: 06/18/2023]
Abstract
Emerging neuromorphic hardware promises to solve certain problems faster and with higher energy efficiency than traditional computing by using physical processes that take place at the device level as the computational primitives in neural networks. While initial results in photonic neuromorphic hardware are very promising, such hardware requires programming or "training" that is often power-hungry and time-consuming. In this article, we examine the online learning paradigm, where the machinery for training is built deeply into the hardware itself. We argue that some form of online learning will be necessary if photonic neuromorphic hardware is to achieve its true potential.
Collapse
Affiliation(s)
- Sonia Mary Buckley
- Applied Physics Division, National Institute of Standards and Technology, Boulder, CO80305, USA
| | - Alexander N. Tait
- Department of Physics, Engineering Physics and Astronomy, Queen’s University, Kingston, ON, Canada
| | - Adam N. McCaughan
- Applied Physics Division, National Institute of Standards and Technology, Boulder, CO80305, USA
| | - Bhavin J. Shastri
- Department of Physics, Engineering Physics and Astronomy, Queen’s University, Kingston, ON, Canada
| |
Collapse
|
6
|
Parmar V, Penkovsky B, Querlioz D, Suri M. Hardware-Efficient Stochastic Binary CNN Architectures for Near-Sensor Computing. Front Neurosci 2022; 15:781786. [PMID: 35069101 PMCID: PMC8766965 DOI: 10.3389/fnins.2021.781786] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2021] [Accepted: 11/29/2021] [Indexed: 11/16/2022] Open
Abstract
With recent advances in the field of artificial intelligence (AI) such as binarized neural networks (BNNs), a wide variety of vision applications with energy-optimized implementations have become possible at the edge. Such networks have the first layer implemented with high precision, which poses a challenge in deploying a uniform hardware mapping for the network implementation. Stochastic computing can allow conversion of such high-precision computations to a sequence of binarized operations while maintaining equivalent accuracy. In this work, we propose a fully binarized hardware-friendly computation engine based on stochastic computing as a proof of concept for vision applications involving multi-channel inputs. Stochastic sampling is performed by sampling from a non-uniform (normal) distribution based on analog hardware sources. We first validate the benefits of the proposed pipeline on the CIFAR-10 dataset. To further demonstrate its application for real-world scenarios, we present a case-study of microscopy image diagnostics for pathogen detection. We then evaluate benefits of implementing such a pipeline using OxRAM-based circuits for stochastic sampling as well as in-memory computing-based binarized multiplication. The proposed implementation is about 1,000 times more energy efficient compared to conventional floating-precision-based digital implementations, with memory savings of a factor of 45.
Collapse
Affiliation(s)
- Vivek Parmar
- Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India
| | - Bogdan Penkovsky
- Centre de Nanosciences et de Nanotechnologies, Université Paris-Saclay, CNRS, Palaiseau, France
| | - Damien Querlioz
- Centre de Nanosciences et de Nanotechnologies, Université Paris-Saclay, CNRS, Palaiseau, France
| | - Manan Suri
- Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India
| |
Collapse
|
7
|
Advances in Emerging Memory Technologies: From Data Storage to Artificial Intelligence. APPLIED SCIENCES-BASEL 2021. [DOI: 10.3390/app112311254] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
This paper presents an overview of emerging memory technologies. It begins with the presentation of stand-alone and embedded memory technology evolution, since the appearance of Flash memory in the 1980s. Then, the progress of emerging memory technologies (based on filamentary, phase change, magnetic, and ferroelectric mechanisms) is presented with a review of the major demonstrations in the literature. The potential of these technologies for storage applications addressing various markets and products is discussed. Finally, we discuss how the rise of artificial intelligence and bio-inspired circuits offers an opportunity for emerging memory technology and shifts the application from pure data storage to storage and computing tasks, and also enlarges the range of required specifications at the device level due to the exponential number of new systems and architectures.
Collapse
|
8
|
Covi E, Donati E, Liang X, Kappel D, Heidari H, Payvand M, Wang W. Adaptive Extreme Edge Computing for Wearable Devices. Front Neurosci 2021; 15:611300. [PMID: 34045939 PMCID: PMC8144334 DOI: 10.3389/fnins.2021.611300] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2020] [Accepted: 03/24/2021] [Indexed: 11/13/2022] Open
Abstract
Wearable devices are a fast-growing technology with impact on personal healthcare for both society and economy. Due to the widespread of sensors in pervasive and distributed networks, power consumption, processing speed, and system adaptation are vital in future smart wearable devices. The visioning and forecasting of how to bring computation to the edge in smart sensors have already begun, with an aspiration to provide adaptive extreme edge computing. Here, we provide a holistic view of hardware and theoretical solutions toward smart wearable devices that can provide guidance to research in this pervasive computing era. We propose various solutions for biologically plausible models for continual learning in neuromorphic computing technologies for wearable sensors. To envision this concept, we provide a systematic outline in which prospective low power and low latency scenarios of wearable sensors in neuromorphic platforms are expected. We successively describe vital potential landscapes of neuromorphic processors exploiting complementary metal-oxide semiconductors (CMOS) and emerging memory technologies (e.g., memristive devices). Furthermore, we evaluate the requirements for edge computing within wearable devices in terms of footprint, power consumption, latency, and data size. We additionally investigate the challenges beyond neuromorphic computing hardware, algorithms and devices that could impede enhancement of adaptive edge computing in smart wearable devices.
Collapse
Affiliation(s)
| | - Elisa Donati
- Institute of Neuroinformatics, University of Zurich, Eidgenössische Technische Hochschule Zürich (ETHZ), Zurich, Switzerland
| | - Xiangpeng Liang
- Microelectronics Lab, James Watt School of Engineering, University of Glasgow, Glasgow, United Kingdom
| | - David Kappel
- Bernstein Center for Computational Neuroscience, III Physikalisches Institut–Biophysik, Georg-August Universität, Göttingen, Germany
| | - Hadi Heidari
- Microelectronics Lab, James Watt School of Engineering, University of Glasgow, Glasgow, United Kingdom
| | - Melika Payvand
- Institute of Neuroinformatics, University of Zurich, Eidgenössische Technische Hochschule Zürich (ETHZ), Zurich, Switzerland
| | - Wei Wang
- The Andrew and Erna Viterbi Department of Electrical Engineering, Technion–Israel Institute of Technology, Haifa, Israel
| |
Collapse
|
9
|
Laborieux A, Ernoult M, Hirtzlin T, Querlioz D. Synaptic metaplasticity in binarized neural networks. Nat Commun 2021; 12:2549. [PMID: 33953183 PMCID: PMC8100137 DOI: 10.1038/s41467-021-22768-y] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2020] [Accepted: 03/25/2021] [Indexed: 11/09/2022] Open
Abstract
While deep neural networks have surpassed human performance in multiple situations, they are prone to catastrophic forgetting: upon training a new task, they rapidly forget previously learned ones. Neuroscience studies, based on idealized tasks, suggest that in the brain, synapses overcome this issue by adjusting their plasticity depending on their past history. However, such "metaplastic" behaviors do not transfer directly to mitigate catastrophic forgetting in deep neural networks. In this work, we interpret the hidden weights used by binarized neural networks, a low-precision version of deep neural networks, as metaplastic variables, and modify their training technique to alleviate forgetting. Building on this idea, we propose and demonstrate experimentally, in situations of multitask and stream learning, a training technique that reduces catastrophic forgetting without needing previously presented data, nor formal boundaries between datasets and with performance approaching more mainstream techniques with task boundaries. We support our approach with a theoretical analysis on a tractable task. This work bridges computational neuroscience and deep learning, and presents significant assets for future embedded and neuromorphic systems, especially when using novel nanodevices featuring physics analogous to metaplasticity.
Collapse
Affiliation(s)
- Axel Laborieux
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France.
| | - Maxence Ernoult
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
- Unité Mixte de Physique, CNRS, Thales, Université Paris-Saclay, Palaiseau, France
| | - Tifenn Hirtzlin
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
| | - Damien Querlioz
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France.
| |
Collapse
|
10
|
Laborieux A, Ernoult M, Scellier B, Bengio Y, Grollier J, Querlioz D. Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing Its Gradient Estimator Bias. Front Neurosci 2021; 15:633674. [PMID: 33679315 PMCID: PMC7930909 DOI: 10.3389/fnins.2021.633674] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Accepted: 01/26/2021] [Indexed: 11/24/2022] Open
Abstract
Equilibrium Propagation is a biologically-inspired algorithm that trains convergent recurrent neural networks with a local learning rule. This approach constitutes a major lead to allow learning-capable neuromophic systems and comes with strong theoretical guarantees. Equilibrium propagation operates in two phases, during which the network is let to evolve freely and then "nudged" toward a target; the weights of the network are then updated based solely on the states of the neurons that they connect. The weight updates of Equilibrium Propagation have been shown mathematically to approach those provided by Backpropagation Through Time (BPTT), the mainstream approach to train recurrent neural networks, when nudging is performed with infinitely small strength. In practice, however, the standard implementation of Equilibrium Propagation does not scale to visual tasks harder than MNIST. In this work, we show that a bias in the gradient estimate of equilibrium propagation, inherent in the use of finite nudging, is responsible for this phenomenon and that canceling it allows training deep convolutional neural networks. We show that this bias can be greatly reduced by using symmetric nudging (a positive nudging and a negative one). We also generalize Equilibrium Propagation to the case of cross-entropy loss (by opposition to squared error). As a result of these advances, we are able to achieve a test error of 11.7% on CIFAR-10, which approaches the one achieved by BPTT and provides a major improvement with respect to the standard Equilibrium Propagation that gives 86% test error. We also apply these techniques to train an architecture with unidirectional forward and backward connections, yielding a 13.2% test error. These results highlight equilibrium propagation as a compelling biologically-plausible approach to compute error gradients in deep neuromorphic systems.
Collapse
Affiliation(s)
- Axel Laborieux
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
| | - Maxence Ernoult
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
- Unité Mixte de Physique, CNRS, Thales, Université Paris-Saclay, Palaiseau, France
- Mila, Université de Montréal, Montreal, QC, Canada
| | | | - Yoshua Bengio
- Mila, Université de Montréal, Montreal, QC, Canada
- Canadian Institute for Advanced Research, Toronto, ON, Canada
| | - Julie Grollier
- Unité Mixte de Physique, CNRS, Thales, Université Paris-Saclay, Palaiseau, France
| | - Damien Querlioz
- Université Paris-Saclay, CNRS, Centre de Nanosciences et de Nanotechnologies, Palaiseau, France
| |
Collapse
|
11
|
Azghadi MR, Lammie C, Eshraghian JK, Payvand M, Donati E, Linares-Barranco B, Indiveri G. Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications. IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS 2020; 14:1138-1159. [PMID: 33156792 DOI: 10.1109/tbcas.2020.3036081] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
The advent of dedicated Deep Learning (DL) accelerators and neuromorphic processors has brought on new opportunities for applying both Deep and Spiking Neural Network (SNN) algorithms to healthcare and biomedical applications at the edge. This can facilitate the advancement of medical Internet of Things (IoT) systems and Point of Care (PoC) devices. In this paper, we provide a tutorial describing how various technologies including emerging memristive devices, Field Programmable Gate Arrays (FPGAs), and Complementary Metal Oxide Semiconductor (CMOS) can be used to develop efficient DL accelerators to solve a wide variety of diagnostic, pattern recognition, and signal processing problems in healthcare. Furthermore, we explore how spiking neuromorphic processors can complement their DL counterparts for processing biomedical signals. The tutorial is augmented with case studies of the vast literature on neural network and neuromorphic hardware as applied to the healthcare domain. We benchmark various hardware platforms by performing a sensor fusion signal processing task combining electromyography (EMG) signals with computer vision. Comparisons are made between dedicated neuromorphic processors and embedded AI accelerators in terms of inference latency and energy. Finally, we provide our analysis of the field and share a perspective on the advantages, disadvantages, challenges, and opportunities that various accelerators and neuromorphic processors introduce to healthcare and biomedical domains.
Collapse
|
12
|
Wang W, Song W, Yao P, Li Y, Van Nostrand J, Qiu Q, Ielmini D, Yang JJ. Integration and Co-design of Memristive Devices and Algorithms for Artificial Intelligence. iScience 2020; 23:101809. [PMID: 33305176 PMCID: PMC7718163 DOI: 10.1016/j.isci.2020.101809] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022] Open
Abstract
Memristive devices share remarkable similarities to biological synapses, dendrites, and neurons at both the physical mechanism level and unit functionality level, making the memristive approach to neuromorphic computing a promising technology for future artificial intelligence. However, these similarities do not directly transfer to the success of efficient computation without device and algorithm co-designs and optimizations. Contemporary deep learning algorithms demand the memristive artificial synapses to ideally possess analog weighting and linear weight-update behavior, requiring substantial device-level and circuit-level optimization. Such co-design and optimization have been the main focus of memristive neuromorphic engineering, which often abandons the “non-ideal” behaviors of memristive devices, although many of them resemble what have been observed in biological components. Novel brain-inspired algorithms are being proposed to utilize such behaviors as unique features to further enhance the efficiency and intelligence of neuromorphic computing, which calls for collaborations among electrical engineers, computing scientists, and neuroscientists.
Collapse
Affiliation(s)
- Wei Wang
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, Milano 20133, Italy
| | - Wenhao Song
- Electrical and Computer Engineering Department, University of Southern California, Los Angeles, CA, USA
| | - Peng Yao
- Electrical and Computer Engineering Department, University of Southern California, Los Angeles, CA, USA
| | - Yang Li
- The Andrew and Erna Viterbi Department of Electrical Engineering, Technion-Israel Institute of Technology, Haifa 32000, Israel
| | | | - Qinru Qiu
- Electrical Engineering and Computer Science Department, Syracuse University, NY, USA
| | - Daniele Ielmini
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, Milano 20133, Italy
| | - J Joshua Yang
- Electrical and Computer Engineering Department, University of Southern California, Los Angeles, CA, USA
| |
Collapse
|
13
|
Nandakumar SR, Le Gallo M, Piveteau C, Joshi V, Mariani G, Boybat I, Karunaratne G, Khaddam-Aljameh R, Egger U, Petropoulos A, Antonakopoulos T, Rajendran B, Sebastian A, Eleftheriou E. Mixed-Precision Deep Learning Based on Computational Memory. Front Neurosci 2020; 14:406. [PMID: 32477047 PMCID: PMC7235420 DOI: 10.3389/fnins.2020.00406] [Citation(s) in RCA: 31] [Impact Index Per Article: 7.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2019] [Accepted: 04/03/2020] [Indexed: 11/29/2022] Open
Abstract
Deep neural networks (DNNs) have revolutionized the field of artificial intelligence and have achieved unprecedented success in cognitive tasks such as image and speech recognition. Training of large DNNs, however, is computationally intensive and this has motivated the search for novel computing architectures targeting this application. A computational memory unit with nanoscale resistive memory devices organized in crossbar arrays could store the synaptic weights in their conductance states and perform the expensive weighted summations in place in a non-von Neumann manner. However, updating the conductance states in a reliable manner during the weight update process is a fundamental challenge that limits the training accuracy of such an implementation. Here, we propose a mixed-precision architecture that combines a computational memory unit performing the weighted summations and imprecise conductance updates with a digital processing unit that accumulates the weight updates in high precision. A combined hardware/software training experiment of a multilayer perceptron based on the proposed architecture using a phase-change memory (PCM) array achieves 97.73% test accuracy on the task of classifying handwritten digits (based on the MNIST dataset), within 0.6% of the software baseline. The architecture is further evaluated using accurate behavioral models of PCM on a wide class of networks, namely convolutional neural networks, long-short-term-memory networks, and generative-adversarial networks. Accuracies comparable to those of floating-point implementations are achieved without being constrained by the non-idealities associated with the PCM devices. A system-level study demonstrates 172 × improvement in energy efficiency of the architecture when used for training a multilayer perceptron compared with a dedicated fully digital 32-bit implementation.
Collapse
Affiliation(s)
| | | | - Christophe Piveteau
- IBM Research - Zurich, Rüschlikon, Switzerland
- Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland
| | - Vinay Joshi
- IBM Research - Zurich, Rüschlikon, Switzerland
- Engineering Department, King's College London, London, United Kingdom
| | | | - Irem Boybat
- IBM Research - Zurich, Rüschlikon, Switzerland
- Ecole Polytechnique Federale de Lausanne (EPFL), Institute of Electrical Engineering, Lausanne, Switzerland
| | - Geethan Karunaratne
- IBM Research - Zurich, Rüschlikon, Switzerland
- Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland
| | - Riduan Khaddam-Aljameh
- IBM Research - Zurich, Rüschlikon, Switzerland
- Department of Information Technology and Electrical Engineering, ETH Zurich, Zurich, Switzerland
| | - Urs Egger
- IBM Research - Zurich, Rüschlikon, Switzerland
| | - Anastasios Petropoulos
- IBM Research - Zurich, Rüschlikon, Switzerland
- Department of Electrical and Computers Engineering, University of Patras, Rio Achaia, Greece
| | - Theodore Antonakopoulos
- Department of Electrical and Computers Engineering, University of Patras, Rio Achaia, Greece
| | - Bipin Rajendran
- Engineering Department, King's College London, London, United Kingdom
| | | | | |
Collapse
|