1
|
Mishra R, Suri M. A survey and perspective on neuromorphic continual learning systems. Front Neurosci 2023; 17:1149410. [PMID: 37214407 PMCID: PMC10194827 DOI: 10.3389/fnins.2023.1149410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2023] [Accepted: 04/03/2023] [Indexed: 05/24/2023] Open
Abstract
With the advent of low-power neuromorphic computing systems, new possibilities have emerged for deployment in various sectors, like healthcare and transport, that require intelligent autonomous applications. These applications require reliable low-power solutions for sequentially adapting to new relevant data without loss of learning. Neuromorphic systems are inherently inspired by biological neural networks that have the potential to offer an efficient solution toward the feat of continual learning. With increasing attention in this area, we present a first comprehensive review of state-of-the-art neuromorphic continual learning (NCL) paradigms. The significance of our study is multi-fold. We summarize the recent progress and propose a plausible roadmap for developing end-to-end NCL systems. We also attempt to identify the gap between research and the real-world deployment of NCL systems in multiple applications. We do so by assessing the recent contributions in neuromorphic continual learning at multiple levels-applications, algorithms, architectures, and hardware. We discuss the relevance of NCL systems and draw out application-specific requisites. We analyze the biological underpinnings that are used for acquiring high-level performance. At the hardware level, we assess the ability of the current neuromorphic platforms and emerging nano-device-based architectures to support these algorithms in the presence of several constraints. Further, we propose refinements to continual learning metrics for applying them to NCL systems. Finally, the review identifies gaps and possible solutions that are not yet focused upon for deploying application-specific NCL systems in real-life scenarios.
Collapse
|
2
|
Lu Y, Li X, Yan B, Yan L, Zhang T, Song Z, Huang R, Yang Y. In-Memory Realization of Eligibility Traces Based on Conductance Drift of Phase Change Memory for Energy-Efficient Reinforcement Learning. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2022; 34:e2107811. [PMID: 34791712 DOI: 10.1002/adma.202107811] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/29/2021] [Revised: 11/02/2021] [Indexed: 06/13/2023]
Abstract
Reinforcement learning (RL) has shown outstanding performance in handling complex tasks in recent years. Eligibility trace (ET), a fundamental and important mechanism in reinforcement learning, records critical states with attenuation and guides the update of policy, which plays a crucial role in accelerating the convergence of RL training. However, ET implementation on conventional digital computing hardware is energy hungry and restricted by the memory wall due to massive calculation of exponential decay functions. Here, in-memory realization of ET for energy-efficient reinforcement learning with outstanding performance in discrete- and continuous-state RL tasks is demonstrated. For the first time, the inherent conductance drift of phase change memory is exploited as physical decay function to realize in-memory eligibility trace, demonstrating excellent performance during RL training in various tasks. The spontaneous in-memory decay computing and storage of policy in the same phase change memory give rise to significantly enhanced energy efficiency compared with traditional graphics processing unit platforms. This work therefore provides a holistic energy and hardware efficient method for both training and inference of reinforcement learning.
Collapse
Affiliation(s)
- Yingming Lu
- Key Laboratory of Microelectronic Devices and Circuits (MOE), School of Integrated Circuits, Peking University, Beijing, 100871, China
| | - Xi Li
- Shanghai Key Laboratory of Nanofabrication Technology for Memory, Shanghai Institute of Micro-system and Information Technology, Chinese Academy of Sciences, Shanghai, 200050, China
| | - Bonan Yan
- Center for Brain Inspired Chips, Institute for Artificial Intelligence, Peking University, Beijing, 100871, China
| | - Longhao Yan
- Key Laboratory of Microelectronic Devices and Circuits (MOE), School of Integrated Circuits, Peking University, Beijing, 100871, China
| | - Teng Zhang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), School of Integrated Circuits, Peking University, Beijing, 100871, China
| | - Zhitang Song
- Shanghai Key Laboratory of Nanofabrication Technology for Memory, Shanghai Institute of Micro-system and Information Technology, Chinese Academy of Sciences, Shanghai, 200050, China
| | - Ru Huang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), School of Integrated Circuits, Peking University, Beijing, 100871, China
- Center for Brain Inspired Chips, Institute for Artificial Intelligence, Peking University, Beijing, 100871, China
- Center for Brain Inspired Intelligence, Chinese Institute for Brain Research (CIBR), Beijing, Beijing, 102206, China
| | - Yuchao Yang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), School of Integrated Circuits, Peking University, Beijing, 100871, China
- Center for Brain Inspired Chips, Institute for Artificial Intelligence, Peking University, Beijing, 100871, China
- Center for Brain Inspired Intelligence, Chinese Institute for Brain Research (CIBR), Beijing, Beijing, 102206, China
| |
Collapse
|