1
|
Goupy G, Tirilly P, Bilasco IM. Paired competing neurons improving STDP supervised local learning in Spiking Neural Networks. Front Neurosci 2024; 18:1401690. [PMID: 39119458 PMCID: PMC11307446 DOI: 10.3389/fnins.2024.1401690] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Accepted: 07/11/2024] [Indexed: 08/10/2024] Open
Abstract
Direct training of Spiking Neural Networks (SNNs) on neuromorphic hardware has the potential to significantly reduce the energy consumption of artificial neural network training. SNNs trained with Spike Timing-Dependent Plasticity (STDP) benefit from gradient-free and unsupervised local learning, which can be easily implemented on ultra-low-power neuromorphic hardware. However, classification tasks cannot be performed solely with unsupervised STDP. In this paper, we propose Stabilized Supervised STDP (S2-STDP), a supervised STDP learning rule to train the classification layer of an SNN equipped with unsupervised STDP for feature extraction. S2-STDP integrates error-modulated weight updates that align neuron spikes with desired timestamps derived from the average firing time within the layer. Then, we introduce a training architecture called Paired Competing Neurons (PCN) to further enhance the learning capabilities of our classification layer trained with S2-STDP. PCN associates each class with paired neurons and encourages neuron specialization toward target or non-target samples through intra-class competition. We evaluate our methods on image recognition datasets, including MNIST, Fashion-MNIST, and CIFAR-10. Results show that our methods outperform state-of-the-art supervised STDP learning rules, for comparable architectures and numbers of neurons. Further analysis demonstrates that the use of PCN enhances the performance of S2-STDP, regardless of the hyperparameter set and without introducing any additional hyperparameters.
Collapse
|
2
|
Pei Y, Xu C, Wu Z, Liu Y, Yang Y. ALBSNN: ultra-low latency adaptive local binary spiking neural network with accuracy loss estimator. Front Neurosci 2023; 17:1225871. [PMID: 37771337 PMCID: PMC10525310 DOI: 10.3389/fnins.2023.1225871] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2023] [Accepted: 08/24/2023] [Indexed: 09/30/2023] Open
Abstract
Spiking neural network (SNN) is a brain-inspired model with more spatio-temporal information processing capacity and computational energy efficiency. However, with the increasing depth of SNNs, the memory problem caused by the weights of SNNs has gradually attracted attention. In this study, we propose an ultra-low latency adaptive local binary spiking neural network (ALBSNN) with accuracy loss estimators, which dynamically selects the network layers to be binarized to ensure a balance between quantization degree and classification accuracy by evaluating the error caused by the binarized weights during the network learning process. At the same time, to accelerate the training speed of the network, the global average pooling (GAP) layer is introduced to replace the fully connected layers by combining convolution and pooling. Finally, to further reduce the error caused by the binary weight, we propose binary weight optimization (BWO), which updates the overall weight by directly adjusting the binary weight. This method further reduces the loss of the network that reaches the training bottleneck. The combination of the above methods balances the network's quantization and recognition ability, enabling the network to maintain the recognition capability equivalent to the full precision network and reduce the storage space by more than 20%. So, SNNs can use a small number of time steps to obtain better recognition accuracy. In the extreme case of using only a one-time step, we still can achieve 93.39, 92.12, and 69.55% testing accuracy on three traditional static datasets, Fashion- MNIST, CIFAR-10, and CIFAR-100, respectively. At the same time, we evaluate our method on neuromorphic N-MNIST, CIFAR10-DVS, and IBM DVS128 Gesture datasets and achieve advanced accuracy in SNN with binary weights. Our network has greater advantages in terms of storage resources and training time.
Collapse
Affiliation(s)
- Yijian Pei
- Guangzhou Institute of Technology, Xidian University, Xi'an, China
| | - Changqing Xu
- Guangzhou Institute of Technology, Xidian University, Xi'an, China
- School of Microelectronics, Xidian University, Xi'an, China
| | - Zili Wu
- School of Computer Science and Technology, Xidian University, Xi'an, China
| | - Yi Liu
- School of Microelectronics, Xidian University, Xi'an, China
| | - Yintang Yang
- School of Microelectronics, Xidian University, Xi'an, China
| |
Collapse
|
3
|
Park TJ, Deng S, Manna S, Islam ANMN, Yu H, Yuan Y, Fong DD, Chubykin AA, Sengupta A, Sankaranarayanan SKRS, Ramanathan S. Complex Oxides for Brain-Inspired Computing: A Review. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2023; 35:e2203352. [PMID: 35723973 DOI: 10.1002/adma.202203352] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Revised: 06/02/2022] [Indexed: 06/15/2023]
Abstract
The fields of brain-inspired computing, robotics, and, more broadly, artificial intelligence (AI) seek to implement knowledge gleaned from the natural world into human-designed electronics and machines. In this review, the opportunities presented by complex oxides, a class of electronic ceramic materials whose properties can be elegantly tuned by doping, electron interactions, and a variety of external stimuli near room temperature, are discussed. The review begins with a discussion of natural intelligence at the elementary level in the nervous system, followed by collective intelligence and learning at the animal colony level mediated by social interactions. An important aspect highlighted is the vast spatial and temporal scales involved in learning and memory. The focus then turns to collective phenomena, such as metal-to-insulator transitions (MITs), ferroelectricity, and related examples, to highlight recent demonstrations of artificial neurons, synapses, and circuits and their learning. First-principles theoretical treatments of the electronic structure, and in situ synchrotron spectroscopy of operating devices are then discussed. The implementation of the experimental characteristics into neural networks and algorithm design is then revewed. Finally, outstanding materials challenges that require a microscopic understanding of the physical mechanisms, which will be essential for advancing the frontiers of neuromorphic computing, are highlighted.
Collapse
Affiliation(s)
- Tae Joon Park
- School of Materials Engineering, Purdue University, West Lafayette, IN, 47907, USA
| | - Sunbin Deng
- School of Materials Engineering, Purdue University, West Lafayette, IN, 47907, USA
| | - Sukriti Manna
- Center for Nanoscale Materials, Argonne National Laboratory, Argonne, IL, 60439, USA
| | - A N M Nafiul Islam
- Department of Electrical Engineering, The Pennsylvania State University, University Park, PA, 16802, USA
| | - Haoming Yu
- School of Materials Engineering, Purdue University, West Lafayette, IN, 47907, USA
| | - Yifan Yuan
- School of Materials Engineering, Purdue University, West Lafayette, IN, 47907, USA
| | - Dillon D Fong
- Materials Science Division, Argonne National Laboratory, Lemont, IL, 60439, USA
| | - Alexander A Chubykin
- Department of Biological Sciences, Purdue Institute for Integrative Neuroscience, Purdue University, West Lafayette, IN, 47907, USA
| | - Abhronil Sengupta
- Department of Electrical Engineering, The Pennsylvania State University, University Park, PA, 16802, USA
| | - Subramanian K R S Sankaranarayanan
- Center for Nanoscale Materials, Argonne National Laboratory, Argonne, IL, 60439, USA
- Department of Mechanical and Industrial Engineering, University of Illinois Chicago, Chicago, IL, 60607, USA
| | - Shriram Ramanathan
- School of Materials Engineering, Purdue University, West Lafayette, IN, 47907, USA
| |
Collapse
|
4
|
Srinivasan G, Roy K. BlocTrain: Block-Wise Conditional Training and Inference for Efficient Spike-Based Deep Learning. Front Neurosci 2021; 15:603433. [PMID: 34776834 PMCID: PMC8586528 DOI: 10.3389/fnins.2021.603433] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2020] [Accepted: 07/23/2021] [Indexed: 12/04/2022] Open
Abstract
Spiking neural networks (SNNs), with their inherent capability to learn sparse spike-based input representations over time, offer a promising solution for enabling the next generation of intelligent autonomous systems. Nevertheless, end-to-end training of deep SNNs is both compute- and memory-intensive because of the need to backpropagate error gradients through time. We propose BlocTrain, which is a scalable and complexity-aware incremental algorithm for memory-efficient training of deep SNNs. We divide a deep SNN into blocks, where each block consists of few convolutional layers followed by a classifier. We train the blocks sequentially using local errors from the classifier. Once a given block is trained, our algorithm dynamically figures out easy vs. hard classes using the class-wise accuracy, and trains the deeper block only on the hard class inputs. In addition, we also incorporate a hard class detector (HCD) per block that is used during inference to exit early for the easy class inputs and activate the deeper blocks only for the hard class inputs. We trained ResNet-9 SNN divided into three blocks, using BlocTrain, on CIFAR-10 and obtained 86.4% accuracy, which is achieved with up to 2.95× lower memory requirement during the course of training, and 1.89× compute efficiency per inference (due to early exit strategy) with 1.45× memory overhead (primarily due to classifier weights) compared to end-to-end network. We also trained ResNet-11, divided into four blocks, on CIFAR-100 and obtained 58.21% accuracy, which is one of the first reported accuracy for SNN trained entirely with spike-based backpropagation on CIFAR-100.
Collapse
Affiliation(s)
- Gopalakrishnan Srinivasan
- Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, United States
| | - Kaushik Roy
- Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, United States
| |
Collapse
|
5
|
张 慧, 徐 桂, 郭 嘉, 郭 磊. [A review of brain-like spiking neural network and its neuromorphic chip research]. SHENG WU YI XUE GONG CHENG XUE ZA ZHI = JOURNAL OF BIOMEDICAL ENGINEERING = SHENGWU YIXUE GONGCHENGXUE ZAZHI 2021; 38:986-994. [PMID: 34713667 PMCID: PMC9927433 DOI: 10.7507/1001-5515.202011005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 11/02/2020] [Revised: 05/14/2021] [Indexed: 11/03/2022]
Abstract
Under the current situation of the rapid development of brain-like artificial intelligence and the increasingly complex electromagnetic environment, the most bionic and anti-interference spiking neural network has shown great potential in computing speed, real-time information processing, and spatiotemporal data processing. Spiking neural network is the core part of brain-like artificial intelligence, which realizes brain-like computing by simulating the structure of biological neural network and the way of information transmission. This article first summarizes the advantages and disadvantages of the five models, and analyzes the characteristics of several network topologies. Then, it summarizes the spiking neural network algorithms. The unsupervised learning based on spike timing dependent plasticity (STDP) rules and four types of supervised learning algorithms are analyzed. Finally, the research on brain-like neuromorphic chips at home and abroad are reviewed. This paper aims to provide learning ideas and research directions for new colleagues in the field of spiking neural network.
Collapse
Affiliation(s)
- 慧港 张
- 河北工业大学 省部共建电工装备可靠性与智能化国家重点实验室(天津 300130)State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 天津市生物电工与智能健康重点实验室(天津 300130)Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 河北省生物电磁与神经工程重点实验室(天津 300130)Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Hebei University of Technology, Tianjin 300130, P.R.China
| | - 桂芝 徐
- 河北工业大学 省部共建电工装备可靠性与智能化国家重点实验室(天津 300130)State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 天津市生物电工与智能健康重点实验室(天津 300130)Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 河北省生物电磁与神经工程重点实验室(天津 300130)Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Hebei University of Technology, Tianjin 300130, P.R.China
| | - 嘉荣 郭
- 河北工业大学 省部共建电工装备可靠性与智能化国家重点实验室(天津 300130)State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 天津市生物电工与智能健康重点实验室(天津 300130)Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 河北省生物电磁与神经工程重点实验室(天津 300130)Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Hebei University of Technology, Tianjin 300130, P.R.China
| | - 磊 郭
- 河北工业大学 省部共建电工装备可靠性与智能化国家重点实验室(天津 300130)State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 天津市生物电工与智能健康重点实验室(天津 300130)Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent Health, Hebei University of Technology, Tianjin 300130, P.R.China
- 河北工业大学 河北省生物电磁与神经工程重点实验室(天津 300130)Hebei Key Laboratory of Bioelectromagnetics and Neuroengineering, Hebei University of Technology, Tianjin 300130, P.R.China
| |
Collapse
|
6
|
Qiao G, Ning N, Zuo Y, Hu S, Yu Q, Liu Y. Direct training of hardware-friendly weight binarized spiking neural network with surrogate gradient learning towards spatio-temporal event-based dynamic data recognition. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.06.070] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
|
7
|
Nishi Y, Nomura K, Marukame T, Mizushima K. Stochastic binary synapses having sigmoidal cumulative distribution functions for unsupervised learning with spike timing-dependent plasticity. Sci Rep 2021; 11:18282. [PMID: 34521895 PMCID: PMC8440757 DOI: 10.1038/s41598-021-97583-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2021] [Accepted: 08/23/2021] [Indexed: 11/17/2022] Open
Abstract
Spike timing-dependent plasticity (STDP), which is widely studied as a fundamental synaptic update rule for neuromorphic hardware, requires precise control of continuous weights. From the viewpoint of hardware implementation, a simplified update rule is desirable. Although simplified STDP with stochastic binary synapses was proposed previously, we find that it leads to degradation of memory maintenance during learning, which is unfavourable for unsupervised online learning. In this work, we propose a stochastic binary synaptic model where the cumulative probability of the weight change evolves in a sigmoidal fashion with potentiation or depression trials, which can be implemented using a pair of switching devices consisting of serially connected multiple binary memristors. As a benchmark test we perform simulations of unsupervised learning of MNIST images with a two-layer network and show that simplified STDP in combination with this model can outperform conventional rules with continuous weights not only in memory maintenance but also in recognition accuracy. Our method achieves 97.3% in recognition accuracy, which is higher than that reported with standard STDP in the same framework. We also show that the high performance of our learning rule is robust against device-to-device variability of the memristor's probabilistic behaviour.
Collapse
Affiliation(s)
- Yoshifumi Nishi
- Frontier Research Laboratory, Corporate R&D Center, Toshiba Corporation, 1, Komukai-Toshiba-Cho, Saiwai-ku, Kawasaki, 212-8582, Japan.
| | - Kumiko Nomura
- Frontier Research Laboratory, Corporate R&D Center, Toshiba Corporation, 1, Komukai-Toshiba-Cho, Saiwai-ku, Kawasaki, 212-8582, Japan
| | - Takao Marukame
- Frontier Research Laboratory, Corporate R&D Center, Toshiba Corporation, 1, Komukai-Toshiba-Cho, Saiwai-ku, Kawasaki, 212-8582, Japan
| | - Koichi Mizushima
- Frontier Research Laboratory, Corporate R&D Center, Toshiba Corporation, 1, Komukai-Toshiba-Cho, Saiwai-ku, Kawasaki, 212-8582, Japan
| |
Collapse
|
8
|
A Cost-Efficient High-Speed VLSI Architecture for Spiking Convolutional Neural Network Inference Using Time-Step Binary Spike Maps. SENSORS 2021; 21:s21186006. [PMID: 34577214 PMCID: PMC8471769 DOI: 10.3390/s21186006] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/17/2021] [Revised: 08/31/2021] [Accepted: 09/03/2021] [Indexed: 11/23/2022]
Abstract
Neuromorphic hardware systems have been gaining ever-increasing focus in many embedded applications as they use a brain-inspired, energy-efficient spiking neural network (SNN) model that closely mimics the human cortex mechanism by communicating and processing sensory information via spatiotemporally sparse spikes. In this paper, we fully leverage the characteristics of spiking convolution neural network (SCNN), and propose a scalable, cost-efficient, and high-speed VLSI architecture to accelerate deep SCNN inference for real-time low-cost embedded scenarios. We leverage the snapshot of binary spike maps at each time-step, to decompose the SCNN operations into a series of regular and simple time-step CNN-like processing to reduce hardware resource consumption. Moreover, our hardware architecture achieves high throughput by employing a pixel stream processing mechanism and fine-grained data pipelines. Our Zynq-7045 FPGA prototype reached a high processing speed of 1250 frames/s and high recognition accuracies on the MNIST and Fashion-MNIST image datasets, demonstrating the plausibility of our SCNN hardware architecture for many embedded applications.
Collapse
|
9
|
Wang G, Ma S, Wu Y, Pei J, Zhao R, Shi L. End-to-End Implementation of Various Hybrid Neural Networks on a Cross-Paradigm Neuromorphic Chip. Front Neurosci 2021; 15:615279. [PMID: 33603643 PMCID: PMC7884322 DOI: 10.3389/fnins.2021.615279] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2020] [Accepted: 01/08/2021] [Indexed: 11/13/2022] Open
Abstract
Integration of computer-science oriented artificial neural networks (ANNs) and neuroscience oriented spiking neural networks (SNNs) has emerged as a highly promising direction to achieve further breakthroughs in artificial intelligence through complementary advantages. This integration needs to support individual modeling of ANNs and SNNs as well as their hybrid modeling, which not only simultaneously calculates single-paradigm networks but also converts their different information representations. It remains challenging to realize effective calculation and signal conversion on the existing dedicated hardware platforms. To solve this problem, we propose an end-to-end mapping framework for implementing various hybrid neural networks on many-core neuromorphic architectures based on the cross-paradigm Tianjic chip. We construct hardware configuration schemes for four typical signal conversions and establish a global timing adjustment mechanism among different heterogeneous modules. Experimental results show that our framework can implement these hybrid models with low execution latency and low power consumption with nearly no accuracy degradation. This work provides a new approach of developing hybrid neural network models for brain-inspired computing chips and further tapping the potential of these models.
Collapse
Affiliation(s)
- Guanrui Wang
- Department of Precision Instrument, Center for Brain-Inspired Computing Research (CBICR), Beijing Innovation Center for Future Chip, Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China
| | - Songchen Ma
- Department of Precision Instrument, Center for Brain-Inspired Computing Research (CBICR), Beijing Innovation Center for Future Chip, Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China
| | - Yujie Wu
- Department of Precision Instrument, Center for Brain-Inspired Computing Research (CBICR), Beijing Innovation Center for Future Chip, Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China
| | - Jing Pei
- Department of Precision Instrument, Center for Brain-Inspired Computing Research (CBICR), Beijing Innovation Center for Future Chip, Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China
| | - Rong Zhao
- Department of Precision Instrument, Center for Brain-Inspired Computing Research (CBICR), Beijing Innovation Center for Future Chip, Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China
| | - Luping Shi
- Department of Precision Instrument, Center for Brain-Inspired Computing Research (CBICR), Beijing Innovation Center for Future Chip, Optical Memory National Engineering Research Center, Tsinghua University, Beijing, China
| |
Collapse
|
10
|
Wang T, Shi C, Zhou X, Lin Y, He J, Gan P, Li P, Wang Y, Liu L, Wu N, Luo G. CompSNN: A lightweight spiking neural network based on spatiotemporally compressive spike features. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.10.100] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
11
|
She X, Dash S, Kim D, Mukhopadhyay S. A Heterogeneous Spiking Neural Network for Unsupervised Learning of Spatiotemporal Patterns. Front Neurosci 2021; 14:615756. [PMID: 33519366 PMCID: PMC7841292 DOI: 10.3389/fnins.2020.615756] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2020] [Accepted: 12/11/2020] [Indexed: 11/19/2022] Open
Abstract
This paper introduces a heterogeneous spiking neural network (H-SNN) as a novel, feedforward SNN structure capable of learning complex spatiotemporal patterns with spike-timing-dependent plasticity (STDP) based unsupervised training. Within H-SNN, hierarchical spatial and temporal patterns are constructed with convolution connections and memory pathways containing spiking neurons with different dynamics. We demonstrate analytically the formation of long and short term memory in H-SNN and distinct response functions of memory pathways. In simulation, the network is tested on visual input of moving objects to simultaneously predict for object class and motion dynamics. Results show that H-SNN achieves prediction accuracy on similar or higher level than supervised deep neural networks (DNN). Compared to SNN trained with back-propagation, H-SNN effectively utilizes STDP to learn spatiotemporal patterns that have better generalizability to unknown motion and/or object classes encountered during inference. In addition, the improved performance is achieved with 6x fewer parameters than complex DNNs, showing H-SNN as an efficient approach for applications with constrained computation resources.
Collapse
Affiliation(s)
- Xueyuan She
- Department of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, United States
| | | | | | | |
Collapse
|
12
|
Panda P, Aketi SA, Roy K. Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization. Front Neurosci 2020; 14:653. [PMID: 32694977 PMCID: PMC7339963 DOI: 10.3389/fnins.2020.00653] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2020] [Accepted: 05/26/2020] [Indexed: 11/24/2022] Open
Abstract
Spiking Neural Networks (SNNs) may offer an energy-efficient alternative for implementing deep learning applications. In recent years, there have been several proposals focused on supervised (conversion, spike-based gradient descent) and unsupervised (spike timing dependent plasticity) training methods to improve the accuracy of SNNs on large-scale tasks. However, each of these methods suffer from scalability, latency, and accuracy limitations. In this paper, we propose novel algorithmic techniques of modifying the SNN configuration with backward residual connections, stochastic softmax, and hybrid artificial-and-spiking neuronal activations to improve the learning ability of the training methodologies to yield competitive accuracy, while, yielding large efficiency gains over their artificial counterparts. Note, artificial counterparts refer to conventional deep learning/artificial neural networks. Our techniques apply to VGG/Residual architectures, and are compatible with all forms of training methodologies. Our analysis reveals that the proposed solutions yield near state-of-the-art accuracy with significant energy-efficiency and reduced parameter overhead translating to hardware improvements on complex visual recognition tasks, such as, CIFAR10, Imagenet datatsets.
Collapse
Affiliation(s)
- Priyadarshini Panda
- Department of Electrical Engineering, Yale University, New Haven, CT, United States
| | - Sai Aparna Aketi
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, United States
| | - Kaushik Roy
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, United States
| |
Collapse
|
13
|
Lu S, Sengupta A. Exploring the Connection Between Binary and Spiking Neural Networks. Front Neurosci 2020; 14:535. [PMID: 32670002 PMCID: PMC7327094 DOI: 10.3389/fnins.2020.00535] [Citation(s) in RCA: 34] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2020] [Accepted: 04/30/2020] [Indexed: 11/13/2022] Open
Abstract
On-chip edge intelligence has necessitated the exploration of algorithmic techniques to reduce the compute requirements of current machine learning frameworks. This work aims to bridge the recent algorithmic progress in training Binary Neural Networks and Spiking Neural Networks-both of which are driven by the same motivation and yet synergies between the two have not been fully explored. We show that training Spiking Neural Networks in the extreme quantization regime results in near full precision accuracies on large-scale datasets like CIFAR-100 and ImageNet. An important implication of this work is that Binary Spiking Neural Networks can be enabled by "In-Memory" hardware accelerators catered for Binary Neural Networks without suffering any accuracy degradation due to binarization. We utilize standard training techniques for non-spiking networks to generate our spiking networks by conversion process and also perform an extensive empirical analysis and explore simple design-time and run-time optimization techniques for reducing inference latency of spiking networks (both for binary and full-precision models) by an order of magnitude over prior work. Our implementation source code and trained models are available at https://github.com/NeuroCompLab-psu/SNN-Conversion.
Collapse
Affiliation(s)
- Sen Lu
- School of Electrical Engineering and Computer Science, The Pennsylvania State University, University Park, PA, United States
| | - Abhronil Sengupta
- School of Electrical Engineering and Computer Science, The Pennsylvania State University, University Park, PA, United States
| |
Collapse
|
14
|
Chakraborty I, Agrawal A, Jaiswal A, Srinivasan G, Roy K. In situ unsupervised learning using stochastic switching in magneto-electric magnetic tunnel junctions. PHILOSOPHICAL TRANSACTIONS. SERIES A, MATHEMATICAL, PHYSICAL, AND ENGINEERING SCIENCES 2020; 378:20190157. [PMID: 31865881 PMCID: PMC6939242 DOI: 10.1098/rsta.2019.0157] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 09/30/2019] [Indexed: 06/10/2023]
Abstract
Spiking neural networks (SNNs) offer a bio-plausible and potentially power-efficient alternative to conventional deep learning. Although there has been progress towards implementing SNN functionalities in custom CMOS-based hardware using beyond Von Neumann architectures, the power-efficiency of the human brain has remained elusive. This has necessitated investigations of novel material systems which can efficiently mimic the functional units of SNNs, such as neurons and synapses. In this paper, we present a magnetoelectric-magnetic tunnel junction (ME-MTJ) device as a synapse. We arrange these synapses in a crossbar fashion and perform in situ unsupervised learning. We leverage the capacitive nature of write-ports in ME-MTJs, wherein by applying appropriately shaped voltage pulses across the write-port, the ME-MTJ can be switched in a probabilistic manner. We further exploit the sigmoidal switching characteristics of ME-MTJ to tune the synapses to follow the well-known spike timing-dependent plasticity (STDP) rule in a stochastic fashion. Finally, we use the stochastic STDP rule in ME-MTJ synapses to simulate a two-layered SNN to perform image classification tasks on a handwritten digit dataset. Thus, the capacitive write-port and the decoupled-nature of read-write path of ME-MTJs allow us to construct a transistor-less crossbar, suitable for energy-efficient implementation of in situ learning in SNNs. This article is part of the theme issue 'Harmonizing energy-autonomous computing and intelligence'.
Collapse
|