1
|
Dong Y, Zhao D, Li Y, Zeng Y. An unsupervised STDP-based spiking neural network inspired by biologically plausible learning rules and connections. Neural Netw 2023; 165:799-808. [PMID: 37418862 DOI: 10.1016/j.neunet.2023.06.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2022] [Revised: 06/14/2023] [Accepted: 06/15/2023] [Indexed: 07/09/2023]
Abstract
The backpropagation algorithm has promoted the rapid development of deep learning, but it relies on a large amount of labeled data and still has a large gap with how humans learn. The human brain can quickly learn various conceptual knowledge in a self-organized and unsupervised manner, accomplished through coordinating various learning rules and structures in the human brain. Spike-timing-dependent plasticity (STDP) is a general learning rule in the brain, but spiking neural networks (SNNs) trained with STDP alone is inefficient and perform poorly. In this paper, taking inspiration from short-term synaptic plasticity, we design an adaptive synaptic filter and introduce the adaptive spiking threshold as the neuron plasticity to enrich the representation ability of SNNs. We also introduce an adaptive lateral inhibitory connection to adjust the spikes balance dynamically to help the network learn richer features. To speed up and stabilize the training of unsupervised spiking neural networks, we design a samples temporal batch STDP (STB-STDP), which updates weights based on multiple samples and moments. By integrating the above three adaptive mechanisms and STB-STDP, our model greatly accelerates the training of unsupervised spiking neural networks and improves the performance of unsupervised SNNs on complex tasks. Our model achieves the current state-of-the-art performance of unsupervised STDP-based SNNs in the MNIST and FashionMNIST datasets. Further, we tested on the more complex CIFAR10 dataset, and the results fully illustrate the superiority of our algorithm. Our model is also the first work to apply unsupervised STDP-based SNNs to CIFAR10. At the same time, in the small-sample learning scenario, it will far exceed the supervised ANN using the same structure.
Collapse
Affiliation(s)
- Yiting Dong
- School of Future Technology, University of Chinese Academy of Sciences, Beijing, China; Brain-Inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences (CAS), Beijing, China
| | - Dongcheng Zhao
- Brain-Inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences (CAS), Beijing, China
| | - Yang Li
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China; Brain-Inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences (CAS), Beijing, China
| | - Yi Zeng
- School of Future Technology, University of Chinese Academy of Sciences, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China; Brain-Inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences (CAS), Beijing, China; Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences (CAS), Shanghai, China; State Key Laboratory of Multimodal Artifcial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences (CAS), Beijing, China.
| |
Collapse
|
2
|
Patil AR, Dongale TD, Namade LD, Mohite SV, Kim Y, Sutar SS, Kamat RK, Rajpure KY. Sprayed FeWO4 thin film-based memristive device with negative differential resistance effect for non-volatile memory and synaptic learning applications. J Colloid Interface Sci 2023; 642:540-553. [PMID: 37028161 DOI: 10.1016/j.jcis.2023.03.189] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2023] [Revised: 03/21/2023] [Accepted: 03/29/2023] [Indexed: 04/04/2023]
Abstract
Resistive switching (RS) memories have attracted great attention as promising solutions to next-generation non-volatile memories and computing technologies because of their simple device configuration, high on/off ratio, low power consumption, fast switching, long retention, and significant cyclic stability. In this work, uniform and adherent iron tungstate (FeWO4) thin films were synthesized by the spray pyrolysis method with various precursor solution volumes, and these were tested as a switching layer for the fabrication of Ag/FWO/FTO memristive devices. The detailed structural investigation was done through various analytical and physio-chemical characterizations viz. X-ray diffraction (XRD) and its Rietveld refinement, Raman spectroscopy, and X-ray photoelectron spectroscopy (XPS) techniques. The results reveal the pure and single-phase FeWO4 compound thin film formation. Surface morphological study shows the spherical particle formation having a diameter in the range of 20 to 40 nm. The RS characteristics of the Ag/FWO/FTO memristive device demonstrate non-volatile memory characteristics with significant endurance and retention properties. Interestingly, the memory devices show stable and reproducible negative differential resistance (NDR) effects. The in-depth statistical analysis suggests the good operational uniformity of the device. Moreover, the switching voltages of the Ag/FWO/FTO memristive device were modeled using the time series analysis technique by utilizing Holt's Winter Exponential Smoothing (HWES) approach. Additionally, the device mimics bio-synaptic properties such as potentiation/depression, excitatory post-synaptic current (EPSC), and spike-timing-dependent plasticity (STDP) learning rules. For the present device, the space-charge-limited current (SCLC) and trap-controlled-SCLC effects dominated during positive and negative bias I-V characteristics, respectively. The RS mechanism dominated in the low resistance state (LRS), and the high resistance state (HRS) was explained based on the formation and rupture of conductive filament composed of Ag ions and oxygen vacancies. This work demonstrates the RS in the metal tungstate-based memristive devices and demonstrates a low-cost approach for fabricating memristive devices.
Collapse
Affiliation(s)
- Amitkumar R Patil
- Electrochemical Materials Laboratory, Department of Physics, Shivaji University, Kolhapur 416004, India
| | - Tukaram D Dongale
- Computational Electronics and Nanoscience Research Laboratory, School of Nanoscience and Biotechnology, Shivaji University, Kolhapur 416004, India
| | - Lahu D Namade
- Electrochemical Materials Laboratory, Department of Physics, Shivaji University, Kolhapur 416004, India
| | - Santosh V Mohite
- Department of Applied Chemistry, Konkuk University, Chungju 27478, Republic of Korea
| | - Yeonho Kim
- Department of Applied Chemistry, Konkuk University, Chungju 27478, Republic of Korea
| | - Santosh S Sutar
- Yashwantrao Chavan School of Rural Development, Shivaji University, Kolhapur 416004, India
| | - Rajanish K Kamat
- Department of Electronics, Shivaji University, Kolhapur 416004, India; Dr. Homi Bhabha State University, 15, Madam Cama Road, Mumbai 400032, India
| | - Keshav Y Rajpure
- Electrochemical Materials Laboratory, Department of Physics, Shivaji University, Kolhapur 416004, India.
| |
Collapse
|
3
|
Galinsky VL, Frank LR. Critically synchronized brain waves form an effective, robust and flexible basis for human memory and learning. Sci Rep 2023; 13:4343. [PMID: 36928606 PMCID: PMC10020450 DOI: 10.1038/s41598-023-31365-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2023] [Accepted: 03/10/2023] [Indexed: 03/18/2023] Open
Abstract
The effectiveness, robustness, and flexibility of memory and learning constitute the very essence of human natural intelligence, cognition, and consciousness. However, currently accepted views on these subjects have, to date, been put forth without any basis on a true physical theory of how the brain communicates internally via its electrical signals. This lack of a solid theoretical framework has implications not only for our understanding of how the brain works, but also for wide range of computational models developed from the standard orthodox view of brain neuronal organization and brain network derived functioning based on the Hodgkin-Huxley ad-hoc circuit analogies that have produced a multitude of Artificial, Recurrent, Convolution, Spiking, etc., Neural Networks (ARCSe NNs) that have in turn led to the standard algorithms that form the basis of artificial intelligence (AI) and machine learning (ML) methods. Our hypothesis, based upon our recently developed physical model of weakly evanescent brain wave propagation (WETCOW) is that, contrary to the current orthodox model that brain neurons just integrate and fire under accompaniment of slow leaking, they can instead perform much more sophisticated tasks of efficient coherent synchronization/desynchronization guided by the collective influence of propagating nonlinear near critical brain waves, the waves that currently assumed to be nothing but inconsequential subthreshold noise. In this paper we highlight the learning and memory capabilities of our WETCOW framework and then apply it to the specific application of AI/ML and Neural Networks. We demonstrate that the learning inspired by these critically synchronized brain waves is shallow, yet its timing and accuracy outperforms deep ARCSe counterparts on standard test datasets. These results have implications for both our understanding of brain function and for the wide range of AI/ML applications.
Collapse
Affiliation(s)
- Vitaly L Galinsky
- Center for Scientific Computation in Imaging, University of California at San Diego, La Jolla, CA, 92037-0854, USA.
| | - Lawrence R Frank
- Center for Scientific Computation in Imaging, University of California at San Diego, La Jolla, CA, 92037-0854, USA
- Center for Functional MRI, University of California at San Diego, La Jolla, CA, 92037-0677, USA
| |
Collapse
|
4
|
Pietrzak P, Szczęsny S, Huderek D, Przyborowski Ł. Overview of Spiking Neural Network Learning Approaches and Their Computational Complexities. SENSORS (BASEL, SWITZERLAND) 2023; 23:3037. [PMID: 36991750 PMCID: PMC10053242 DOI: 10.3390/s23063037] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 03/08/2023] [Accepted: 03/09/2023] [Indexed: 06/19/2023]
Abstract
Spiking neural networks (SNNs) are subjects of a topic that is gaining more and more interest nowadays. They more closely resemble actual neural networks in the brain than their second-generation counterparts, artificial neural networks (ANNs). SNNs have the potential to be more energy efficient than ANNs on event-driven neuromorphic hardware. This can yield drastic maintenance cost reduction for neural network models, as the energy consumption would be much lower in comparison to regular deep learning models hosted in the cloud today. However, such hardware is still not yet widely available. On standard computer architectures consisting mainly of central processing units (CPUs) and graphics processing units (GPUs) ANNs, due to simpler models of neurons and simpler models of connections between neurons, have the upper hand in terms of execution speed. In general, they also win in terms of learning algorithms, as SNNs do not reach the same levels of performance as their second-generation counterparts in typical machine learning benchmark tasks, such as classification. In this paper, we review existing learning algorithms for spiking neural networks, divide them into categories by type, and assess their computational complexity.
Collapse
|
5
|
Guo W, Fouda ME, Eltawil AM, Salama KN. Efficient training of spiking neural networks with temporally-truncated local backpropagation through time. Front Neurosci 2023; 17:1047008. [PMID: 37090791 PMCID: PMC10117667 DOI: 10.3389/fnins.2023.1047008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2022] [Accepted: 03/20/2023] [Indexed: 04/25/2023] Open
Abstract
Directly training spiking neural networks (SNNs) has remained challenging due to complex neural dynamics and intrinsic non-differentiability in firing functions. The well-known backpropagation through time (BPTT) algorithm proposed to train SNNs suffers from large memory footprint and prohibits backward and update unlocking, making it impossible to exploit the potential of locally-supervised training methods. This work proposes an efficient and direct training algorithm for SNNs that integrates a locally-supervised training method with a temporally-truncated BPTT algorithm. The proposed algorithm explores both temporal and spatial locality in BPTT and contributes to significant reduction in computational cost including GPU memory utilization, main memory access and arithmetic operations. We thoroughly explore the design space concerning temporal truncation length and local training block size and benchmark their impact on classification accuracy of different networks running different types of tasks. The results reveal that temporal truncation has a negative effect on the accuracy of classifying frame-based datasets, but leads to improvement in accuracy on event-based datasets. In spite of resulting information loss, local training is capable of alleviating overfitting. The combined effect of temporal truncation and local training can lead to the slowdown of accuracy drop and even improvement in accuracy. In addition, training deep SNNs' models such as AlexNet classifying CIFAR10-DVS dataset leads to 7.26% increase in accuracy, 89.94% reduction in GPU memory, 10.79% reduction in memory access, and 99.64% reduction in MAC operations compared to the standard end-to-end BPTT. Thus, the proposed method has shown high potential to enable fast and energy-efficient on-chip training for real-time learning at the edge.
Collapse
Affiliation(s)
- Wenzhe Guo
- Sensors Lab, Advanced Membranes and Porous Materials Center (AMPMC), Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
- Communication and Computing Systems Lab, Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
| | - Mohammed E. Fouda
- Center for Embedded & Cyber-Physical Systems, University of California, Irvine, Irvine, CA, United States
| | - Ahmed M. Eltawil
- Communication and Computing Systems Lab, Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
- Center for Embedded & Cyber-Physical Systems, University of California, Irvine, Irvine, CA, United States
| | - Khaled Nabil Salama
- Sensors Lab, Advanced Membranes and Porous Materials Center (AMPMC), Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
- *Correspondence: Khaled Nabil Salama
| |
Collapse
|
6
|
Ou W, Xiao S, Zhu C, Han W, Zhang Q. An overview of brain-like computing: Architecture, applications, and future trends. Front Neurorobot 2022; 16:1041108. [PMID: 36506817 PMCID: PMC9730831 DOI: 10.3389/fnbot.2022.1041108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2022] [Accepted: 10/31/2022] [Indexed: 11/25/2022] Open
Abstract
With the development of technology, Moore's law will come to an end, and scientists are trying to find a new way out in brain-like computing. But we still know very little about how the brain works. At the present stage of research, brain-like models are all structured to mimic the brain in order to achieve some of the brain's functions, and then continue to improve the theories and models. This article summarizes the important progress and status of brain-like computing, summarizes the generally accepted and feasible brain-like computing models, introduces, analyzes, and compares the more mature brain-like computing chips, outlines the attempts and challenges of brain-like computing applications at this stage, and looks forward to the future development of brain-like computing. It is hoped that the summarized results will help relevant researchers and practitioners to quickly grasp the research progress in the field of brain-like computing and acquire the application methods and related knowledge in this field.
Collapse
Affiliation(s)
- Wei Ou
- The School of Cyberspace Security, Hainan University, Hainan, China
- Henan Key Laboratory of Network Cryptography Technology, Zhengzhou, China
| | - Shitao Xiao
- The School of Computer Science and Technology, Hainan, China
| | - Chengyu Zhu
- The School of Cyberspace Security, Hainan University, Hainan, China
| | - Wenbao Han
- The School of Cyberspace Security, Hainan University, Hainan, China
| | - Qionglu Zhang
- State Key Laboratory of Information Security, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
7
|
Li Y, Zhao D, Zeng Y. BSNN: Towards faster and better conversion of artificial neural networks to spiking neural networks with bistable neurons. Front Neurosci 2022; 16:991851. [PMID: 36312025 PMCID: PMC9597447 DOI: 10.3389/fnins.2022.991851] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2022] [Accepted: 09/26/2022] [Indexed: 11/30/2022] Open
Abstract
The spiking neural network (SNN) computes and communicates information through discrete binary events. Recent work has achieved essential progress on an excellent performance by converting ANN to SNN. Due to the difference in information processing, the converted deep SNN usually suffers serious performance loss and large time delay. In this paper, we analyze the reasons for the performance loss and propose a novel bistable spiking neural network (BSNN) that addresses the problem of the phase lead and phase lag. Also, we design synchronous neurons (SN) to help efficiently improve performance when ResNet structure-based ANNs are converted. BSNN significantly improves the performance of the converted SNN by enabling more accurate delivery of information to the next layer after one cycle. Experimental results show that the proposed method only needs 1/4-1/10 of the time steps compared to previous work to achieve nearly lossless conversion. We demonstrate better ANN-SNN conversion for VGG16, ResNet20, and ResNet34 on challenging datasets including CIFAR-10 (95.16% top-1), CIFAR-100 (78.12% top-1), and ImageNet (72.64% top-1).
Collapse
Affiliation(s)
- Yang Li
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Dongcheng Zhao
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Yi Zeng
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
- Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China
- National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| |
Collapse
|