1
|
Baek S, Lee J. Snn and sound: a comprehensive review of spiking neural networks in sound. Biomed Eng Lett 2024; 14:981-991. [PMID: 39220030 PMCID: PMC11362401 DOI: 10.1007/s13534-024-00406-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Revised: 05/08/2024] [Accepted: 06/24/2024] [Indexed: 09/04/2024] Open
Abstract
The rapid advancement of AI and machine learning has significantly enhanced sound and acoustic recognition technologies, moving beyond traditional models to more sophisticated neural network-based methods. Among these, Spiking Neural Networks (SNNs) are particularly noteworthy. SNNs mimic biological neurons and operate on principles similar to the human brain, using analog computing mechanisms. This capability allows for efficient sound processing with low power consumption and minimal latency, ideal for real-time applications in embedded systems. This paper reviews recent developments in SNNs for sound recognition, underscoring their potential to overcome the limitations of digital computing and suggesting directions for future research. The unique attributes of SNNs could lead to breakthroughs in mimicking human auditory processing more closely.
Collapse
Affiliation(s)
- Suwhan Baek
- AI R &D Laboratory, Posco-Holdings, Cheongam-ro, Pohang-si, Gyeongsangbuk-do 37673 Korea
- Department of Computer Science, Kwangwoon University, Gwangun-ro, Nowon-gu, Seoul, 01899 Republic of Korea
| | - Jaewon Lee
- Department of Psychology, Seoul National University, Gwanak-ro, Gwanak-gu, Seoul, 08826 Republic of Korea
| |
Collapse
|
2
|
Yang G, Kang Y, Charlton PH, Kyriacou PA, Kim KK, Li L, Park C. Energy-Efficient PPG-Based Respiratory Rate Estimation Using Spiking Neural Networks. SENSORS (BASEL, SWITZERLAND) 2024; 24:3980. [PMID: 38931763 PMCID: PMC11207339 DOI: 10.3390/s24123980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/16/2024] [Revised: 06/12/2024] [Accepted: 06/18/2024] [Indexed: 06/28/2024]
Abstract
Respiratory rate (RR) is a vital indicator for assessing the bodily functions and health status of patients. RR is a prominent parameter in the field of biomedical signal processing and is strongly associated with other vital signs such as blood pressure, heart rate, and heart rate variability. Various physiological signals, such as photoplethysmogram (PPG) signals, are used to extract respiratory information. RR is also estimated by detecting peak patterns and cycles in the signals through signal processing and deep-learning approaches. In this study, we propose an end-to-end RR estimation approach based on a third-generation artificial neural network model-spiking neural network. The proposed model employs PPG segments as inputs, and directly converts them into sequential spike events. This design aims to reduce information loss during the conversion of the input data into spike events. In addition, we use feedback-based integrate-and-fire neurons as the activation functions, which effectively transmit temporal information. The network is evaluated using the BIDMC respiratory dataset with three different window sizes (16, 32, and 64 s). The proposed model achieves mean absolute errors of 1.37 ± 0.04, 1.23 ± 0.03, and 1.15 ± 0.07 for the 16, 32, and 64 s window sizes, respectively. Furthermore, it demonstrates superior energy efficiency compared with other deep learning models. This study demonstrates the potential of the spiking neural networks for RR monitoring, offering a novel approach for RR estimation from the PPG signal.
Collapse
Affiliation(s)
- Geunbo Yang
- Department of Computer Engineering, Kwangwoon University, Seoul 01897, Republic of Korea; (G.Y.); (Y.K.)
| | - Youngshin Kang
- Department of Computer Engineering, Kwangwoon University, Seoul 01897, Republic of Korea; (G.Y.); (Y.K.)
| | - Peter H. Charlton
- Department of Public Health and Primary Care, University of Cambridge, Cambridge CB1 8RN, UK;
| | - Panayiotis A. Kyriacou
- Department of Engineering, School of Science and Technology (SST), City University of London, London EC1V 0HB, UK;
| | - Ko Keun Kim
- AI Lab, LG Electronics, Seoul 06763, Republic of Korea;
| | - Ling Li
- Department of Engineering, School of Science and Technology (SST), City University of London, London EC1V 0HB, UK;
| | - Cheolsoo Park
- Department of Computer Engineering, Kwangwoon University, Seoul 01897, Republic of Korea; (G.Y.); (Y.K.)
| |
Collapse
|
3
|
Bukh AV, Rybalova EV, Shepelev IA, Vadivasova TE. Classification of musical intervals by spiking neural networks: Perfect student in solfége classes. CHAOS (WOODBURY, N.Y.) 2024; 34:063102. [PMID: 38829796 DOI: 10.1063/5.0210790] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2024] [Accepted: 05/12/2024] [Indexed: 06/05/2024]
Abstract
We investigate a spike activity of a network of excitable FitzHugh-Nagumo neurons, which is under constant two-frequency auditory signals. The neurons are supplemented with linear frequency filters and nonlinear input signal converters. We show that it is possible to configure the network to recognize a specific frequency ratio (musical interval) by selecting the parameters of the neurons, input filters, and coupling between neurons. A set of appropriately configured subnetworks with different topologies and coupling strengths can serve as a classifier for musical intervals. We have found that the selective properties of the classifier are due to the presence of a specific topology of coupling between the neurons of the network.
Collapse
Affiliation(s)
- A V Bukh
- Institute of Physics, Saratov State University, 83 Astrakhanskaya Street, Saratov 410012, Russia
| | - E V Rybalova
- Institute of Physics, Saratov State University, 83 Astrakhanskaya Street, Saratov 410012, Russia
| | - I A Shepelev
- Institute of Physics, Saratov State University, 83 Astrakhanskaya Street, Saratov 410012, Russia
- Almetyevsk State Petroleum Institute, 2 Lenin Street, Almetyevsk 423462, Russia
| | - T E Vadivasova
- Institute of Physics, Saratov State University, 83 Astrakhanskaya Street, Saratov 410012, Russia
| |
Collapse
|
4
|
Liu F, Zheng H, Ma S, Zhang W, Liu X, Chua Y, Shi L, Zhao R. Advancing brain-inspired computing with hybrid neural networks. Natl Sci Rev 2024; 11:nwae066. [PMID: 38577666 PMCID: PMC10989656 DOI: 10.1093/nsr/nwae066] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2023] [Revised: 01/25/2024] [Accepted: 01/31/2024] [Indexed: 04/06/2024] Open
Abstract
Brain-inspired computing, drawing inspiration from the fundamental structure and information-processing mechanisms of the human brain, has gained significant momentum in recent years. It has emerged as a research paradigm centered on brain-computer dual-driven and multi-network integration. One noteworthy instance of this paradigm is the hybrid neural network (HNN), which integrates computer-science-oriented artificial neural networks (ANNs) with neuroscience-oriented spiking neural networks (SNNs). HNNs exhibit distinct advantages in various intelligent tasks, including perception, cognition and learning. This paper presents a comprehensive review of HNNs with an emphasis on their origin, concepts, biological perspective, construction framework and supporting systems. Furthermore, insights and suggestions for potential research directions are provided aiming to propel the advancement of the HNN paradigm.
Collapse
Affiliation(s)
- Faqiang Liu
- Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co. Joint Research Center for Brain-inspired Computing, IDG/McGovern Institute for Brain Research, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Hao Zheng
- Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co. Joint Research Center for Brain-inspired Computing, IDG/McGovern Institute for Brain Research, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Songchen Ma
- Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co. Joint Research Center for Brain-inspired Computing, IDG/McGovern Institute for Brain Research, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Weihao Zhang
- Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co. Joint Research Center for Brain-inspired Computing, IDG/McGovern Institute for Brain Research, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Xue Liu
- Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co. Joint Research Center for Brain-inspired Computing, IDG/McGovern Institute for Brain Research, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Yansong Chua
- Neuromorphic Computing Laboratory, China Nanhu Academy of Electronics and Information Technology, Jiaxing 314001, China
| | - Luping Shi
- Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co. Joint Research Center for Brain-inspired Computing, IDG/McGovern Institute for Brain Research, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| | - Rong Zhao
- Center for Brain-Inspired Computing Research, Optical Memory National Engineering Research Center, Tsinghua University-China Electronics Technology HIK Group Co. Joint Research Center for Brain-inspired Computing, IDG/McGovern Institute for Brain Research, Department of Precision Instrument, Tsinghua University, Beijing 100084, China
| |
Collapse
|
5
|
Chen X, Yang Q, Wu J, Li H, Tan KC. A Hybrid Neural Coding Approach for Pattern Recognition With Spiking Neural Networks. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:3064-3078. [PMID: 38055367 DOI: 10.1109/tpami.2023.3339211] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/08/2023]
Abstract
Recently, brain-inspired spiking neural networks (SNNs) have demonstrated promising capabilities in solving pattern recognition tasks. However, these SNNs are grounded on homogeneous neurons that utilize a uniform neural coding for information representation. Given that each neural coding scheme possesses its own merits and drawbacks, these SNNs encounter challenges in achieving optimal performance such as accuracy, response time, efficiency, and robustness, all of which are crucial for practical applications. In this study, we argue that SNN architectures should be holistically designed to incorporate heterogeneous coding schemes. As an initial exploration in this direction, we propose a hybrid neural coding and learning framework, which encompasses a neural coding zoo with diverse neural coding schemes discovered in neuroscience. Additionally, it incorporates a flexible neural coding assignment strategy to accommodate task-specific requirements, along with novel layer-wise learning methods to effectively implement hybrid coding SNNs. We demonstrate the superiority of the proposed framework on image classification and sound localization tasks. Specifically, the proposed hybrid coding SNNs achieve comparable accuracy to state-of-the-art SNNs, while exhibiting significantly reduced inference latency and energy consumption, as well as high noise robustness. This study yields valuable insights into hybrid neural coding designs, paving the way for developing high-performance neuromorphic systems.
Collapse
|
6
|
Yu Q, Gao J, Wei J, Li J, Tan KC, Huang T. Improving Multispike Learning With Plastic Synaptic Delays. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:10254-10265. [PMID: 35442893 DOI: 10.1109/tnnls.2022.3165527] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Emulating the spike-based processing in the brain, spiking neural networks (SNNs) are developed and act as a promising candidate for the new generation of artificial neural networks that aim to produce efficient cognitions as the brain. Due to the complex dynamics and nonlinearity of SNNs, designing efficient learning algorithms has remained a major difficulty, which attracts great research attention. Most existing ones focus on the adjustment of synaptic weights. However, other components, such as synaptic delays, are found to be adaptive and important in modulating neural behavior. How could plasticity on different components cooperate to improve the learning of SNNs remains as an interesting question. Advancing our previous multispike learning, we propose a new joint weight-delay plasticity rule, named TDP-DL, in this article. Plastic delays are integrated into the learning framework, and as a result, the performance of multispike learning is significantly improved. Simulation results highlight the effectiveness and efficiency of our TDP-DL rule compared to baseline ones. Moreover, we reveal the underlying principle of how synaptic weights and delays cooperate with each other through a synthetic task of interval selectivity and show that plastic delays can enhance the selectivity and flexibility of neurons by shifting information across time. Due to this capability, useful information distributed away in the time domain can be effectively integrated for a better accuracy performance, as highlighted in our generalization tasks of the image, speech, and event-based object recognitions. Our work is thus valuable and significant to improve the performance of spike-based neuromorphic computing.
Collapse
|
7
|
Luo X, Qu H, Wang Y, Yi Z, Zhang J, Zhang M. Supervised Learning in Multilayer Spiking Neural Networks With Spike Temporal Error Backpropagation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:10141-10153. [PMID: 35436200 DOI: 10.1109/tnnls.2022.3164930] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
The brain-inspired spiking neural networks (SNNs) hold the advantages of lower power consumption and powerful computing capability. However, the lack of effective learning algorithms has obstructed the theoretical advance and applications of SNNs. The majority of the existing learning algorithms for SNNs are based on the synaptic weight adjustment. However, neuroscience findings confirm that synaptic delays can also be modulated to play an important role in the learning process. Here, we propose a gradient descent-based learning algorithm for synaptic delays to enhance the sequential learning performance of single spiking neuron. Moreover, we extend the proposed method to multilayer SNNs with spike temporal-based error backpropagation. In the proposed multilayer learning algorithm, information is encoded in the relative timing of individual neuronal spikes, and learning is performed based on the exact derivatives of the postsynaptic spike times with respect to presynaptic spike times. Experimental results on both synthetic and realistic datasets show significant improvements in learning efficiency and accuracy over the existing spike temporal-based learning algorithms. We also evaluate the proposed learning method in an SNN-based multimodal computational model for audiovisual pattern recognition, and it achieves better performance compared with its counterparts.
Collapse
|
8
|
Sun P, Chua Y, Devos P, Botteldooren D. Learnable axonal delay in spiking neural networks improves spoken word recognition. Front Neurosci 2023; 17:1275944. [PMID: 38027508 PMCID: PMC10665570 DOI: 10.3389/fnins.2023.1275944] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Accepted: 10/23/2023] [Indexed: 12/01/2023] Open
Abstract
Spiking neural networks (SNNs), which are composed of biologically plausible spiking neurons, and combined with bio-physically realistic auditory periphery models, offer a means to explore and understand human auditory processing-especially in tasks where precise timing is essential. However, because of the inherent temporal complexity in spike sequences, the performance of SNNs has remained less competitive compared to artificial neural networks (ANNs). To tackle this challenge, a fundamental research topic is the configuration of spike-timing and the exploration of more intricate architectures. In this work, we demonstrate a learnable axonal delay combined with local skip-connections yields state-of-the-art performance on challenging benchmarks for spoken word recognition. Additionally, we introduce an auxiliary loss term to further enhance accuracy and stability. Experiments on the neuromorphic speech benchmark datasets, NTIDIDIGITS and SHD, show improvements in performance when incorporating our delay module in comparison to vanilla feedforward SNNs. Specifically, with the integration of our delay module, the performance on NTIDIDIGITS and SHD improves by 14% and 18%, respectively. When paired with local skip-connections and the auxiliary loss, our approach surpasses both recurrent and convolutional neural networks, yet uses 10 × fewer parameters for NTIDIDIGITS and 7 × fewer for SHD.
Collapse
Affiliation(s)
- Pengfei Sun
- Department of Information Technology, WAVES Research Group, Ghent University, Ghent, Belgium
| | - Yansong Chua
- Neuromorphic Computing Laboratory, China Nanhu Academy of Electronics and Information Technology, Jiaxing, China
| | - Paul Devos
- Department of Information Technology, WAVES Research Group, Ghent University, Ghent, Belgium
| | - Dick Botteldooren
- Department of Information Technology, WAVES Research Group, Ghent University, Ghent, Belgium
| |
Collapse
|
9
|
Pan W, Zhao F, Zeng Y, Han B. Adaptive structure evolution and biologically plausible synaptic plasticity for recurrent spiking neural networks. Sci Rep 2023; 13:16924. [PMID: 37805632 PMCID: PMC10560283 DOI: 10.1038/s41598-023-43488-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Accepted: 09/25/2023] [Indexed: 10/09/2023] Open
Abstract
The architecture design and multi-scale learning principles of the human brain that evolved over hundreds of millions of years are crucial to realizing human-like intelligence. Spiking neural network based Liquid State Machine (LSM) serves as a suitable architecture to study brain-inspired intelligence because of its brain-inspired structure and the potential for integrating multiple biological principles. Existing researches on LSM focus on different certain perspectives, including high-dimensional encoding or optimization of the liquid layer, network architecture search, and application to hardware devices. There is still a lack of in-depth inspiration from the learning and structural evolution mechanism of the brain. Considering these limitations, this paper presents a novel LSM learning model that integrates adaptive structural evolution and multi-scale biological learning rules. For structural evolution, an adaptive evolvable LSM model is developed to optimize the neural architecture design of liquid layer with separation property. For brain-inspired learning of LSM, we propose a dopamine-modulated Bienenstock-Cooper-Munros (DA-BCM) method that incorporates global long-term dopamine regulation and local trace-based BCM synaptic plasticity. Comparative experimental results on different decision-making tasks show that introducing structural evolution of the liquid layer, and the DA-BCM regulation of the liquid layer and the readout layer could improve the decision-making ability of LSM and flexibly adapt to rule reversal. This work is committed to exploring how evolution can help to design more appropriate network architectures and how multi-scale neuroplasticity principles coordinated to enable the optimization and learning of LSMs for relatively complex decision-making tasks.
Collapse
Affiliation(s)
- Wenxuan Pan
- Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Feifei Zhao
- Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Yi Zeng
- Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China.
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
- School of Future Technology, University of Chinese Academy of Sciences, Beijing, China.
- Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China.
| | - Bing Han
- Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
10
|
Guo Y, Huang X, Ma Z. Direct learning-based deep spiking neural networks: a review. Front Neurosci 2023; 17:1209795. [PMID: 37397460 PMCID: PMC10313197 DOI: 10.3389/fnins.2023.1209795] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2023] [Accepted: 06/01/2023] [Indexed: 07/04/2023] Open
Abstract
The spiking neural network (SNN), as a promising brain-inspired computational model with binary spike information transmission mechanism, rich spatially-temporal dynamics, and event-driven characteristics, has received extensive attention. However, its intricately discontinuous spike mechanism brings difficulty to the optimization of the deep SNN. Since the surrogate gradient method can greatly mitigate the optimization difficulty and shows great potential in directly training deep SNNs, a variety of direct learning-based deep SNN works have been proposed and achieved satisfying progress in recent years. In this paper, we present a comprehensive survey of these direct learning-based deep SNN works, mainly categorized into accuracy improvement methods, efficiency improvement methods, and temporal dynamics utilization methods. In addition, we also divide these categorizations into finer granularities further to better organize and introduce them. Finally, the challenges and trends that may be faced in future research are prospected.
Collapse
Affiliation(s)
- Yufei Guo
- Intelligent Science & Technology Academy of CASIC, Beijing, China
- Scientific Research Laboratory of Aerospace Intelligent Systems and Technology, Beijing, China
| | - Xuhui Huang
- Intelligent Science & Technology Academy of CASIC, Beijing, China
- Scientific Research Laboratory of Aerospace Intelligent Systems and Technology, Beijing, China
| | - Zhe Ma
- Intelligent Science & Technology Academy of CASIC, Beijing, China
- Scientific Research Laboratory of Aerospace Intelligent Systems and Technology, Beijing, China
| |
Collapse
|
11
|
Lin R, Dai B, Zhao Y, Chen G, Lu H. Constrain Bias Addition to Train Low-Latency Spiking Neural Networks. Brain Sci 2023; 13:brainsci13020319. [PMID: 36831862 PMCID: PMC9954654 DOI: 10.3390/brainsci13020319] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2022] [Revised: 01/19/2023] [Accepted: 02/10/2023] [Indexed: 02/16/2023] Open
Abstract
In recent years, a third-generation neural network, namely, spiking neural network, has received plethora of attention in the broad areas of Machine learning and Artificial Intelligence. In this paper, a novel differential-based encoding method is proposed and new spike-based learning rules for backpropagation is derived by constraining the addition of bias voltage in spiking neurons. The proposed differential encoding method can effectively exploit the correlation between the data and improve the performance of the proposed model, and the new learning rule can take complete advantage of the modulation properties of bias on the spike firing threshold. We experiment with the proposed model on the environmental sound dataset RWCP and the image dataset MNIST and Fashion-MNIST, respectively, and assign various conditions to test the learning ability and robustness of the proposed model. The experimental results demonstrate that the proposed model achieves near-optimal results with a smaller time step by maintaining the highest accuracy and robustness with less training data. Among them, in MNIST dataset, compared with the original spiking neural network with the same network structure, we achieved a 0.39% accuracy improvement.
Collapse
Affiliation(s)
- Ranxi Lin
- Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China
- University of Chinese Academy of Sciences, Beijing 100089, China
| | - Benzhe Dai
- Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China
- University of Chinese Academy of Sciences, Beijing 100089, China
| | - Yingkai Zhao
- Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China
- University of Chinese Academy of Sciences, Beijing 100089, China
| | - Gang Chen
- Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China
- Semiconductor Neural Network Intelligent Perception and Computing Technology Beijing Key Laboratory, Beijing 100083, China
- Correspondence:
| | - Huaxiang Lu
- Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China
- University of Chinese Academy of Sciences, Beijing 100089, China
- Semiconductor Neural Network Intelligent Perception and Computing Technology Beijing Key Laboratory, Beijing 100083, China
- Collage of Microelectronics, University of Chinese Academy of Sciences, Beijing 100049, China
- Materials and Optoelectronics Research Center, University of Chinese Academy of Sciences, Beijing 200031, China
| |
Collapse
|
12
|
Cheng X, Zhang T, Jia S, Xu B. Meta neurons improve spiking neural networks for efficient spatio-temporal learning. Neurocomputing 2023. [DOI: 10.1016/j.neucom.2023.02.029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/18/2023]
|
13
|
Yi Z, Lian J, Liu Q, Zhu H, Liang D, Liu J. Learning Rules in Spiking Neural Networks: A Survey. Neurocomputing 2023. [DOI: 10.1016/j.neucom.2023.02.026] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/18/2023]
|
14
|
Zhang T, Jia S, Cheng X, Xu B. Tuning Convolutional Spiking Neural Network With Biologically Plausible Reward Propagation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2022; 33:7621-7631. [PMID: 34125691 DOI: 10.1109/tnnls.2021.3085966] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Spiking neural networks (SNNs) contain more biologically realistic structures and biologically inspired learning principles than those in standard artificial neural networks (ANNs). SNNs are considered the third generation of ANNs, powerful on the robust computation with a low computational cost. The neurons in SNNs are nondifferential, containing decayed historical states and generating event-based spikes after their states reaching the firing threshold. These dynamic characteristics of SNNs make it difficult to be directly trained with the standard backpropagation (BP), which is also considered not biologically plausible. In this article, a biologically plausible reward propagation (BRP) algorithm is proposed and applied to the SNN architecture with both spiking-convolution (with both 1-D and 2-D convolutional kernels) and full-connection layers. Unlike the standard BP that propagates error signals from postsynaptic to presynaptic neurons layer by layer, the BRP propagates target labels instead of errors directly from the output layer to all prehidden layers. This effort is more consistent with the top-down reward-guiding learning in cortical columns of the neocortex. Synaptic modifications with only local gradient differences are induced with pseudo-BP that might also be replaced with the spike-timing-dependent plasticity (STDP). The performance of the proposed BRP-SNN is further verified on the spatial (including MNIST and Cifar-10) and temporal (including TIDigits and DvsGesture) tasks, where the SNN using BRP has reached a similar accuracy compared to other state-of-the-art (SOTA) BP-based SNNs and saved 50% more computational cost than ANNs. We think that the introduction of biologically plausible learning rules to the training procedure of biologically realistic SNNs will give us more hints and inspiration toward a better understanding of the biological system's intelligent nature.
Collapse
|
15
|
Hu L, Liao X. Voltage slope guided learning in spiking neural networks. Front Neurosci 2022; 16:1012964. [PMID: 36440266 PMCID: PMC9685168 DOI: 10.3389/fnins.2022.1012964] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2022] [Accepted: 10/25/2022] [Indexed: 04/19/2024] Open
Abstract
A thorny problem in machine learning is how to extract useful clues related to delayed feedback signals from the clutter of input activity, known as the temporal credit-assignment problem. The aggregate-label learning algorithms make an explicit representation of this problem by training spiking neurons to assign the aggregate feedback signal to potentially effective clues. However, earlier aggregate-label learning algorithms suffered from inefficiencies due to the large amount of computation, while recent algorithms that have solved this problem may fail to learn due to the inability to find adjustment points. Therefore, we propose a membrane voltage slope guided algorithm (VSG) to further cope with this limitation. Direct dependence on the membrane voltage when finding the key point of weight adjustment makes VSG avoid intensive calculation, but more importantly, the membrane voltage that always exists makes it impossible to lose the adjustment point. Experimental results show that the proposed algorithm can correlate delayed feedback signals with the effective clues embedded in background spiking activity, and also achieves excellent performance on real medical classification datasets and speech classification datasets. The superior performance makes it a meaningful reference for aggregate-label learning on spiking neural networks.
Collapse
Affiliation(s)
- Lvhui Hu
- School of Intelligent Medicine, Chengdu University of Traditional Chinese Medicine, Chengdu, China
| | - Xin Liao
- Information Center, Hospital of Chengdu University of Traditional Chinese Medicine, Chengdu, China
| |
Collapse
|
16
|
Wang R, Shi T, Zhang X, Wei J, Lu J, Zhu J, Wu Z, Liu Q, Liu M. Implementing in-situ self-organizing maps with memristor crossbar arrays for data mining and optimization. Nat Commun 2022; 13:2289. [PMID: 35484107 PMCID: PMC9051161 DOI: 10.1038/s41467-022-29411-4] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Accepted: 03/14/2022] [Indexed: 11/30/2022] Open
Abstract
A self-organizing map (SOM) is a powerful unsupervised learning neural network for analyzing high-dimensional data in various applications. However, hardware implementation of SOM is challenging because of the complexity in calculating the similarities and determining neighborhoods. We experimentally demonstrated a memristor-based SOM based on Ta/TaOx/Pt 1T1R chips for the first time, which has advantages in computing speed, throughput, and energy efficiency compared with the CMOS digital counterpart, by utilizing the topological structure of the array and physical laws for computing without complicated circuits. We employed additional rows in the crossbar arrays and identified the best matching units by directly calculating the similarities between the input vectors and the weight matrix in the hardware. Using the memristor-based SOM, we demonstrated data clustering, image processing and solved the traveling salesman problem with much-improved energy efficiency and computing throughput. The physical implementation of SOM in memristor crossbar arrays extends the capability of memristor-based neuromorphic computing systems in machine learning and artificial intelligence. Self-organizing maps are data mining tools for unsupervised learning algorithms dealing with big data problems. The authors experimentally demonstrate a memristor-based self-organizing map that is more efficient in computing speed and energy consumption for data clustering, image processing and solving optimization problems.
Collapse
Affiliation(s)
- Rui Wang
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China.,The Frontier institute of Chip and System, Fudan University, 200433, Shanghai, PR China.,University of Chinese Academy of Sciences, 100049, Beijing, PR China
| | - Tuo Shi
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China. .,University of Chinese Academy of Sciences, 100049, Beijing, PR China. .,Institute of Intelligent Computing, Zhejiang Laboratory, 311122, Hangzhou, PR China.
| | - Xumeng Zhang
- The Frontier institute of Chip and System, Fudan University, 200433, Shanghai, PR China
| | - Jinsong Wei
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China.,Institute of Intelligent Computing, Zhejiang Laboratory, 311122, Hangzhou, PR China
| | - Jian Lu
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China.,Institute of Intelligent Computing, Zhejiang Laboratory, 311122, Hangzhou, PR China
| | - Jiaxue Zhu
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China.,University of Chinese Academy of Sciences, 100049, Beijing, PR China
| | - Zuheng Wu
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China.,University of Chinese Academy of Sciences, 100049, Beijing, PR China
| | - Qi Liu
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China. .,The Frontier institute of Chip and System, Fudan University, 200433, Shanghai, PR China. .,University of Chinese Academy of Sciences, 100049, Beijing, PR China.
| | - Ming Liu
- The Key Laboratory of Microelectronics Devices and Integrated Technology, Institute of Microelectronics Chinese Academy of Sciences, 100029, Beijing, PR China.,The Frontier institute of Chip and System, Fudan University, 200433, Shanghai, PR China.,University of Chinese Academy of Sciences, 100049, Beijing, PR China
| |
Collapse
|
17
|
Mo L, Wang G, Long E, Zhuo M. ALSA: Associative Learning Based Supervised Learning Algorithm for SNN. Front Neurosci 2022; 16:838832. [PMID: 35431777 PMCID: PMC9008323 DOI: 10.3389/fnins.2022.838832] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2021] [Accepted: 03/07/2022] [Indexed: 11/13/2022] Open
Abstract
Spiking neural network (SNN) is considered to be the brain-like model that best conforms to the biological mechanism of the brain. Due to the non-differentiability of the spike, the training method of SNNs is still incomplete. This paper proposes a supervised learning method for SNNs based on associative learning: ALSA. The method is based on the associative learning mechanism, and its realization is similar to the animal conditioned reflex process, with strong physiological plausibility and rationality. This method uses improved spike-timing-dependent plasticity (STDP) rules, combined with a teacher layer to induct spikes of neurons, to strengthen synaptic connections between input spike patterns and specified output neurons, and weaken synaptic connections between unrelated patterns and unrelated output neurons. Based on ALSA, this paper also completed the supervised learning classification tasks of the IRIS dataset and the MNIST dataset, and achieved 95.7 and 91.58% recognition accuracy, respectively, which fully proves that ALSA is a feasible SNNs supervised learning method. The innovation of this paper is to establish a biological plausible supervised learning method for SNNs, which is based on the STDP learning rules and the associative learning mechanism that exists widely in animal training.
Collapse
|
18
|
Yu Q, Song S, Ma C, Pan L, Tan KC. Synaptic Learning With Augmented Spikes. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2022; 33:1134-1146. [PMID: 33471768 DOI: 10.1109/tnnls.2020.3040969] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Traditional neuron models use analog values for information representation and computation, while all-or-nothing spikes are employed in the spiking ones. With a more brain-like processing paradigm, spiking neurons are more promising for improvements in efficiency and computational capability. They extend the computation of traditional neurons with an additional dimension of time carried by all-or-nothing spikes. Could one benefit from both the accuracy of analog values and the time-processing capability of spikes? In this article, we introduce a concept of augmented spikes to carry complementary information with spike coefficients in addition to spike latencies. New augmented spiking neuron model and synaptic learning rules are proposed to process and learn patterns of augmented spikes. We provide systematic insights into the properties and characteristics of our methods, including classification of augmented spike patterns, learning capacity, construction of causality, feature detection, robustness, and applicability to practical tasks, such as acoustic and visual pattern recognition. Our augmented approaches show several advanced learning properties and reliably outperform the baseline ones that use typical all-or-nothing spikes. Our approaches significantly improve the accuracies of a temporal-based approach on sound and MNIST recognition tasks to 99.38% and 97.90%, respectively, highlighting the effectiveness and potential merits of our methods. More importantly, our augmented approaches are versatile and can be easily generalized to other spike-based systems, contributing to a potential development for them, including neuromorphic computing.
Collapse
|
19
|
Deng B, Fan Y, Wang J, Yang S. Reconstruction of a Fully Paralleled Auditory Spiking Neural Network and FPGA Implementation. IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS 2021; 15:1320-1331. [PMID: 34699367 DOI: 10.1109/tbcas.2021.3122549] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
This paper presents a field-programmable gate array (FPGA) implementation of an auditory system, which is biologically inspired and has the advantages of robustness and anti-noise ability. We propose an FPGA implementation of an eleven-channel hierarchical spiking neuron network (SNN) model, which has a sparsely connected architecture with low power consumption. According to the mechanism of the auditory pathway in human brain, spiking trains generated by the cochlea are analyzed in the hierarchical SNN, and the specific word can be identified by a Bayesian classifier. Modified leaky integrate-and-fire (LIF) model is used to realize the hierarchical SNN, which achieves both high efficiency and low hardware consumption. The hierarchical SNN implemented on FPGA enables the auditory system to be operated at high speed and can be interfaced and applied with external machines and sensors. A set of speech from different speakers mixed with noise are used as input to test the performance our system, and the experimental results show that the system can classify words in a biologically plausible way with the presence of noise. The method of our system is flexible and the system can be modified into desirable scale. These confirm that the proposed biologically plausible auditory system provides a better method for on-chip speech recognition. Compare to the state-of-the-art, our auditory system achieves a higher speed with a maximum frequency of 65.03 MHz and a lower energy consumption of 276.83 μJ for a single operation. It can be applied in the field of brain-computer interface and intelligent robots.
Collapse
|
20
|
Wu J, Liu Q, Zhang M, Pan Z, Li H, Tan KC. HuRAI: A brain-inspired computational model for human-robot auditory interface. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.08.115] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
21
|
Attention Augmented Convolutional Neural Network for acoustics based machine state estimation. Appl Soft Comput 2021. [DOI: 10.1016/j.asoc.2021.107630] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
22
|
Lan Y, Wang X, Wang Y. Spatio-Temporal Sequential Memory Model With Mini-Column Neural Network. Front Neurosci 2021; 15:650430. [PMID: 34121986 PMCID: PMC8195288 DOI: 10.3389/fnins.2021.650430] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2021] [Accepted: 03/15/2021] [Indexed: 11/13/2022] Open
Abstract
Memory is an intricate process involving various faculties of the brain and is a central component in human cognition. However, the exact mechanism that brings about memory in our brain remains elusive and the performance of the existing memory models is not satisfactory. To overcome these problems, this paper puts forward a brain-inspired spatio-temporal sequential memory model based on spiking neural networks (SNNs). Inspired by the structure of the neocortex, the proposed model is structured by many mini-columns composed of biological spiking neurons. Each mini-column represents one memory item, and the firing of different spiking neurons in the mini-column depends on the context of the previous inputs. The Spike-Timing-Dependant Plasticity (STDP) is used to update the connections between excitatory neurons and formulates association between two memory items. In addition, the inhibitory neurons are employed to prevent incorrect prediction, which contributes to improving the retrieval accuracy. Experimental results demonstrate that the proposed model can effectively store a huge number of data and accurately retrieve them when sufficient context is provided. This work not only provides a new memory model but also suggests how memory could be formulated with excitatory/inhibitory neurons, spike-based encoding, and mini-column structure.
Collapse
Affiliation(s)
- Yawen Lan
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China.,School of Information Engineering, Southwest University of Science and Technology, Mianyang, China
| | - Xiaobin Wang
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Yuchen Wang
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
| |
Collapse
|
23
|
Jia S, Zhang T, Cheng X, Liu H, Xu B. Neuronal-Plasticity and Reward-Propagation Improved Recurrent Spiking Neural Networks. Front Neurosci 2021; 15:654786. [PMID: 33776644 PMCID: PMC7994752 DOI: 10.3389/fnins.2021.654786] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2021] [Accepted: 02/12/2021] [Indexed: 11/13/2022] Open
Abstract
Different types of dynamics and plasticity principles found through natural neural networks have been well-applied on Spiking neural networks (SNNs) because of their biologically-plausible efficient and robust computations compared to their counterpart deep neural networks (DNNs). Here, we further propose a special Neuronal-plasticity and Reward-propagation improved Recurrent SNN (NRR-SNN). The historically-related adaptive threshold with two channels is highlighted as important neuronal plasticity for increasing the neuronal dynamics, and then global labels instead of errors are used as a reward for the paralleling gradient propagation. Besides, a recurrent loop with proper sparseness is designed for robust computation. Higher accuracy and stronger robust computation are achieved on two sequential datasets (i.e., TIDigits and TIMIT datasets), which to some extent, shows the power of the proposed NRR-SNN with biologically-plausible improvements.
Collapse
Affiliation(s)
- Shuncheng Jia
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing, China
| | - Tielin Zhang
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing, China
| | - Xiang Cheng
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing, China
| | - Hongxing Liu
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China
- Faculty of Information Technology, Beijing University of Technology, Beijing, China
| | - Bo Xu
- Research Center for Brain-Inspired Intelligence, Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing, China
- Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China
| |
Collapse
|
24
|
Zhang Y, Qu H, Luo X, Chen Y, Wang Y, Zhang M, Li Z. A new recursive least squares-based learning algorithm for spiking neurons. Neural Netw 2021; 138:110-125. [PMID: 33636484 DOI: 10.1016/j.neunet.2021.01.016] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2019] [Revised: 12/15/2020] [Accepted: 01/18/2021] [Indexed: 10/22/2022]
Abstract
Spiking neural networks (SNNs) are regarded as effective models for processing spatio-temporal information. However, their inherent complexity of temporal coding makes it an arduous task to put forward an effective supervised learning algorithm, which still puzzles researchers in this area. In this paper, we propose a Recursive Least Squares-Based Learning Rule (RLSBLR) for SNN to generate the desired spatio-temporal spike train. During the learning process of our method, the weight update is driven by the cost function defined by the difference between the membrane potential and the firing threshold. The amount of weight modification depends not only on the impact of the current error function, but also on the previous error functions which are evaluated by current weights. In order to improve the learning performance, we integrate a modified synaptic delay learning to the proposed RLSBLR. We conduct experiments in different settings, such as spiking lengths, number of inputs, firing rates, noises and learning parameters, to thoroughly investigate the performance of this learning algorithm. The proposed RLSBLR is compared with competitive algorithms of Perceptron-Based Spiking Neuron Learning Rule (PBSNLR) and Remote Supervised Method (ReSuMe). Experimental results demonstrate that the proposed RLSBLR can achieve higher learning accuracy, higher efficiency and better robustness against different types of noise. In addition, we apply the proposed RLSBLR to open source database TIDIGITS, and the results show that our algorithm has a good practical application performance.
Collapse
Affiliation(s)
- Yun Zhang
- Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, PR China
| | - Hong Qu
- Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, PR China.
| | - Xiaoling Luo
- Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, PR China
| | - Yi Chen
- Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, PR China
| | - Yuchen Wang
- Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, PR China
| | - Malu Zhang
- Department of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610054, PR China
| | - Zefang Li
- China Coal Research Institute, Beijing 100013, PR China
| |
Collapse
|
25
|
Bensimon M, Greenberg S, Haiut M. Using a Low-Power Spiking Continuous Time Neuron (SCTN) for Sound Signal Processing. SENSORS 2021; 21:s21041065. [PMID: 33557214 PMCID: PMC7913968 DOI: 10.3390/s21041065] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/30/2020] [Revised: 01/23/2021] [Accepted: 02/01/2021] [Indexed: 11/16/2022]
Abstract
This work presents a new approach based on a spiking neural network for sound preprocessing and classification. The proposed approach is biologically inspired by the biological neuron’s characteristic using spiking neurons, and Spike-Timing-Dependent Plasticity (STDP)-based learning rule. We propose a biologically plausible sound classification framework that uses a Spiking Neural Network (SNN) for detecting the embedded frequencies contained within an acoustic signal. This work also demonstrates an efficient hardware implementation of the SNN network based on the low-power Spike Continuous Time Neuron (SCTN). The proposed sound classification framework suggests direct Pulse Density Modulation (PDM) interfacing of the acoustic sensor with the SCTN-based network avoiding the usage of costly digital-to-analog conversions. This paper presents a new connectivity approach applied to Spiking Neuron (SN)-based neural networks. We suggest considering the SCTN neuron as a basic building block in the design of programmable analog electronics circuits. Usually, a neuron is used as a repeated modular element in any neural network structure, and the connectivity between the neurons located at different layers is well defined. Thus, generating a modular Neural Network structure composed of several layers with full or partial connectivity. The proposed approach suggests controlling the behavior of the spiking neurons, and applying smart connectivity to enable the design of simple analog circuits based on SNN. Unlike existing NN-based solutions for which the preprocessing phase is carried out using analog circuits and analog-to-digital conversion, we suggest integrating the preprocessing phase into the network. This approach allows referring to the basic SCTN as an analog module enabling the design of simple analog circuits based on SNN with unique inter-connections between the neurons. The efficiency of the proposed approach is demonstrated by implementing SCTN-based resonators for sound feature extraction and classification. The proposed SCTN-based sound classification approach demonstrates a classification accuracy of 98.73% using the Real-World Computing Partnership (RWCP) database.
Collapse
Affiliation(s)
- Moshe Bensimon
- School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beersheba 8400711, Israel;
| | - Shlomo Greenberg
- School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beersheba 8400711, Israel;
- Correspondence:
| | | |
Collapse
|
26
|
Yu Q, Yao Y, Wang L, Tang H, Dang J, Tan KC. Robust Environmental Sound Recognition With Sparse Key-Point Encoding and Efficient Multispike Learning. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2021; 32:625-638. [PMID: 32203038 DOI: 10.1109/tnnls.2020.2978764] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
The capability for environmental sound recognition (ESR) can determine the fitness of individuals in a way to avoid dangers or pursue opportunities when critical sound events occur. It still remains mysterious about the fundamental principles of biological systems that result in such a remarkable ability. Additionally, the practical importance of ESR has attracted an increasing amount of research attention, but the chaotic and nonstationary difficulties continue to make it a challenging task. In this article, we propose a spike-based framework from a more brain-like perspective for the ESR task. Our framework is a unifying system with consistent integration of three major functional parts which are sparse encoding, efficient learning, and robust readout. We first introduce a simple sparse encoding, where key points are used for feature representation, and demonstrate its generalization to both spike- and nonspike-based systems. Then, we evaluate the learning properties of different learning rules in detail with our contributions being added for improvements. Our results highlight the advantages of multispike learning, providing a selection reference for various spike-based developments. Finally, we combine the multispike readout with the other parts to form a system for ESR. Experimental results show that our framework performs the best as compared to other baseline approaches. In addition, we show that our spike-based framework has several advantageous characteristics including early decision making, small dataset acquiring, and ongoing dynamic processing. Our framework is the first attempt to apply the multispike characteristic of nervous neurons to ESR. The outstanding performance of our approach would potentially contribute to draw more research efforts to push the boundaries of spike-based paradigm to a new horizon.
Collapse
|
27
|
Zhang M, Wu J, Belatreche A, Pan Z, Xie X, Chua Y, Li G, Qu H, Li H. Supervised learning in spiking neural networks with synaptic delay-weight plasticity. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2020.03.079] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
|
28
|
Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences. Neural Netw 2020; 132:108-120. [PMID: 32866745 DOI: 10.1016/j.neunet.2020.08.001] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2020] [Revised: 06/13/2020] [Accepted: 08/03/2020] [Indexed: 01/16/2023]
Abstract
Neuromorphic data, recording frameless spike events, have attracted considerable attention for the spatiotemporal information components and the event-driven processing fashion. Spiking neural networks (SNNs) represent a family of event-driven models with spatiotemporal dynamics for neuromorphic computing, which are widely benchmarked on neuromorphic data. Interestingly, researchers in the machine learning community can argue that recurrent (artificial) neural networks (RNNs) also have the capability to extract spatiotemporal features although they are not event-driven. Thus, the question of "what will happen if we benchmark these two kinds of models together on neuromorphic data" comes out but remains unclear. In this work, we make a systematic study to compare SNNs and RNNs on neuromorphic data, taking the vision datasets as a case study. First, we identify the similarities and differences between SNNs and RNNs (including the vanilla RNNs and LSTM) from the modeling and learning perspectives. To improve comparability and fairness, we unify the supervised learning algorithm based on backpropagation through time (BPTT), the loss function exploiting the outputs at all timesteps, the network structure with stacked fully-connected or convolutional layers, and the hyper-parameters during training. Especially, given the mainstream loss function used in RNNs, we modify it inspired by the rate coding scheme to approach that of SNNs. Furthermore, we tune the temporal resolution of datasets to test model robustness and generalization. At last, a series of contrast experiments are conducted on two types of neuromorphic datasets: DVS-converted (N-MNIST) and DVS-captured (DVS Gesture). Extensive insights regarding recognition accuracy, feature extraction, temporal resolution and contrast, learning generalization, computational complexity and parameter volume are provided, which are beneficial for the model selection on different workloads and even for the invention of novel neural models in the future.
Collapse
|
29
|
Wu J, Yılmaz E, Zhang M, Li H, Tan KC. Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition. Front Neurosci 2020; 14:199. [PMID: 32256308 PMCID: PMC7090229 DOI: 10.3389/fnins.2020.00199] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2019] [Accepted: 02/24/2020] [Indexed: 11/13/2022] Open
Abstract
Artificial neural networks (ANN) have become the mainstream acoustic modeling technique for large vocabulary automatic speech recognition (ASR). A conventional ANN features a multi-layer architecture that requires massive amounts of computation. The brain-inspired spiking neural networks (SNN) closely mimic the biological neural networks and can operate on low-power neuromorphic hardware with spike-based computation. Motivated by their unprecedented energy-efficiency and rapid information processing capability, we explore the use of SNNs for speech recognition. In this work, we use SNNs for acoustic modeling and evaluate their performance on several large vocabulary recognition scenarios. The experimental results demonstrate competitive ASR accuracies to their ANN counterparts, while require only 10 algorithmic time steps and as low as 0.68 times total synaptic operations to classify each audio frame. Integrating the algorithmic power of deep SNNs with energy-efficient neuromorphic hardware, therefore, offer an attractive solution for ASR applications running locally on mobile and embedded devices.
Collapse
Affiliation(s)
- Jibin Wu
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
| | - Emre Yılmaz
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
| | - Malu Zhang
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
| | - Haizhou Li
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
- Faculty for Computer Science and Mathematics, University of Bremen, Bremen, Germany
| | - Kay Chen Tan
- Department of Computer Science, City University of Hong Kong, Kowloon Tong, Hong Kong
| |
Collapse
|
30
|
Pan Z, Chua Y, Wu J, Zhang M, Li H, Ambikairajah E. An Efficient and Perceptually Motivated Auditory Neural Encoding and Decoding Algorithm for Spiking Neural Networks. Front Neurosci 2020; 13:1420. [PMID: 32038132 PMCID: PMC6987407 DOI: 10.3389/fnins.2019.01420] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2019] [Accepted: 12/16/2019] [Indexed: 12/11/2022] Open
Abstract
The auditory front-end is an integral part of a spiking neural network (SNN) when performing auditory cognitive tasks. It encodes the temporal dynamic stimulus, such as speech and audio, into an efficient, effective and reconstructable spike pattern to facilitate the subsequent processing. However, most of the auditory front-ends in current studies have not made use of recent findings in psychoacoustics and physiology concerning human listening. In this paper, we propose a neural encoding and decoding scheme that is optimized for audio processing. The neural encoding scheme, that we call Biologically plausible Auditory Encoding (BAE), emulates the functions of the perceptual components of the human auditory system, that include the cochlear filter bank, the inner hair cells, auditory masking effects from psychoacoustic models, and the spike neural encoding by the auditory nerve. We evaluate the perceptual quality of the BAE scheme using PESQ; the performance of the BAE based on sound classification and speech recognition experiments. Finally, we also built and published two spike-version of speech datasets: the Spike-TIDIGITS and the Spike-TIMIT, for researchers to use and benchmarking of future SNN research.
Collapse
Affiliation(s)
- Zihan Pan
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
| | - Yansong Chua
- Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore, Singapore
| | - Jibin Wu
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
| | - Malu Zhang
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
| | - Haizhou Li
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
| | - Eliathamby Ambikairajah
- School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, NSW, Australia
| |
Collapse
|
31
|
Luo X, Qu H, Zhang Y, Chen Y. First Error-Based Supervised Learning Algorithm for Spiking Neural Networks. Front Neurosci 2019; 13:559. [PMID: 31244594 PMCID: PMC6563788 DOI: 10.3389/fnins.2019.00559] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2019] [Accepted: 05/15/2019] [Indexed: 11/13/2022] Open
Abstract
Neural circuits respond to multiple sensory stimuli by firing precisely timed spikes. Inspired by this phenomenon, the spike timing-based spiking neural networks (SNNs) are proposed to process and memorize the spatiotemporal spike patterns. However, the response speed and accuracy of the existing learning algorithms of SNNs are still lacking compared to the human brain. To further improve the performance of learning precisely timed spikes, we propose a new weight updating mechanism which always adjusts the synaptic weights at the first wrong output spike time. The proposed learning algorithm can accurately adjust the synaptic weights that contribute to the membrane potential of desired and non-desired firing time. Experimental results demonstrate that the proposed algorithm shows higher accuracy, better robustness, and less computational resources compared with the remote supervised method (ReSuMe) and the spike pattern association neuron (SPAN), which are classic sequence learning algorithms. In addition, the SNN-based computational model equipped with the proposed learning method achieves better recognition results in speech recognition task compared with other bio-inspired baseline systems.
Collapse
Affiliation(s)
- Xiaoling Luo
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Hong Qu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Yun Zhang
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Yi Chen
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
| |
Collapse
|
32
|
Wijesinghe P, Srinivasan G, Panda P, Roy K. Analysis of Liquid Ensembles for Enhancing the Performance and Accuracy of Liquid State Machines. Front Neurosci 2019; 13:504. [PMID: 31191219 PMCID: PMC6546930 DOI: 10.3389/fnins.2019.00504] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2019] [Accepted: 05/01/2019] [Indexed: 11/13/2022] Open
Abstract
Liquid state machine (LSM), a bio-inspired computing model consisting of the input sparsely connected to a randomly interlinked reservoir (or liquid) of spiking neurons followed by a readout layer, finds utility in a range of applications varying from robot control and sequence generation to action, speech, and image recognition. LSMs stand out among other Recurrent Neural Network (RNN) architectures due to their simplistic structure and lower training complexity. Plethora of recent efforts have been focused toward mimicking certain characteristics of biological systems to enhance the performance of modern artificial neural networks. It has been shown that biological neurons are more likely to be connected to other neurons in the close proximity, and tend to be disconnected as the neurons are spatially far apart. Inspired by this, we propose a group of locally connected neuron reservoirs, or an ensemble of liquids approach, for LSMs. We analyze how the segmentation of a single large liquid to create an ensemble of multiple smaller liquids affects the latency and accuracy of an LSM. In our analysis, we quantify the ability of the proposed ensemble approach to provide an improved representation of the input using the Separation Property (SP) and Approximation Property (AP). Our results illustrate that the ensemble approach enhances class discrimination (quantified as the ratio between the SP and AP), leading to better accuracy in speech and image recognition tasks, when compared to a single large liquid. Furthermore, we obtain performance benefits in terms of improved inference time and reduced memory requirements, due to lowered number of connections and the freedom to parallelize the liquid evaluation process.
Collapse
Affiliation(s)
- Parami Wijesinghe
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, United States
| | | | | | | |
Collapse
|
33
|
Odor Recognition with a Spiking Neural Network for Bioelectronic Nose. SENSORS 2019; 19:s19050993. [PMID: 30813574 PMCID: PMC6427646 DOI: 10.3390/s19050993] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/05/2019] [Revised: 02/18/2019] [Accepted: 02/20/2019] [Indexed: 12/02/2022]
Abstract
Electronic noses recognize odors using sensor arrays, and usually face difficulties for odor complicacy, while animals have their own biological sensory capabilities for various types of odors. By implanting electrodes into the olfactory bulb of mammalian animals, odors may be recognized by decoding the recorded neural signals, in order to construct a bioelectronic nose. This paper proposes a spiking neural network (SNN)-based odor recognition method from spike trains recorded by the implanted electrode array. The proposed SNN-based approach exploits rich timing information well in precise time points of spikes. To alleviate the overfitting problem, we design a new SNN learning method with a voltage-based regulation strategy. Experiments are carried out using spike train signals recorded from the main olfactory bulb in rats. Results show that our SNN-based approach achieves the state-of-the-art performance, compared with other methods. With the proposed voltage regulation strategy, it achieves about 15% improvement compared with a classical SNN model.
Collapse
|