1
|
Jiang C, Huang Z, Pedapati T, Chen PY, Sun Y, Gao J. Network properties determine neural network performance. Nat Commun 2024; 15:5718. [PMID: 38977665 PMCID: PMC11231255 DOI: 10.1038/s41467-024-48069-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 04/17/2024] [Indexed: 07/10/2024] Open
Abstract
Machine learning influences numerous aspects of modern society, empowers new technologies, from Alphago to ChatGPT, and increasingly materializes in consumer products such as smartphones and self-driving cars. Despite the vital role and broad applications of artificial neural networks, we lack systematic approaches, such as network science, to understand their underlying mechanism. The difficulty is rooted in many possible model configurations, each with different hyper-parameters and weighted architectures determined by noisy data. We bridge the gap by developing a mathematical framework that maps the neural network's performance to the network characters of the line graph governed by the edge dynamics of stochastic gradient descent differential equations. This framework enables us to derive a neural capacitance metric to universally capture a model's generalization capability on a downstream task and predict model performance using only early training results. The numerical results on 17 pre-trained ImageNet models across five benchmark datasets and one NAS benchmark indicate that our neural capacitance metric is a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods.
Collapse
Affiliation(s)
- Chunheng Jiang
- Network Science and Technology Center, Rensselaer Polytechnic Institute, Troy, NY, USA
- Department of Computer Science, Rensselaer Polytechnic Institute, Troy, NY, USA
| | - Zhenhan Huang
- Network Science and Technology Center, Rensselaer Polytechnic Institute, Troy, NY, USA
- Department of Computer Science, Rensselaer Polytechnic Institute, Troy, NY, USA
| | | | - Pin-Yu Chen
- IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA
| | - Yizhou Sun
- Department of Computer Science, University of California, Los Angeles, CA, USA
| | - Jianxi Gao
- Network Science and Technology Center, Rensselaer Polytechnic Institute, Troy, NY, USA.
- Department of Computer Science, Rensselaer Polytechnic Institute, Troy, NY, USA.
| |
Collapse
|
2
|
Mallinson JB, Steel JK, Heywood ZE, Studholme SJ, Bones PJ, Brown SA. Experimental Demonstration of Reservoir Computing with Self-Assembled Percolating Networks of Nanoparticles. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2024:e2402319. [PMID: 38558447 DOI: 10.1002/adma.202402319] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/13/2024] [Revised: 03/13/2024] [Indexed: 04/04/2024]
Abstract
The complex self-assembled network of neurons and synapses that comprises the biological brain enables natural information processing with remarkable efficiency. Percolating networks of nanoparticles (PNNs) are complex self-assembled nanoscale systems that have been shown to possess many promising brain-like attributes and which are therefore appealing systems for neuromorphic computation. Here experiments are performed that show that PNNs can be utilized as physical reservoirs within a nanoelectronic reservoir computing framework and demonstrate successful computation for several benchmark tasks (chaotic time series prediction, nonlinear transformation, and memory capacity). For each task, relevant literature results are compiled and it is shown that the performance of the PNNs compares favorably to that previously reported from nanoelectronic reservoirs. It is then demonstrated experimentally that PNNs can be used for spoken digit recognition with state-of-the-art accuracy. Finally, a parallel reservoir architecture is emulated, which increases the dimensionality and richness of the reservoir outputs and results in further improvements in performance across all tasks.
Collapse
Affiliation(s)
- Joshua B Mallinson
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Private Bag 4800, Christchurch, 8140, New Zealand
| | - Jamie K Steel
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Private Bag 4800, Christchurch, 8140, New Zealand
| | - Zachary E Heywood
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Private Bag 4800, Christchurch, 8140, New Zealand
- Electrical and Computer Engineering, University of Canterbury, Private Bag 4800, Christchurch, 8140, New Zealand
| | - Sofie J Studholme
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Private Bag 4800, Christchurch, 8140, New Zealand
| | - Philip J Bones
- Electrical and Computer Engineering, University of Canterbury, Private Bag 4800, Christchurch, 8140, New Zealand
| | - Simon A Brown
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Private Bag 4800, Christchurch, 8140, New Zealand
| |
Collapse
|
3
|
Yang L, Wang Z, Wang G, Liang L, Liu M, Wang J. Brain-inspired modular echo state network for EEG-based emotion recognition. Front Neurosci 2024; 18:1305284. [PMID: 38495107 PMCID: PMC10940514 DOI: 10.3389/fnins.2024.1305284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2023] [Accepted: 01/10/2024] [Indexed: 03/19/2024] Open
Abstract
Previous studies have successfully applied a lightweight recurrent neural network (RNN) called Echo State Network (ESN) for EEG-based emotion recognition. These studies use intrinsic plasticity (IP) and synaptic plasticity (SP) to tune the hidden reservoir layer of ESN, yet they require extra training procedures and are often computationally complex. Recent neuroscientific research reveals that the brain is modular, consisting of internally dense and externally sparse subnetworks. Furthermore, it has been proved that this modular topology facilitates information processing efficiency in both biological and artificial neural networks (ANNs). Motivated by these findings, we propose Modular Echo State Network (M-ESN), where the hidden layer of ESN is directly initialized to a more efficient modular structure. In this paper, we first describe our novel implementation method, which enables us to find the optimal module numbers, local and global connectivity. Then, the M-ESN is benchmarked on the DEAP dataset. Lastly, we explain why network modularity improves model performance. We demonstrate that modular organization leads to a more diverse distribution of node degrees, which increases network heterogeneity and subsequently improves classification accuracy. On the emotion arousal, valence, and stress/calm classification tasks, our M-ESN outperforms regular ESN by 5.44, 5.90, and 5.42%, respectively, while this difference when comparing with adaptation rules tuned ESNs are 0.77, 5.49, and 0.95%. Notably, our results are obtained using M-ESN with a much smaller reservoir size and simpler training process.
Collapse
Affiliation(s)
- Liuyi Yang
- College of Big Data and Internet, Shenzhen Technology University, Shenzhen, China
| | - Zhaoze Wang
- School of Engineering and Applied Science, University of Pennsylvania, Pennsylvania, PA, United States
| | - Guoyu Wang
- Department of Auromation, Tiangong University, Tianjin, China
| | - Lixin Liang
- College of Big Data and Internet, Shenzhen Technology University, Shenzhen, China
| | - Meng Liu
- College of Big Data and Internet, Shenzhen Technology University, Shenzhen, China
| | - Junsong Wang
- College of Big Data and Internet, Shenzhen Technology University, Shenzhen, China
| |
Collapse
|
4
|
Wang X, Jin Y, Du W, Wang J. Evolving Dual-Threshold Bienenstock-Cooper-Munro Learning Rules in Echo State Networks. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:1572-1583. [PMID: 35763483 DOI: 10.1109/tnnls.2022.3184004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The strengthening and the weakening of synaptic strength in existing Bienenstock-Cooper-Munro (BCM) learning rule are determined by a long-term potentiation (LTP) sliding modification threshold and the afferent synaptic activities. However, synaptic long-term depression (LTD) even affects low-active synapses during the induction of synaptic plasticity, which may lead to information loss. Biological experiments have found another LTD threshold that can induce either potentiation or depression or no change, even at the activated synapses. In addition, existing BCM learning rules can only select a set of fixed rule parameters, which is biologically implausible and practically inflexible to learn the structural information of input signals. In this article, an evolved dual-threshold BCM learning rule is proposed to regulate the reservoir internal connection weights of the echo-state-network (ESN), which can contribute to alleviating information loss and enhancing learning performance by introducing different optimal LTD thresholds for different postsynaptic neurons. Our experimental results show that the evolved dual-threshold BCM learning rule can result in the synergistic learning of different plasticity rules, effectively improving the learning performance of an ESN in comparison with existing neural plasticity learning rules and some state-of-the-art ESN variants on three widely used benchmark tasks and the prediction of an esterification process.
Collapse
|
5
|
Fu J, Li G, Tang J, Xia L, Wang L, Duan S. A double-cycle echo state network topology for time series prediction. CHAOS (WOODBURY, N.Y.) 2023; 33:093113. [PMID: 37695924 DOI: 10.1063/5.0159966] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Accepted: 08/17/2023] [Indexed: 09/13/2023]
Abstract
Echo state network (ESN) has gained wide acceptance in the field of time series prediction, relying on sufficiently complex reservoir connections to remember the historical features of the data and using these features to obtain the outputs by a simple linear readout. However, the randomness of its input and reservoir connections pose negative impacts on the prediction performance and performance stability of the models, the complexity of reservoir connections brings high time consumption during network computing, and the presence of randomness and complexity makes the hardware implementation of the ESN difficult. In response, we propose a double-cycle ESN (DCESN) based on the Li-ESN model, which has fixed weights to improve prediction performance and performance stability and simpler reservoir connections compared to the classical ESN to reduce the time consumption. The existence of both greatly reduces the difficulty of hardware implementation of the ESN and provides many conveniences for the future application of the ESN. Experimental results on many widely used time series datasets show that the DCESN has comparable or even better prediction performance than the ESN and good robustness against noise and parameter fluctuations.
Collapse
Affiliation(s)
- Jun Fu
- College of Artificial Intelligence, Southwest University, Chongqing 400715, People's Republic of China
| | - Guangli Li
- College of Artificial Intelligence, Southwest University, Chongqing 400715, People's Republic of China
| | - Jianfeng Tang
- College of Artificial Intelligence, Southwest University, Chongqing 400715, People's Republic of China
| | - Lei Xia
- College of Artificial Intelligence, Southwest University, Chongqing 400715, People's Republic of China
| | - Lidan Wang
- College of Artificial Intelligence, Southwest University, Chongqing 400715, People's Republic of China
- National and Local Joint Engineering Research Center of Intelligent Transmission and Control Technology, Chongqing 400715, People's Republic of China
- Chongqing Key Laboratory of Brain-inspired Computing and Intelligent Chips, Chongqing 400715, People's Republic of China
- Key Laboratory of Luminescence Analysis and Molecular Sensing (Southwest University), Ministry of Education, Southwest University, Chongqing 400715, People's Republic of China
| | - Shukai Duan
- College of Artificial Intelligence, Southwest University, Chongqing 400715, People's Republic of China
- National and Local Joint Engineering Research Center of Intelligent Transmission and Control Technology, Chongqing 400715, People's Republic of China
- Chongqing Key Laboratory of Brain-inspired Computing and Intelligent Chips, Chongqing 400715, People's Republic of China
- Key Laboratory of Luminescence Analysis and Molecular Sensing (Southwest University), Ministry of Education, Southwest University, Chongqing 400715, People's Republic of China
| |
Collapse
|
6
|
Kawai Y, Park J, Tsuda I, Asada M. Learning long-term motor timing/patterns on an orthogonal basis in random neural networks. Neural Netw 2023; 163:298-311. [PMID: 37087852 DOI: 10.1016/j.neunet.2023.04.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Revised: 03/15/2023] [Accepted: 04/05/2023] [Indexed: 04/25/2023]
Abstract
The ability of the brain to generate complex spatiotemporal patterns with specific timings is essential for motor learning and temporal processing. An approach that can model this function, using the spontaneous activity of a random neural network (RNN), is associated with orbital instability. We propose a simple system that learns an arbitrary time series as the linear sum of stable trajectories produced by several small network modules. New finding in computer experiments is that the trajectories of the module outputs are orthogonal to each other. They created a dynamic orthogonal basis acquiring a high representational capacity, which enabled the system to learn the timing of extremely long intervals, such as tens of seconds for a millisecond computation unit, and also the complex time series of Lorenz attractors. This self-sustained system satisfies the stability and orthogonality requirements and thus provides a new neurocomputing framework and perspective for the neural mechanisms of motor learning.
Collapse
Affiliation(s)
- Yuji Kawai
- Symbiotic Intelligent Systems Research Center, Institute for Open and Transdisciplinary Research Initiatives, Osaka University, 1-1 Yamadaoka, Suita, Osaka 565-0871, Japan.
| | - Jihoon Park
- Symbiotic Intelligent Systems Research Center, Institute for Open and Transdisciplinary Research Initiatives, Osaka University, 1-1 Yamadaoka, Suita, Osaka 565-0871, Japan; Center for Information and Neural Networks, National Institute of Information and Communications Technology, 1-4 Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Ichiro Tsuda
- Chubu University Academy of Emerging Sciences/Center for Mathematical Science and Artificial Intelligence, Chubu University, 1200 Matsumoto-cho, Kasugai, Aichi 487-8501, Japan
| | - Minoru Asada
- Symbiotic Intelligent Systems Research Center, Institute for Open and Transdisciplinary Research Initiatives, Osaka University, 1-1 Yamadaoka, Suita, Osaka 565-0871, Japan; Center for Information and Neural Networks, National Institute of Information and Communications Technology, 1-4 Yamadaoka, Suita, Osaka 565-0871, Japan; Chubu University Academy of Emerging Sciences/Center for Mathematical Science and Artificial Intelligence, Chubu University, 1200 Matsumoto-cho, Kasugai, Aichi 487-8501, Japan; International Professional University of Technology in Osaka, 3-3-1 Umeda, Kita-ku, Osaka 530-0001, Japan
| |
Collapse
|
7
|
Li Z, Liu Y, Tanaka G. Multi-Reservoir Echo State Networks with Hodrick–Prescott Filter for nonlinear time-series prediction. Appl Soft Comput 2023. [DOI: 10.1016/j.asoc.2023.110021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
|
8
|
Kitayama KI. Guiding principle of reservoir computing based on "small-world" network. Sci Rep 2022; 12:16697. [PMID: 36202989 PMCID: PMC9537422 DOI: 10.1038/s41598-022-21235-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2022] [Accepted: 09/26/2022] [Indexed: 11/09/2022] Open
Abstract
Reservoir computing is a computational framework of recurrent neural networks and is gaining attentions because of its drastically simplified training process. For a given task to solve, however, the methodology has not yet been established how to construct an optimal reservoir. While, "small-world" network has been known to represent networks in real-world such as biological systems and social community. This network is categorized amongst those that are completely regular and totally disordered, and it is characterized by highly-clustered nodes with a short path length. This study aims at providing a guiding principle of systematic synthesis of desired reservoirs by taking advantage of controllable parameters of the small-world network. We will validate the methodology using two different types of benchmark tests-classification task and prediction task.
Collapse
Affiliation(s)
- Ken-Ichi Kitayama
- National Institute of Information and Communications Technology, Tokyo, 184-8795, Japan. .,Hamamatsu Photonics K.K., Hamamatsu, 434-8601, Japan.
| |
Collapse
|
9
|
Deep reservoir calculation model and its application in the field of temperature and humidity prediction. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03685-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
10
|
Bose SK, Mallinson JB, Galli E, Acharya SK, Minnai C, Bones PJ, Brown SA. Neuromorphic behaviour in discontinuous metal films. NANOSCALE HORIZONS 2022; 7:437-445. [PMID: 35262143 DOI: 10.1039/d1nh00620g] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Physical systems that exhibit brain-like behaviour are currently under intense investigation as platforms for neuromorphic computing. We show that discontinuous metal films, comprising irregular flat islands on a substrate and formed using simple evaporation processes, exhibit correlated avalanches of electrical signals that mimic those observed in the cortex. We further demonstrate that these signals meet established criteria for criticality. We perform a detailed experimental investigation of the atomic-scale switching processes that are responsible for these signals, and show that they mimic the integrate-and-fire mechanism of biological neurons. Using numerical simulations and a simple circuit model, we show that the characteristic features of the switching events are dependent on the network state and the local position of the switch within the complex network. We conclude that discontinuous films provide an interesting potential platform for brain-inspired computing.
Collapse
Affiliation(s)
- Saurabh K Bose
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, University of Canterbury, Christchurch, New Zealand.
| | - Joshua B Mallinson
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, University of Canterbury, Christchurch, New Zealand.
| | - Edoardo Galli
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, University of Canterbury, Christchurch, New Zealand.
| | - Susant K Acharya
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, University of Canterbury, Christchurch, New Zealand.
| | - Chloé Minnai
- Molecular Cryo-Electron Microscopy Unit, Okinawa Institute of Science and Technology Graduate University, 1919-1 Tancha, Onna-son, Kunigami-gun, Okinawa, Japan
| | - Philip J Bones
- Electrical and Computer Engineering, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Simon A Brown
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, University of Canterbury, Christchurch, New Zealand.
| |
Collapse
|
11
|
Clustered and deep echo state networks for signal noise reduction. Mach Learn 2022. [DOI: 10.1007/s10994-022-06135-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
12
|
Prediction of Chaotic Time Series Based on SALR Model with Its Application on Heating Load Prediction. ARABIAN JOURNAL FOR SCIENCE AND ENGINEERING 2021. [DOI: 10.1007/s13369-021-05407-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
13
|
Daniels RK, Brown SA. Nanowire networks: how does small-world character evolve with dimensionality? NANOSCALE HORIZONS 2021; 6:482-488. [PMID: 33982039 DOI: 10.1039/d0nh00693a] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Networks of nanowires are currently under consideration for a wide range of electronic and optoelectronic applications. Nanowire devices are usually made by sequential deposition, which inevitably leads to stacking of the wires on top of one another. Here we demonstrate the effect of stacking on the topology of the resulting networks. We compare perfectly 2D networks with quasi-3D networks, and compare both nanowire networks to the corresponding Watts Strogatz networks, which are standard benchmark systems. By investigating quantities such as clustering, path length, modularity, and small world propensity we show that the connectivity of the quasi-3D networks is significantly different to that of the 2D networks, a result which may have important implications for applications of nanowire networks.
Collapse
Affiliation(s)
- Ryan K Daniels
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, University of Canterbury, Christchurch, New Zealand.
| | - Simon A Brown
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, University of Canterbury, Christchurch, New Zealand.
| |
Collapse
|
14
|
Zhang A, Xu Z. Chaotic time series prediction using phase space reconstruction based conceptor network. Cogn Neurodyn 2020; 14:849-857. [PMID: 33101536 DOI: 10.1007/s11571-020-09612-7] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2019] [Revised: 04/30/2020] [Accepted: 06/20/2020] [Indexed: 10/23/2022] Open
Abstract
The Conceptor network is a new framework of reservoir computing (RC), in addition to the features of easy training, global convergence, it can online learn new classes of input patterns without complete re-learning from all the training data. The conventional connection topology and weights of the hidden layer (reservoir) of RC are initialized randomly, and are fixed to be no longer fine-tuned after initialization. However, it has been demonstrated that the reservoir connection of RC plays an important role in the computational performance of RC. Therefore, in this paper, we optimize the Conceptor's reservoir connection and propose a phase space reconstruction (PSR) -based reservoir generation method. We tested the generation method on time series prediction task, and the experiment results showed that the proposed PSR-based method can improve the prediction accuracy of Conceptor networks. Further, we compared the PSR-based Conceptor with two Conceptor networks of other typical reservoir topologies (random connected, cortex-like connected), and found that all of their prediction accuracy showed a nonlinear decline trend with increasing storage load, but in comparison, our proposed PSR-based method has the best accuracy under different storage loads.
Collapse
Affiliation(s)
- Anguo Zhang
- College of Physics and Information Engineering, Fuzhou University, Fuzhou, 350108 China.,Key Laboratory of Medical Instrumentation and Pharmaceutical Technology of Fujian Province, Fuzhou, 350116 China.,Research Institute of Ruijie, Ruijie Networks Co., Ltd., Fuzhou, 350002 China
| | - Zheng Xu
- School of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing, 210094 China
| |
Collapse
|
15
|
Pike MD, Bose SK, Mallinson JB, Acharya SK, Shirai S, Galli E, Weddell SJ, Bones PJ, Arnold MD, Brown SA. Atomic Scale Dynamics Drive Brain-like Avalanches in Percolating Nanostructured Networks. NANO LETTERS 2020; 20:3935-3942. [PMID: 32347733 DOI: 10.1021/acs.nanolett.0c01096] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Self-assembled networks of nanoparticles and nanowires have recently emerged as promising systems for brain-like computation. Here, we focus on percolating networks of nanoparticles which exhibit brain-like dynamics. We use a combination of experiments and simulations to show that the brain-like network dynamics emerge from atomic-scale switching dynamics inside tunnel gaps that are distributed throughout the network. The atomic-scale dynamics emulate leaky integrate and fire (LIF) mechanisms in biological neurons, leading to the generation of critical avalanches of signals. These avalanches are quantitatively the same as those observed in cortical tissue and are signatures of the correlations that are required for computation. We show that the avalanches are associated with dynamical restructuring of the networks which self-tune to balanced states consistent with self-organized criticality. Our simulations allow visualization of the network states and detailed mechanisms of signal propagation.
Collapse
Affiliation(s)
- Matthew D Pike
- Electrical and Computer Engineering, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Saurabh K Bose
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matu̅, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Joshua B Mallinson
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matu̅, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Susant K Acharya
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matu̅, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Shota Shirai
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matu̅, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Edoardo Galli
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matu̅, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Stephen J Weddell
- Electrical and Computer Engineering, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Philip J Bones
- Electrical and Computer Engineering, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| | - Matthew D Arnold
- School of Mathematical and Physical Sciences, University of Technology Sydney, Sydney, Australia
| | - Simon A Brown
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matu̅, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand
| |
Collapse
|
16
|
Shirai S, Acharya SK, Bose SK, Mallinson JB, Galli E, Pike MD, Arnold MD, Brown SA. Long-range temporal correlations in scale-free neuromorphic networks. Netw Neurosci 2020; 4:432-447. [PMID: 32537535 PMCID: PMC7286302 DOI: 10.1162/netn_a_00128] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2019] [Accepted: 01/17/2020] [Indexed: 12/05/2022] Open
Abstract
Biological neuronal networks are the computing engines of the mammalian brain. These networks exhibit structural characteristics such as hierarchical architectures, small-world attributes, and scale-free topologies, providing the basis for the emergence of rich temporal characteristics such as scale-free dynamics and long-range temporal correlations. Devices that have both the topological and the temporal features of a neuronal network would be a significant step toward constructing a neuromorphic system that can emulate the computational ability and energy efficiency of the human brain. Here we use numerical simulations to show that percolating networks of nanoparticles exhibit structural properties that are reminiscent of biological neuronal networks, and then show experimentally that stimulation of percolating networks by an external voltage stimulus produces temporal dynamics that are self-similar, follow power-law scaling, and exhibit long-range temporal correlations. These results are expected to have important implications for the development of neuromorphic devices, especially for those based on the concept of reservoir computing. Biological neuronal networks exhibit well-defined properties such as hierarchical structures and scale-free topologies, as well as a high degree of local clustering and short path lengths between nodes. These structural properties are intimately connected to the observed long-range temporal correlations in the network dynamics. Fabrication of artificial networks with similar structural properties would facilitate brain-like (“neuromorphic”) computing. Here we show experimentally that percolating networks of nanoparticles exhibit similar long-range temporal correlations to those of biological neuronal networks and use simulations to demonstrate that the dynamics arise from an underlying scale-free network architecture. We discuss similarities between the biological and percolating systems and highlight the potential for the percolating networks to be used in neuromorphic computing applications.
Collapse
Affiliation(s)
- Shota Shirai
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Christchurch, New Zealand
| | - Susant Kumar Acharya
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Christchurch, New Zealand
| | - Saurabh Kumar Bose
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Christchurch, New Zealand
| | - Joshua Brian Mallinson
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Christchurch, New Zealand
| | - Edoardo Galli
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Christchurch, New Zealand
| | - Matthew D Pike
- Electrical and Electronics Engineering, University of Canterbury, Christchurch, New Zealand
| | - Matthew D Arnold
- School of Mathematical and Physical Sciences, University of Technology Sydney, Australia
| | - Simon Anthony Brown
- The MacDiarmid Institute for Advanced Materials and Nanotechnology, School of Physical and Chemical Sciences, Te Kura Matū, University of Canterbury, Christchurch, New Zealand
| |
Collapse
|
17
|
Liu K, Zhang J. Nonlinear process modelling using echo state networks optimised by covariance matrix adaption evolutionary strategy. Comput Chem Eng 2020. [DOI: 10.1016/j.compchemeng.2020.106730] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
18
|
Wang X, Jin Y, Hao K. Evolving Local Plasticity Rules for Synergistic Learning in Echo State Networks. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2020; 31:1363-1374. [PMID: 31247578 DOI: 10.1109/tnnls.2019.2919903] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Existing synaptic plasticity rules for optimizing the connections between neurons within the reservoir of echo state networks (ESNs) remain to be global in that the same type of plasticity rule with the same parameters is applied to all neurons. However, this is biologically implausible and practically inflexible for learning the structures in the input signals, thereby limiting the learning performance of ESNs. In this paper, we propose to use local plasticity rules that allow different neurons to use different types of plasticity rules and different parameters, which are achieved by optimizing the parameters of the local plasticity rules using the evolution strategy (ES) with covariance matrix adaptation (CMA-ES). We show that evolving neural plasticity will result in a synergistic learning of different plasticity rules, which plays an important role in improving the learning performance. Meanwhile, we show that the local plasticity rules can effectively alleviate synaptic interferences in learning the structure in sensory inputs. The proposed local plasticity rules are compared with a number of the state-of-the-art ESN models and the canonical ESN using a global plasticity rule on a set of widely used prediction and classification benchmark problems to demonstrate its competitive learning performance.
Collapse
|
19
|
Liu J, Sun T, Luo Y, Yang S, Cao Y, Zhai J. An echo state network architecture based on quantum logic gate and its optimization. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2019.09.002] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
20
|
|
21
|
Zeng G, Huang X, Jiang T, Yu S. Short-term synaptic plasticity expands the operational range of long-term synaptic changes in neural networks. Neural Netw 2019; 118:140-147. [DOI: 10.1016/j.neunet.2019.06.002] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2018] [Revised: 04/23/2019] [Accepted: 06/03/2019] [Indexed: 02/04/2023]
|
22
|
|
23
|
Wang X, Jin Y, Hao K. Echo state networks regulated by local intrinsic plasticity rules for regression. Neurocomputing 2019. [DOI: 10.1016/j.neucom.2019.03.032] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
|
24
|
Heave compensation prediction based on echo state network with correntropy induced loss function. PLoS One 2019; 14:e0217361. [PMID: 31194791 PMCID: PMC6563959 DOI: 10.1371/journal.pone.0217361] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2019] [Accepted: 05/09/2019] [Indexed: 11/20/2022] Open
Abstract
In this paper, a new prediction approach is proposed for ocean vessel heave compensation based on echo state network (ESN). To improve the prediction accuracy and enhance the robustness against noise and outliers, a generalized similarity measure called correntropy is introduced into ESN training, which is referred as corr-ESN. An iterative method based on half-quadratic minimization is derived to train corr-ESN. The proposed corr-ESN is used for the heave motion prediction. The experimental results verify its effectiveness.
Collapse
|
25
|
Kawai Y, Park J, Asada M. A small-world topology enhances the echo state property and signal propagation in reservoir computing. Neural Netw 2019; 112:15-23. [DOI: 10.1016/j.neunet.2019.01.002] [Citation(s) in RCA: 58] [Impact Index Per Article: 11.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2018] [Revised: 10/19/2018] [Accepted: 01/07/2019] [Indexed: 01/22/2023]
|
26
|
Rodriguez N, Izquierdo E, Ahn YY. Optimal modularity and memory capacity of neural reservoirs. Netw Neurosci 2019; 3:551-566. [PMID: 31089484 PMCID: PMC6497001 DOI: 10.1162/netn_a_00082] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2018] [Accepted: 02/25/2019] [Indexed: 11/04/2022] Open
Abstract
The neural network is a powerful computing framework that has been exploited by biological evolution and by humans for solving diverse problems. Although the computational capabilities of neural networks are determined by their structure, the current understanding of the relationships between a neural network's architecture and function is still primitive. Here we reveal that a neural network's modular architecture plays a vital role in determining the neural dynamics and memory performance of the network of threshold neurons. In particular, we demonstrate that there exists an optimal modularity for memory performance, where a balance between local cohesion and global connectivity is established, allowing optimally modular networks to remember longer. Our results suggest that insights from dynamical analysis of neural networks and information-spreading processes can be leveraged to better design neural networks and may shed light on the brain's modular organization.
Collapse
Affiliation(s)
- Nathaniel Rodriguez
- School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA
| | - Eduardo Izquierdo
- School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA
- Cognitive Science Program, Indiana University, Bloomington, IN, USA
| | - Yong-Yeol Ahn
- School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA
- Indiana University Network Science Institute, Bloomington, IN, USA
| |
Collapse
|
27
|
|
28
|
Xue F, Li Q, Li X. The combination of circle topology and leaky integrator neurons remarkably improves the performance of echo state network on time series prediction. PLoS One 2017; 12:e0181816. [PMID: 28759581 PMCID: PMC5536322 DOI: 10.1371/journal.pone.0181816] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2016] [Accepted: 06/26/2017] [Indexed: 11/18/2022] Open
Abstract
Recently, echo state network (ESN) has attracted a great deal of attention due to its high accuracy and efficient learning performance. Compared with the traditional random structure and classical sigmoid units, simple circle topology and leaky integrator neurons have more advantages on reservoir computing of ESN. In this paper, we propose a new model of ESN with both circle reservoir structure and leaky integrator units. By comparing the prediction capability on Mackey-Glass chaotic time series of four ESN models: classical ESN, circle ESN, traditional leaky integrator ESN, circle leaky integrator ESN, we find that our circle leaky integrator ESN shows significantly better performance than other ESNs with roughly 2 orders of magnitude reduction of the predictive error. Moreover, this model has stronger ability to approximate nonlinear dynamics and resist noise than conventional ESN and ESN with only simple circle structure or leaky integrator neurons. Our results show that the combination of circle topology and leaky integrator neurons can remarkably increase dynamical diversity and meanwhile decrease the correlation of reservoir states, which contribute to the significant improvement of computational performance of Echo state network on time series prediction.
Collapse
Affiliation(s)
- Fangzheng Xue
- Key Laboratory of Dependable Service Computing in Cyber Physical Society of Ministry of Education, Chongqing University, Chongqing 400044, China
- College of Automation, Chongqing University, Chongqing 400044, China
| | - Qian Li
- Key Laboratory of Dependable Service Computing in Cyber Physical Society of Ministry of Education, Chongqing University, Chongqing 400044, China
- College of Automation, Chongqing University, Chongqing 400044, China
| | - Xiumin Li
- Key Laboratory of Dependable Service Computing in Cyber Physical Society of Ministry of Education, Chongqing University, Chongqing 400044, China
- College of Automation, Chongqing University, Chongqing 400044, China
- * E-mail:
| |
Collapse
|
29
|
|
30
|
Reservoir Computing with Both Neuronal Intrinsic Plasticity and Multi-Clustered Structure. Cognit Comput 2017. [DOI: 10.1007/s12559-017-9467-3] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
31
|
Dolinský J, Hirose K, Konishi S. Readouts for echo-state networks built using locally regularized orthogonal forward regression. J Appl Stat 2017. [DOI: 10.1080/02664763.2017.1305331] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
| | - Kei Hirose
- Institute of Mathematics for Industry, Kyushu University, Nishi-ku, Fukuoka, Japan
| | - Sadanori Konishi
- Department of Mathematics, Faculty of Science and Engineering, Chuo University, Tokyo, Japan
| |
Collapse
|
32
|
Qiao J, Li F, Han H, Li W. Growing Echo-State Network With Multiple Subreservoirs. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2017; 28:391-404. [PMID: 26800553 DOI: 10.1109/tnnls.2016.2514275] [Citation(s) in RCA: 56] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
An echo-state network (ESN) is an effective alternative to gradient methods for training recurrent neural network. However, it is difficult to determine the structure (mainly the reservoir) of the ESN to match with the given application. In this paper, a growing ESN (GESN) is proposed to design the size and topology of the reservoir automatically. First, the GESN makes use of the block matrix theory to add hidden units to the existing reservoir group by group, which leads to a GESN with multiple subreservoirs. Second, every subreservoir weight matrix in the GESN is created with a predefined singular value spectrum, which ensures the echo-sate property of the ESN without posterior scaling of the weights. Third, during the growth of the network, the output weights of the GESN are updated in an incremental way. Moreover, the convergence of the GESN is proved. Finally, the GESN is tested on some artificial and real-world time-series benchmarks. Simulation results show that the proposed GESN has better prediction performance and faster leaning speed than some ESNs with fixed sizes and topologies.
Collapse
|
33
|
|
34
|
|
35
|
Cheng Z, Deng Z, Hu X, Zhang B, Yang T. Efficient reinforcement learning of a reservoir network model of parametric working memory achieved with a cluster population winner-take-all readout mechanism. J Neurophysiol 2015; 114:3296-305. [PMID: 26445865 DOI: 10.1152/jn.00378.2015] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2015] [Accepted: 10/07/2015] [Indexed: 11/22/2022] Open
Abstract
The brain often has to make decisions based on information stored in working memory, but the neural circuitry underlying working memory is not fully understood. Many theoretical efforts have been focused on modeling the persistent delay period activity in the prefrontal areas that is believed to represent working memory. Recent experiments reveal that the delay period activity in the prefrontal cortex is neither static nor homogeneous as previously assumed. Models based on reservoir networks have been proposed to model such a dynamical activity pattern. The connections between neurons within a reservoir are random and do not require explicit tuning. Information storage does not depend on the stable states of the network. However, it is not clear how the encoded information can be retrieved for decision making with a biologically realistic algorithm. We therefore built a reservoir-based neural network to model the neuronal responses of the prefrontal cortex in a somatosensory delayed discrimination task. We first illustrate that the neurons in the reservoir exhibit a heterogeneous and dynamical delay period activity observed in previous experiments. Then we show that a cluster population circuit decodes the information from the reservoir with a winner-take-all mechanism and contributes to the decision making. Finally, we show that the model achieves a good performance rapidly by shaping only the readout with reinforcement learning. Our model reproduces important features of previous behavior and neurophysiology data. We illustrate for the first time how task-specific information stored in a reservoir network can be retrieved with a biologically plausible reinforcement learning training scheme.
Collapse
Affiliation(s)
- Zhenbo Cheng
- State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China; Department of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China; and
| | - Zhidong Deng
- State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China
| | - Xiaolin Hu
- State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China
| | - Bo Zhang
- State Key Laboratory of Intelligent Technology and Systems, Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China
| | - Tianming Yang
- Institute of Neuroscience, Key Laboratory of Primate Neurobiology, CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai Institutes for Biological Sciences, Chinese Academy of Sciences, Shanghai, China
| |
Collapse
|
36
|
Wang H, Yan X. Optimizing the echo state network with a binary particle swarm optimization algorithm. Knowl Based Syst 2015. [DOI: 10.1016/j.knosys.2015.06.003] [Citation(s) in RCA: 74] [Impact Index Per Article: 8.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
37
|
Li X, Zhong L, Xue F, Zhang A. A priori data-driven multi-clustered reservoir generation algorithm for echo state network. PLoS One 2015; 10:e0120750. [PMID: 25875296 PMCID: PMC4395262 DOI: 10.1371/journal.pone.0120750] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2014] [Accepted: 02/06/2015] [Indexed: 11/18/2022] Open
Abstract
Echo state networks (ESNs) with multi-clustered reservoir topology perform better in reservoir computing and robustness than those with random reservoir topology. However, these ESNs have a complex reservoir topology, which leads to difficulties in reservoir generation. This study focuses on the reservoir generation problem when ESN is used in environments with sufficient priori data available. Accordingly, a priori data-driven multi-cluster reservoir generation algorithm is proposed. The priori data in the proposed algorithm are used to evaluate reservoirs by calculating the precision and standard deviation of ESNs. The reservoirs are produced using the clustering method; only the reservoir with a better evaluation performance takes the place of a previous one. The final reservoir is obtained when its evaluation score reaches the preset requirement. The prediction experiment results obtained using the Mackey-Glass chaotic time series show that the proposed reservoir generation algorithm provides ESNs with extra prediction precision and increases the structure complexity of the network. Further experiments also reveal the appropriate values of the number of clusters and time window size to obtain optimal performance. The information entropy of the reservoir reaches the maximum when ESN gains the greatest precision.
Collapse
Affiliation(s)
- Xiumin Li
- Key Laboratory of Dependable Service Computing in Cyber Physical Society of Ministry of Education, Chongqing University, Chongqing 400044, China
- College of Automation, Chongqing University, Chongqing 400044, China
| | - Ling Zhong
- Key Laboratory of Dependable Service Computing in Cyber Physical Society of Ministry of Education, Chongqing University, Chongqing 400044, China
- College of Automation, Chongqing University, Chongqing 400044, China
| | - Fangzheng Xue
- Key Laboratory of Dependable Service Computing in Cyber Physical Society of Ministry of Education, Chongqing University, Chongqing 400044, China
- College of Automation, Chongqing University, Chongqing 400044, China
- * E-mail:
| | - Anguo Zhang
- Key Laboratory of Dependable Service Computing in Cyber Physical Society of Ministry of Education, Chongqing University, Chongqing 400044, China
- College of Automation, Chongqing University, Chongqing 400044, China
| |
Collapse
|
38
|
SCESN, SPESN, SWESN: Three recurrent neural echo state networks with clustered reservoirs for prediction of nonlinear and chaotic time series. APPL INTELL 2015. [DOI: 10.1007/s10489-015-0652-3] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
39
|
Soh H, Demiris Y. Spatio-temporal learning with the online finite and infinite echo-state Gaussian processes. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2015; 26:522-536. [PMID: 25720008 DOI: 10.1109/tnnls.2014.2316291] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
Successful biological systems adapt to change. In this paper, we are principally concerned with adaptive systems that operate in environments where data arrives sequentially and is multivariate in nature, for example, sensory streams in robotic systems. We contribute two reservoir inspired methods: 1) the online echostate Gaussian process (OESGP) and 2) its infinite variant, the online infinite echostate Gaussian process (OIESGP) Both algorithms are iterative fixed-budget methods that learn from noisy time series. In particular, the OESGP combines the echo-state network with Bayesian online learning for Gaussian processes. Extending this to infinite reservoirs yields the OIESGP, which uses a novel recursive kernel with automatic relevance determination that enables spatial and temporal feature weighting. When fused with stochastic natural gradient descent, the kernel hyperparameters are iteratively adapted to better model the target system. Furthermore, insights into the underlying system can be gleamed from inspection of the resulting hyperparameters. Experiments on noisy benchmark problems (one-step prediction and system identification) demonstrate that our methods yield high accuracies relative to state-of-the-art methods, and standard kernels with sliding windows, particularly on problems with irrelevant dimensions. In addition, we describe two case studies in robotic learning-by-demonstration involving the Nao humanoid robot and the Assistive Robot Transport for Youngsters (ARTY) smart wheelchair.
Collapse
|
40
|
Improved simple deterministically constructed Cycle Reservoir Network with Sensitive Iterative Pruning Algorithm. Neurocomputing 2014. [DOI: 10.1016/j.neucom.2014.05.024] [Citation(s) in RCA: 23] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
41
|
Ma Q, Chen W, Wei J, Yu Z. Direct model of memory properties and the linear reservoir topologies in echo state networks. Appl Soft Comput 2014. [DOI: 10.1016/j.asoc.2014.04.038] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
42
|
Effect of hybrid circle reservoir injected with wavelet-neurons on performance of echo state network. Neural Netw 2014; 57:141-51. [DOI: 10.1016/j.neunet.2014.05.013] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2013] [Revised: 04/14/2014] [Accepted: 05/26/2014] [Indexed: 11/24/2022]
|
43
|
Oubbati M, Kord B, Koprinkova-Hristova P, Palm G. Learning of embodied interaction dynamics with recurrent neural networks: some exploratory experiments. J Neural Eng 2014; 11:026019. [PMID: 24658453 DOI: 10.1088/1741-2560/11/2/026019] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
The new tendency of artificial intelligence suggests that intelligence must be seen as a result of the interaction between brains, bodies and environments. This view implies that designing sophisticated behaviour requires a primary focus on how agents are functionally coupled to their environments. Under this perspective, we present early results with the application of reservoir computing as an efficient tool to understand how behaviour emerges from interaction. Specifically, we present reservoir computing models, that are inspired by imitation learning designs, to extract the essential components of behaviour that results from agent-environment interaction dynamics. Experimental results using a mobile robot are reported to validate the learning architectures.
Collapse
Affiliation(s)
- Mohamed Oubbati
- Institute of Neural Information Processing, Ulm University, Germany
| | | | | | | |
Collapse
|
44
|
|
45
|
Cui H, Liu X, Li L. The architecture of dynamic reservoir in the echo state network. CHAOS (WOODBURY, N.Y.) 2012; 22:033127. [PMID: 23020466 DOI: 10.1063/1.4746765] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
Echo state network (ESN) has recently attracted increasing interests because of its superior capability in modeling nonlinear dynamic systems. In the conventional echo state network model, its dynamic reservoir (DR) has a random and sparse topology, which is far from the real biological neural networks from both structural and functional perspectives. We hereby propose three novel types of echo state networks with new dynamic reservoir topologies based on complex network theory, i.e., with a small-world topology, a scale-free topology, and a mixture of small-world and scale-free topologies, respectively. We then analyze the relationship between the dynamic reservoir structure and its prediction capability. We utilize two commonly used time series to evaluate the prediction performance of the three proposed echo state networks and compare them to the conventional model. We also use independent and identically distributed time series to analyze the short-term memory and prediction precision of these echo state networks. Furthermore, we study the ratio of scale-free topology and the small-world topology in the mixed-topology network, and examine its influence on the performance of the echo state networks. Our simulation results show that the proposed echo state network models have better prediction capabilities, a wider spectral radius, but retain almost the same short-term memory capacity as compared to the conventional echo state network model. We also find that the smaller the ratio of the scale-free topology over the small-world topology, the better the memory capacities.
Collapse
Affiliation(s)
- Hongyan Cui
- Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China.
| | | | | |
Collapse
|
46
|
Rodan A, Tiňo P. Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps. Neural Comput 2012; 24:1822-52. [DOI: 10.1162/neco_a_00297] [Citation(s) in RCA: 61] [Impact Index Per Article: 5.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
A new class of state-space models, reservoir models, with a fixed state transition structure (the “reservoir”) and an adaptable readout from the state space, has recently emerged as a way for time series processing and modeling. Echo state network (ESN) is one of the simplest, yet powerful, reservoir models. ESN models are generally constructed in a randomized manner. In our previous study (Rodan & Tiňo, 2011 ), we showed that a very simple, cyclic, deterministically generated reservoir can yield performance competitive with standard ESN. In this contribution, we extend our previous study in three aspects. First, we introduce a novel simple deterministic reservoir model, cycle reservoir with jumps (CRJ), with highly constrained weight values, that has superior performance to standard ESN on a variety of temporal tasks of different origin and characteristics. Second, we elaborate on the possible link between reservoir characterizations, such as eigenvalue distribution of the reservoir matrix or pseudo-Lyapunov exponent of the input-driven reservoir dynamics, and the model performance. It has been suggested that a uniform coverage of the unit disk by such eigenvalues can lead to superior model performance. We show that despite highly constrained eigenvalue distribution, CRJ consistently outperforms ESN (which has much more uniform eigenvalue coverage of the unit disk). Also, unlike in the case of ESN, pseudo-Lyapunov exponents of the selected optimal CRJ models are consistently negative. Third, we present a new framework for determining the short-term memory capacity of linear reservoir models to a high degree of precision. Using the framework, we study the effect of shortcut connections in the CRJ reservoir topology on its memory capacity.
Collapse
Affiliation(s)
- Ali Rodan
- School of Computer Science, University of Birmingham, Birmingham B15 2TT, U.K
| | - Peter Tiňo
- School of Computer Science, University of Birmingham, Birmingham B15 2TT, U.K
| |
Collapse
|
47
|
Zhang B, Miller DJ, Wang Y. Nonlinear system modeling with random matrices: echo state networks revisited. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2012; 23:175-182. [PMID: 24808467 PMCID: PMC4107715 DOI: 10.1109/tnnls.2011.2178562] [Citation(s) in RCA: 18] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
Echo state networks (ESNs) are a novel form of recurrent neural networks (RNNs) that provide an efficient and powerful computational model approximating nonlinear dynamical systems. A unique feature of an ESN is that a large number of neurons (the "reservoir") are used, whose synaptic connections are generated randomly, with only the connections from the reservoir to the output modified by learning. Why a large randomly generated fixed RNN gives such excellent performance in approximating nonlinear systems is still not well understood. In this brief, we apply random matrix theory to examine the properties of random reservoirs in ESNs under different topologies (sparse or fully connected) and connection weights (Bernoulli or Gaussian). We quantify the asymptotic gap between the scaling factor bounds for the necessary and sufficient conditions previously proposed for the echo state property. We then show that the state transition mapping is contractive with high probability when only the necessary condition is satisfied, which corroborates and thus analytically explains the observation that in practice one obtains echo states when the spectral radius of the reservoir weight matrix is smaller than 1.
Collapse
Affiliation(s)
- Bai Zhang
- Bradley Department of Electrical and Computer Engineering, Virginia Polytechnic Institute and State University, Arlington, VA 22203 USA
| | - David J. Miller
- Department of Electrical Engineering, Pennsylvania State University, University Park, PA 16802 USA
| | - Yue Wang
- Bradley Department of Electrical and Computer Engineering, Virginia Polytechnic Institute and State University, Arlington, VA 22203 USA
| |
Collapse
|
48
|
Abstract
Reservoir computing (RC) refers to a new class of state-space models with a fixed state transition structure (the reservoir) and an adaptable readout form the state space. The reservoir is supposed to be sufficiently complex so as to capture a large number of features of the input stream that can be exploited by the reservoir-to-output readout mapping. The field of RC has been growing rapidly with many successful applications. However, RC has been criticized for not being principled enough. Reservoir construction is largely driven by a series of randomized model-building stages, with both researchers and practitioners having to rely on a series of trials and errors. To initialize a systematic study of the field, we concentrate on one of the most popular classes of RC methods, namely echo state network, and ask: What is the minimal complexity of reservoir construction for obtaining competitive models and what is the memory capacity (MC) of such simplified reservoirs? On a number of widely used time series benchmarks of different origin and characteristics, as well as by conducting a theoretical analysis we show that a simple deterministically constructed cycle reservoir is comparable to the standard echo state network methodology. The (short-term) MC of linear cyclic reservoirs can be made arbitrarily close to the proved optimal value.
Collapse
Affiliation(s)
- Ali Rodan
- School of Computer Science, University of Birmingham, Birmingham B15 2TT, UK.
| | | |
Collapse
|
49
|
Jarvis S, Rotter S, Egert U. Extending stability through hierarchical clusters in echo state networks. Front Neuroinform 2010; 4. [PMID: 20725523 PMCID: PMC2914532 DOI: 10.3389/fninf.2010.00011] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2009] [Accepted: 06/10/2010] [Indexed: 11/21/2022] Open
Abstract
Echo State Networks (ESN) are reservoir networks that satisfy well-established criteria for stability when constructed as feedforward networks. Recent evidence suggests that stability criteria are altered in the presence of reservoir substructures, such as clusters. Understanding how the reservoir architecture affects stability is thus important for the appropriate design of any ESN. To quantitatively determine the influence of the most relevant network parameters, we analyzed the impact of reservoir substructures on stability in hierarchically clustered ESNs, as they allow a smooth transition from highly structured to increasingly homogeneous reservoirs. Previous studies used the largest eigenvalue of the reservoir connectivity matrix (spectral radius) as a predictor for stable network dynamics. Here, we evaluate the impact of clusters, hierarchy and intercluster connectivity on the predictive power of the spectral radius for stability. Both hierarchy and low relative cluster sizes extend the range of spectral radius values, leading to stable networks, while increasing intercluster connectivity decreased maximal spectral radius.
Collapse
|