Recurrent neural network architecture with pre-synaptic inhibition for incremental learning

Presynaptic inhibition rapidly stabilises recurrent excitation in the face of plasticity

Hebbian plasticity, a mechanism believed to be the substrate of learning and memory, detects and further enhances correlated neural activity. Because this constitutes an unstable positive feedback loop, it requires additional homeostatic control. Computational work suggests that in recurrent networks, the homeostatic mechanisms observed in experiments are too slow to compensate instabilities arising from Hebbian plasticity and need to be complemented by rapid compensatory processes. We suggest presynaptic inhibition as a candidate that rapidly provides stability by compensating recurrent excitation induced by Hebbian changes. Presynaptic inhibition is mediated by presynaptic GABA receptors that effectively and reversibly attenuate transmitter release. Activation of these receptors can be triggered by excess network activity, hence providing a stabilising negative feedback loop that weakens recurrent interactions on sub-second timescales. We study the stabilising effect of presynaptic inhibition in recurrent networks, in which presynaptic inhibition is implemented as a multiplicative reduction of recurrent synaptic weights in response to increasing inhibitory activity. We show that networks with presynaptic inhibition display a gradual increase of firing rates with growing excitatory weights, in contrast to traditional excitatory-inhibitory networks. This alleviates the positive feedback loop between Hebbian plasticity and network activity and thereby allows homeostasis to act on timescales similar to those observed in experiments. Our results generalise to spiking networks with a biophysically more detailed implementation of the presynaptic inhibition mechanism. In conclusion, presynaptic inhibition provides a powerful compensatory mechanism that rapidly reduces effective recurrent interactions and thereby stabilises Hebbian learning.

Synapses between neurons change during learning and memory formation, a process termed synaptic plasticity. Established models of plasticity rely on strengthening synapses of co-active neurons. In recurrent networks, mutually connected neurons tend to be co-active. The emerging positive feedback loop is believed to be counteracted by homeostatic mechanisms that aim to keep neural activity at a given set point. However, theoretical work indicates that experimentally observed forms of homeostasis are too slow to maintain stable network activity. In this article, we suggest that presynaptic inhibition can alleviate this problem. Presynaptic inhibition is an inhibitory mechanism that weakens synapses rather than suppressing neural activity. Using mathematical analyses and computer simulations, we show that presynaptic inhibition can compensate the strengthening of recurrent connections and thus stabilises neural networks subject to synaptic plasticity, even if homeostasis acts on biologically plausible timescales.

Multilayered neural network with structural lateral inhibition for incremental learning and conceptualization

Distributed connectionist networks have difficulty learning incrementally because the representations in the network overlap. Therefore, it is necessary to reduce the overlaps of representations for incremental learning. At the same time, the representational overlaps give these networks the ability to generalize. In this study, we use a modified multilayered neural network to numerically examine the trade-off between incremental learning and generalization abilities, and then we propose a novel network model with structural lateral inhibitions to reconcile the two abilities. We also analyze the behavior of the proposed model using Formal Concept Analysis, which reveals that the network implements "conceptualization": differentiation and meditation between intensional and extensional representations. This study suggests a new paradigm for the traditional question, whether representations in the brain are distributed or not.

An incremental neural network with a reduced architecture

This paper proposes a technique, called Evolving Probabilistic Neural Network (ePNN), that presents many interesting features, including incremental learning, evolving architecture, the capacity to learn continually throughout its existence and requiring that each training sample be used only once in the training phase without reprocessing. A series of experiments was performed on data sets in the public domain; the results indicate that ePNN is superior or equal to the other incremental neural networks evaluated in this paper. These results also demonstrate the advantage of the small ePNN architecture and show that its architecture is more stable than the other incremental neural networks evaluated. ePNN thus appears to be a promising alternative for a quick learning system and a fast classifier with a low computational cost.