Overcoming catastrophic forgetting in neural networks

A Dynamic Connectome Supports the Emergence of Stable Computational Function of Neural Circuits through Reward-Based Learning

Synaptic connections between neurons in the brain are dynamic because of continuously ongoing spine dynamics, axonal sprouting, and other processes. In fact, it was recently shown that the spontaneous synapse-autonomous component of spine dynamics is at least as large as the component that depends on the history of pre- and postsynaptic neural activity. These data are inconsistent with common models for network plasticity and raise the following questions: how can neural circuits maintain a stable computational function in spite of these continuously ongoing processes, and what could be functional uses of these ongoing processes? Here, we present a rigorous theoretical framework for these seemingly stochastic spine dynamics and rewiring processes in the context of reward-based learning tasks. We show that spontaneous synapse-autonomous processes, in combination with reward signals such as dopamine, can explain the capability of networks of neurons in the brain to configure themselves for specific computational tasks, and to compensate automatically for later changes in the network or task. Furthermore, we show theoretically and through computer simulations that stable computational performance is compatible with continuously ongoing synapse-autonomous changes. After reaching good computational performance it causes primarily a slow drift of network architecture and dynamics in task-irrelevant dimensions, as observed for neural activity in motor cortex and other areas. On the more abstract level of reinforcement learning the resulting model gives rise to an understanding of reward-driven network plasticity as continuous sampling of network configurations.

Auditory perceptual learning and changes in the conceptualization of auditory cortex

Perceptual learning, improvement in discriminative ability as a consequence of training, is one of the forms of sensory system plasticity that has driven profound changes in our conceptualization of sensory cortical function. Psychophysical and neurophysiological studies of auditory perceptual learning have indicated that the characteristics of the learning, and by implication the nature of the underlying neural changes, are highly task specific. Some studies in animals have indicated that recruitment of neurons to the population responding to the training stimuli, and hence an increase in the so-called cortical "area of representation" of those stimuli, is the substrate of improved performance, but such changes have not been observed in other studies. A possible reconciliation of these conflicting results is provided by evidence that changes in area of representation constitute a transient stage in the processes underlying perceptual learning. This expansion - renormalization hypothesis is supported by evidence from studies of the learning of motor skills, another form of procedural learning, but leaves open the nature of the permanent neural substrate of improved performance. Other studies have suggested that the substrate might be reduced response variability - a decrease in internal noise. Neuroimaging studies in humans have also provided compelling evidence that training results in long-term changes in auditory cortical function and in the auditory brainstem frequency-following response. Musical training provides a valuable model, but the evidence it provides is qualified by the fact that most such training is multimodal and sensorimotor, and that few of the studies are experimental and allow control over confounding variables. More generally, the overwhelming majority of experimental studies of the various forms of auditory perceptual learning have established the co-occurrence of neural and perceptual changes, but have not established that the former are causally related to the latter. Important forms of perceptual learning in humans are those involved in language acquisition and in the improvement in speech perception performance of post-lingually deaf cochlear implantees over the months following implantation. The development of a range of auditory training programs has focused interest on the factors determining the extent to which perceptual learning is specific or generalises to tasks other than those used in training. The context specificity demonstrated in a number of studies of perceptual learning suggests a multiplexing model, in which learning relating to a particular stimulus attribute depends on a subset of the diverse inputs to a given cortical neuron being strengthened, and different subsets being gated by top-down influences. This hypothesis avoids the difficulty of balancing system stability with plasticity, which is a problem for recruitment hypotheses. The characteristics of auditory perceptual learning reflect the fact that auditory cortex forms part of distributed networks that integrate the representation of auditory stimuli with attention, decision, and reward processes.

Data science as a language: challenges for computer science-a position paper

In this paper, I posit that from a research point of view, Data Science is a language. More precisely Data Science is doing Science using computer science as a language for datafied sciences; much as mathematics is the language of, e.g., physics. From this viewpoint, three (classes) of challenges for computer science are identified; complementing the challenges the closely related Big Data problem already poses to computer science. I discuss the challenges with references to, in my opinion, related, interesting directions in computer science research; note, I claim neither that these directions are the most appropriate to solve the challenges nor that the cited references represent the best work in their field, they are inspirational to me. So, what are these challenges? Firstly, if computer science is to be a language, what should that language look like? While our traditional specifications such as pseudocode are an excellent way to convey what has been done, they fail for more mathematics like reasoning about computations. Secondly, if computer science is to function as a foundation of other, datafied, sciences, its own foundations should be in order. While we have excellent foundations for supervised learning—e.g., by having loss functions to optimize and, more general, by PAC learning (Valiant in Commun ACM 27(11):1134–1142, 1984)—this is far less true for unsupervised learning. Kolmogorov complexity—or, more general, Algorithmic Information Theory—provides a solid base (Li and Vitányi in An introduction to Kolmogorov complexity and its applications, Springer, Berlin, 1993). It provides an objective criterion to choose between competing hypotheses, but it lacks, e.g., an objective measure of the uncertainty of a discovery that datafied sciences need. Thirdly, datafied sciences come with new conceptual challenges. Data-driven scientists come up with data analysis questions that sometimes do and sometimes don’t, fit our conceptual toolkit. Clearly, computer science does not suffer from a lack of interesting, deep, research problems. However, the challenges posed by data science point to a large reservoir of untapped problems. Interesting, stimulating problems, not in the least because they are posed by our colleagues in datafied sciences. It is an exciting time to be a computer scientist.

Computational Foundations of Natural Intelligence

New developments in AI and neuroscience are revitalizing the quest to understanding natural intelligence, offering insight about how to equip machines with human-like capabilities. This paper reviews some of the computational principles relevant for understanding natural intelligence and, ultimately, achieving strong AI. After reviewing basic principles, a variety of computational modeling approaches is discussed. Subsequently, I concentrate on the use of artificial neural networks as a framework for modeling cognitive processes. This paper ends by outlining some of the challenges that remain to fulfill the promise of machines that show human-like intelligence.

Deep imitation learning for 3D navigation tasks

Deep learning techniques have shown success in learning from raw high-dimensional data in various applications. While deep reinforcement learning is recently gaining popularity as a method to train intelligent agents, utilizing deep learning in imitation learning has been scarcely explored. Imitation learning can be an efficient method to teach intelligent agents by providing a set of demonstrations to learn from. However, generalizing to situations that are not represented in the demonstrations can be challenging, especially in 3D environments. In this paper, we propose a deep imitation learning method to learn navigation tasks from demonstrations in a 3D environment. The supervised policy is refined using active learning in order to generalize to unseen situations. This approach is compared to two popular deep reinforcement learning techniques: deep-Q-networks and Asynchronous actor-critic (A3C). The proposed method as well as the reinforcement learning methods employ deep convolutional neural networks and learn directly from raw visual input. Methods for combining learning from demonstrations and experience are also investigated. This combination aims to join the generalization ability of learning by experience with the efficiency of learning by imitation. The proposed methods are evaluated on 4 navigation tasks in a 3D simulated environment. Navigation tasks are a typical problem that is relevant to many real applications. They pose the challenge of requiring demonstrations of long trajectories to reach the target and only providing delayed rewards (usually terminal) to the agent. The experiments show that the proposed method can successfully learn navigation tasks from raw visual input while learning from experience methods fail to learn an effective policy. Moreover, it is shown that active learning can significantly improve the performance of the initially learned policy using a small number of active samples.

Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks

A long-term goal of AI is to produce agents that can learn a diversity of skills throughout their lifetimes and continuously improve those skills via experience. A longstanding obstacle towards that goal is catastrophic forgetting, which is when learning new information erases previously learned information. Catastrophic forgetting occurs in artificial neural networks (ANNs), which have fueled most recent advances in AI. A recent paper proposed that catastrophic forgetting in ANNs can be reduced by promoting modularity, which can limit forgetting by isolating task information to specific clusters of nodes and connections (functional modules). While the prior work did show that modular ANNs suffered less from catastrophic forgetting, it was not able to produce ANNs that possessed task-specific functional modules, thereby leaving the main theory regarding modularity and forgetting untested. We introduce diffusion-based neuromodulation, which simulates the release of diffusing, neuromodulatory chemicals within an ANN that can modulate (i.e. up or down regulate) learning in a spatial region. On the simple diagnostic problem from the prior work, diffusion-based neuromodulation 1) induces task-specific learning in groups of nodes and connections (task-specific localized learning), which 2) produces functional modules for each subtask, and 3) yields higher performance by eliminating catastrophic forgetting. Overall, our results suggest that diffusion-based neuromodulation promotes task-specific localized learning and functional modularity, which can help solve the challenging, but important problem of catastrophic forgetting.

How thoughts arise from sights: inferotemporal and prefrontal contributions to vision

We are rapidly approaching a comprehensive understanding of the neural mechanisms behind object recognition. How we use this knowledge of the visual world to plan and act is comparatively mysterious. To fill this gap, we must understand how visual representations are transformed within cognitive regions, and how these cognitive representations of visual information act back upon earlier sensory representations. Here, we summarize our current understanding of visual representation in inferotemporal cortex (IT) and prefrontal cortex (PFC), and the interactions between them. We emphasize the apparent consistency of visual representation in PFC across tasks, and suggest ways to leverage advances in our understanding of high-level vision to better understand cognitive processing.

Plasticity during Sleep Is Linked to Specific Regulation of Cortical Circuit Activity

Sleep is thought to be involved in the regulation of synaptic plasticity in two ways: by enhancing local plastic processes underlying the consolidation of specific memories and by supporting global synaptic homeostasis. Here, we briefly summarize recent structural and functional studies examining sleep-associated changes in synaptic morphology and neural excitability. These studies point to a global down-scaling of synaptic strength across sleep while a subset of synapses increases in strength. Similarly, neuronal excitability on average decreases across sleep, whereas subsets of neurons increase firing rates across sleep. Whether synapse formation and excitability is down or upregulated across sleep appears to partly depend on the cell's activity level during wakefulness. Processes of memory-specific upregulation of synapse formation and excitability are observed during slow wave sleep (SWS), whereas global downregulation resulting in elimination of synapses and decreased neural firing is linked to rapid eye movement sleep (REM sleep). Studies of the excitation/inhibition balance in cortical circuits suggest that both processes are connected to a specific inhibitory regulation of cortical principal neurons, characterized by an enhanced perisomatic inhibition via parvalbumin positive (PV+) cells, together with a release from dendritic inhibition by somatostatin positive (SOM+) cells. Such shift towards increased perisomatic inhibition of principal cells appears to be a general motif which underlies the plastic synaptic changes observed during sleep, regardless of whether towards up or downregulation.

Neuroscience-Inspired Artificial Intelligence

The fields of neuroscience and artificial intelligence (AI) have a long and intertwined history. In more recent times, however, communication and collaboration between the two fields has become less commonplace. In this article, we argue that better understanding biological brains could play a vital role in building intelligent machines. We survey historical interactions between the AI and neuroscience fields and emphasize current advances in AI that have been inspired by the study of neural computation in humans and other animals. We conclude by highlighting shared themes that may be key for advancing future research in both fields.

Avoiding Catastrophic Forgetting

Humans regularly perform new learning without losing memory for previous information, but neural network models suffer from the phenomenon of catastrophic forgetting in which new learning impairs prior function. A recent article presents an algorithm that spares learning at synapses important for previously learned function, reducing catastrophic forgetting.

Continual Learning Through Synaptic Intelligence

While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.