Storing structured sparse memories in a multi-modular cortical network model

Memory retrieval dynamics and storage capacity of a modular network model of association cortex with featural decomposition

The primate heteromodal cortex presents an evident functional modularity at a mesoscopic level, with physiological and anatomical evidence pointing to it as likely substrate of long-term memory. In order to investigate some of its properties, a model of multimodular autoassociator is studied. Each of the many modules represents a neocortical functional ensemble of recurrently connected neurons and operates as a Hebbian autoassociator, storing a number of local features which it can recall upon cue. The global memory patterns are made of combinations of features sparsely distributed across the modules. Intermodular connections are modelled as a finite-connectivity random graph. Any pair of features in any respective pair of modules is allowed to be involved in several memory patterns; the coarse-grained modular network dynamics is defined in such a way as to overcome the consequent ambiguity of associations. Effects of long-range homeostatic synaptic scaling on network performance are also assessed. The dynamical process of cued retrieval almost saturates a natural upper bound while producing negligible spurious activation. The extent of finite-size effects on storage capacity is quantitatively evaluated. In the limit of infinite size, the functional relationship between storage capacity and number of features per module reduces to that which other authors found by methods from equilibrium statistical mechanics, which suggests that the origin of the functional form is of a combinatorial nature. In contrast with its apparent inevitability at intramodular level, long-range synaptic scaling results to be of minor relevance to both retrieval and storage capacity, casting doubt on its existence in the neocortex. A conjecture is also posited about how statistical fluctuation of connectivity across the network may underpin spontaneous emergence of semantic hierarchies through learning.

Latching dynamics as a basis for short-term recall

We discuss simple models for the transient storage in short-term memory of cortical patterns of activity, all based on the notion that their recall exploits the natural tendency of the cortex to hop from state to state—latching dynamics. We show that in one such model, and in simple spatial memory tasks we have given to human subjects, short-term memory can be limited to similar low capacity by interference effects, in tasks terminated by errors, and can exhibit similar sublinear scaling, when errors are overlooked. The same mechanism can drive serial recall if combined with weak order-encoding plasticity. Finally, even when storing randomly correlated patterns of activity the network demonstrates correlation-driven latching waves, which are reflected at the outer extremes of pattern space.

What makes short-term memory so poor, that over a minute we tend to forget even phone numbers, if we cannot rehearse or record them electronically? In comparison, long-term memory can be amazingly rich and accurate. Was it so difficult to equip our brain with a short-term memory device of reasonable capacity?

We discuss the hypothesis that instead of an ad hoc device, short-term memory relies on long-term representations, and that the short-term recall of multiple items exploits the natural tendency of the cortex to jump from state to state, by only adding imprecisely determined “kicks” that spur cortical dynamics towards the states representing those items. We show that a plausible neural model for such kicks performs similarly to human subjects we have tested, both in conditions when short-term recall is terminated by errors, and when errors are overlooked and subjects are asked to keep trying. The same mechanism can drive serial recall, if combined with equally imprecise kicks encoding item order. Our analysis suggests that a proper short-term memory device may have never evolved in our brain, which had, therefore, to make do with tweaking its superb long-term memory capabilities.

Short term memory properties of sensory neural architectures

A functional role of the cerebral cortex is to form and hold representations of the sensory world for behavioral purposes. This is achieved by a sheet of neurons, organized in modules called cortical columns, that receives inputs in a peculiar manner, with only a few neurons driven by sensory inputs through thalamic projections, and a vast majority of neurons receiving mainly cortical inputs. How should cortical modules be organized, with respect to sensory inputs, in order for the cortex to efficiently hold sensory representations in memory? To address this question we investigate the memory performance of trees of recurrent networks (TRN) that are composed of recurrent networks, modeling cortical columns, connected with each others through a tree-shaped feed-forward backbone of connections, with sensory stimuli injected at the root of the tree. On these sensory architectures two types of short-term memory (STM) mechanisms can be implemented, STM via transient dynamics on the feed-forward tree, and STM via reverberating activity on the recurrent connectivity inside modules. We derive equations describing the dynamics of such networks, which allow us to thoroughly explore the space of possible architectures and quantify their memory performance. By varying the divergence ratio of the tree, we show that serial architectures, where sensory inputs are successively processed in different modules, are better suited to implement STM via transient dynamics, while parallel architectures, where sensory inputs are simultaneously processed by all modules, are better suited to implement STM via reverberating dynamics.

The Capacity for Correlated Semantic Memories in the Cortex

A statistical analysis of semantic memory should reflect the complex, multifactorial structure of the relations among its items. Still, a dominant paradigm in the study of semantic memory has been the idea that the mental representation of concepts is structured along a simple branching tree spanned by superordinate and subordinate categories. We propose a generative model of item representation with correlations that overcomes the limitations of a tree structure. The items are generated through “factors” that represent semantic features or real-world attributes. The correlation between items has its source in the extent to which items share such factors and the strength of such factors: if many factors are balanced, correlations are overall low; whereas if a few factors dominate, they become strong. Our model allows for correlations that are neither trivial nor hierarchical, but may reproduce the general spectrum of correlations present in a dataset of nouns. We find that such correlations reduce the storage capacity of a Potts network to a limited extent, so that the number of concepts that can be stored and retrieved in a large, human-scale cortical network may still be of order 10⁷, as originally estimated without correlations. When this storage capacity is exceeded, however, retrieval fails completely only for balanced factors; above a critical degree of imbalance, a phase transition leads to a regime where the network still extracts considerable information about the cued item, even if not recovering its detailed representation: partial categorization seems to emerge spontaneously as a consequence of the dominance of particular factors, rather than being imposed ad hoc. We argue this to be a relevant model of semantic memory resilience in Tulving’s remember/know paradigms.

Cortical computations via transient attractors

The ability of sensory networks to transiently store information on the scale of seconds can confer many advantages in processing time-varying stimuli. How a network could store information on such intermediate time scales, between typical neurophysiological time scales and those of long-term memory, is typically attributed to persistent neural activity. An alternative mechanism which might allow for such information storage is through temporary modifications to the neural connectivity which decay on the same second-long time scale as the underlying memories. Earlier work that has explored this method has done so by emphasizing one attractor from a limited, pre-defined set. Here, we describe an alternative, a Transient Attractor network, which can learn any pattern presented to it, store several simultaneously, and robustly recall them on demand using targeted probes in a manner reminiscent of Hopfield networks. We hypothesize that such functionality could be usefully embedded within sensory cortex, and allow for a flexibly-gated short-term memory, as well as conferring the ability of the network to perform automatic de-noising, and separation of input signals into distinct perceptual objects. We demonstrate that the stored information can be refreshed to extend storage time, is not sensitive to noise in the system, and can be turned on or off by simple neuromodulation. The diverse capabilities of transient attractors, as well as their resemblance to many features observed in sensory cortex, suggest the possibility that their actions might underlie neural processing in many sensory areas.