The what and why of binding: the modeler's perspective

Advancing brain-inspired computing with hybrid neural networks

Brain-inspired computing, drawing inspiration from the fundamental structure and information-processing mechanisms of the human brain, has gained significant momentum in recent years. It has emerged as a research paradigm centered on brain-computer dual-driven and multi-network integration. One noteworthy instance of this paradigm is the hybrid neural network (HNN), which integrates computer-science-oriented artificial neural networks (ANNs) with neuroscience-oriented spiking neural networks (SNNs). HNNs exhibit distinct advantages in various intelligent tasks, including perception, cognition and learning. This paper presents a comprehensive review of HNNs with an emphasis on their origin, concepts, biological perspective, construction framework and supporting systems. Furthermore, insights and suggestions for potential research directions are provided aiming to propel the advancement of the HNN paradigm.

Ripple-locked coactivity of stimulus-specific neurons and human associative memory

Associative memory enables the encoding and retrieval of relations between different stimuli. To better understand its neural basis, we investigated whether associative memory involves temporally correlated spiking of medial temporal lobe (MTL) neurons that exhibit stimulus-specific tuning. Using single-neuron recordings from patients with epilepsy performing an associative object-location memory task, we identified the object-specific and place-specific neurons that represented the separate elements of each memory. When patients encoded and retrieved particular memories, the relevant object-specific and place-specific neurons activated together during hippocampal ripples. This ripple-locked coactivity of stimulus-specific neurons emerged over time as the patients' associative learning progressed. Between encoding and retrieval, the ripple-locked timing of coactivity shifted, suggesting flexibility in the interaction between MTL neurons and hippocampal ripples according to behavioral demands. Our results are consistent with a cellular account of associative memory, in which hippocampal ripples coordinate the activity of specialized cellular populations to facilitate links between stimuli.

A Mechanistic Model of Perceptual Binding Predicts That Binding Mechanism Is Robust against Noise

The concept of the brain's own time and space is central to many models and theories that aim to explain how the brain generates consciousness. For example, the temporo-spatial theory of consciousness postulates that the brain implements its own inner time and space for conscious processing of the outside world. Furthermore, our perception and cognition of time and space can be different from actual time and space. This study presents a mechanistic model of mutually connected processes that encode phenomenal representations of space and time. The model is used to elaborate the binding mechanism between two sets of processes representing internal space and time, respectively. Further, a stochastic version of the model is developed to investigate the interplay between binding strength and noise. Spectral entropy is used to characterize noise effects on the systems of interacting processes when the binding strength between them is varied. The stochastic modeling results reveal that the spectral entropy values for strongly bound systems are similar to those for weakly bound or even decoupled systems. Thus, the analysis performed in this study allows us to conclude that the binding mechanism is noise-resilient.

Co-occurring ripple oscillations facilitate neuronal interactions between cortical locations in humans

How the human cortex integrates ("binds") information encoded by spatially distributed neurons remains largely unknown. One hypothesis suggests that synchronous bursts of high-frequency oscillations ("ripples") contribute to binding by facilitating integration of neuronal firing across different cortical locations. While studies have demonstrated that ripples modulate local activity in the cortex, it is not known whether their co-occurrence coordinates neural firing across larger distances. We tested this hypothesis using local field-potentials and single-unit firing from four 96-channel microelectrode arrays in the supragranular cortex of 3 patients. Neurons in co-rippling locations showed increased short-latency co-firing, prediction of each other's firing, and co-participation in neural assemblies. Effects were similar for putative pyramidal and interneurons, during non-rapid eye movement sleep and waking, in temporal and Rolandic cortices, and at distances up to 16 mm (the longest tested). Increased co-prediction during co-ripples was maintained when firing-rate changes were equated, indicating that it was not secondary to non-oscillatory activation. Co-rippling enhanced prediction was strongly modulated by ripple phase, supporting the most common posited mechanism for binding-by-synchrony. Co-ripple enhanced prediction is reciprocal, synergistic with local upstates, and further enhanced when multiple sites co-ripple, supporting re-entrant facilitation. Together, these results support the hypothesis that trans-cortical co-occurring ripples increase the integration of neuronal firing of neurons in different cortical locations and do so in part through phase-modulation rather than unstructured activation.

The processing of proprioceptive signals in distributed networks: insights from insect motor control

The integration of sensory information is required to maintain body posture and to generate robust yet flexible locomotion through unpredictable environments. To anticipate required adaptations in limb posture and enable compensation of sudden perturbations, an animal's nervous system assembles external (exteroception) and internal (proprioception) cues. Coherent neuronal representations of the proprioceptive context of the body and the appendages arise from the concerted action of multiple sense organs monitoring body kinetics and kinematics. This multimodal proprioceptive information, together with exteroceptive signals and brain-derived descending motor commands, converges onto premotor networks - i.e. the local neuronal circuitry controlling motor output and movements - within the ventral nerve cord (VNC), the insect equivalent of the vertebrate spinal cord. This Review summarizes existing knowledge and recent advances in understanding how local premotor networks in the VNC use convergent information to generate contextually appropriate activity, focusing on the example of posture control. We compare the role and advantages of distributed sensory processing over dedicated neuronal pathways, and the challenges of multimodal integration in distributed networks. We discuss how the gain of distributed networks may be tuned to enable the behavioral repertoire of these systems, and argue that insect premotor networks might compensate for their limited neuronal population size by, in comparison to vertebrate networks, relying more heavily on the specificity of their connections. At a time in which connectomics and physiological recording techniques enable anatomical and functional circuit dissection at an unprecedented resolution, insect motor systems offer unique opportunities to identify the mechanisms underlying multimodal integration for flexible motor control.

Heteroclinic networks for brain dynamics

Heteroclinic networks are a mathematical concept in dynamic systems theory that is suited to describe metastable states and switching events in brain dynamics. The framework is sensitive to external input and, at the same time, reproducible and robust against perturbations. Solutions of the corresponding differential equations are spatiotemporal patterns that are supposed to encode information both in space and time coordinates. We focus on the concept of winnerless competition as realized in generalized Lotka-Volterra equations and report on results for binding and chunking dynamics, synchronization on spatial grids, and entrainment to heteroclinic motion. We summarize proposals of how to design heteroclinic networks as desired in view of reproducing experimental observations from neuronal networks and discuss the subtle role of noise. The review is on a phenomenological level with possible applications to brain dynamics, while we refer to the literature for a rigorous mathematical treatment. We conclude with promising perspectives for future research.

Dynamic neural reconstructions of attended object location and features using EEG

Attention allows us to select relevant and ignore irrelevant information from our complex environments. What happens when attention shifts from one item to another? To answer this question, it is critical to have tools that accurately recover neural representations of both feature and location information with high temporal resolution. In the present study, we used human electroencephalography (EEG) and machine learning to explore how neural representations of object features and locations update across dynamic shifts of attention. We demonstrate that EEG can be used to create simultaneous time courses of neural representations of attended features (time point-by-time point inverted encoding model reconstructions) and attended location (time point-by-time point decoding) during both stable periods and across dynamic shifts of attention. Each trial presented two oriented gratings that flickered at the same frequency but had different orientations; participants were cued to attend one of them and on half of trials received a shift cue midtrial. We trained models on a stable period from Hold attention trials and then reconstructed/decoded the attended orientation/location at each time point on Shift attention trials. Our results showed that both feature reconstruction and location decoding dynamically track the shift of attention and that there may be time points during the shifting of attention when 1) feature and location representations become uncoupled and 2) both the previously attended and currently attended orientations are represented with roughly equal strength. The results offer insight into our understanding of attentional shifts, and the noninvasive techniques developed in the present study lend themselves well to a wide variety of future applications.NEW & NOTEWORTHY We used human EEG and machine learning to reconstruct neural response profiles during dynamic shifts of attention. Specifically, we demonstrated that we could simultaneously read out both location and feature information from an attended item in a multistimulus display. Moreover, we examined how that readout evolves over time during the dynamic process of attentional shifts. These results provide insight into our understanding of attention, and this technique carries substantial potential for versatile extensions and applications.

Dynamic saccade context triggers more stable object-location binding

Our visual systems rapidly perceive and integrate information about object identities and locations. There is long-standing debate about how we achieve world-centered (spatiotopic) object representations across eye movements, with many studies reporting persistent retinotopic (eye-centered) effects even for higher-level object-location binding. But these studies are generally conducted in fairly static experimental contexts. Might spatiotopic object-location binding only emerge in more dynamic saccade contexts? In the present study, we investigated this using the Spatial Congruency Bias paradigm in healthy adults. In the static (single saccade) context, we found purely retinotopic binding, as before. However, robust spatiotopic binding emerged in the dynamic (multiple frequent saccades) context. We further isolated specific factors that modulate retinotopic and spatiotopic binding. Our results provide strong evidence that dynamic saccade context can trigger more stable object-location binding in ecologically-relevant spatiotopic coordinates, perhaps via a more flexible brain state which accommodates improved visual stability in the dynamic world.

Co-occurring ripple oscillations facilitate neuronal interactions between cortical locations in humans

Synchronous bursts of high frequency oscillations ('ripples') are hypothesized to contribute to binding by facilitating integration of neuronal firing across cortical locations. We tested this hypothesis using local field-potentials and single-unit firing from four 96-channel microelectrode arrays in supragranular cortex of 3 patients. Neurons in co-rippling locations showed increased short-latency co-firing, prediction of each-other's firing, and co-participation in neural assemblies. Effects were similar for putative pyramidal and interneurons, during NREM sleep and waking, in temporal and Rolandic cortices, and at distances up to 16mm. Increased co-prediction during co-ripples was maintained when firing-rate changes were equated, and were strongly modulated by ripple phase. Co-ripple enhanced prediction is reciprocal, synergistic with local upstates, and further enhanced when multiple sites co-ripple. Together, these results support the hypothesis that trans-cortical co-ripples increase the integration of neuronal firing of neurons in different cortical locations, and do so in part through phase-modulation rather than unstructured activation.

Generating parallel representations of position and identity in the olfactory system

In Drosophila, a dedicated olfactory channel senses a male pheromone, cis-vaccenyl acetate (cVA), promoting female courtship while repelling males. Here, we show that separate cVA-processing streams extract qualitative and positional information. cVA sensory neurons respond to concentration differences in a 5-mm range around a male. Second-order projection neurons encode the angular position of a male by detecting inter-antennal differences in cVA concentration, which are amplified through contralateral inhibition. At the third circuit layer, we identify 47 cell types with diverse input-output connectivity. One population responds tonically to male flies, a second is tuned to olfactory looming, while a third integrates cVA and taste to coincidentally promote female mating. The separation of olfactory features resembles the mammalian what and where visual streams; together with multisensory integration, this enables behavioral responses appropriate to specific ethological contexts.

Complex economic decisions from simple neurocognitive processes: the role of interactive attention

Neurocognitive theories of value-based choice propose that people additively accumulate choice attributes when making decisions. These theories cannot explain the emergence of complex multiplicative preferences such as those assumed by prospect theory and other economic models. We investigate an interactive attention mechanism, according to which attention to attributes (like payoffs) depends on other attributes (like probabilities) attended to previously. We formalize this mechanism using a Markov attention model combined with an accumulator decision process, and test our model on eye-tracking and mouse-tracking data in risky choice. Our tests show that interactive attention is necessary to make good choices, that most participants display interactive attention and that allowing for interactive attention in accumulation-based decision models improves their predictions. By equipping established decision models with sophisticated attentional dynamics, we extend these models to describe complex economic choice, and in the process, we unify two prominent theoretical approaches to studying value-based decision making.

Widespread ripples synchronize human cortical activity during sleep, waking, and memory recall

Declarative memory encoding, consolidation, and retrieval require the integration of elements encoded in widespread cortical locations. The mechanism whereby such "binding" of different components of mental events into unified representations occurs is unknown. The "binding-by-synchrony" theory proposes that distributed encoding areas are bound by synchronous oscillations enabling enhanced communication. However, evidence for such oscillations is sparse. Brief high-frequency oscillations ("ripples") occur in the hippocampus and cortex and help organize memory recall and consolidation. Here, using intracranial recordings in humans, we report that these ∼70-ms-duration, 90-Hz ripples often couple (within ±500 ms), co-occur (≥ 25-ms overlap), and, crucially, phase-lock (have consistent phase lags) between widely distributed focal cortical locations during both sleep and waking, even between hemispheres. Cortical ripple co-occurrence is facilitated through activation across multiple sites, and phase locking increases with more cortical sites corippling. Ripples in all cortical areas co-occur with hippocampal ripples but do not phase-lock with them, further suggesting that cortico-cortical synchrony is mediated by cortico-cortical connections. Ripple phase lags vary across sleep nights, consistent with participation in different networks. During waking, we show that hippocampo-cortical and cortico-cortical coripples increase preceding successful delayed memory recall, when binding between the cue and response is essential. Ripples increase and phase-modulate unit firing, and coripples increase high-frequency correlations between areas, suggesting synchronized unit spiking facilitating information exchange. co-occurrence, phase synchrony, and high-frequency correlation are maintained with little decrement over very long distances (25 cm). Hippocampo-cortico-cortical coripples appear to possess the essential properties necessary to support binding by synchrony during memory retrieval and perhaps generally in cognition.

Consciousness, Cognition and the Neuronal Cytoskeleton - A New Paradigm Needed in Neuroscience

Viewing the brain as a complex computer of simple neurons cannot account for consciousness nor essential features of cognition. Single cell organisms with no synapses perform purposeful intelligent functions using their cytoskeletal microtubules. A new paradigm is needed to view the brain as a scale-invariant hierarchy extending both upward from the level of neurons to larger and larger neuronal networks, but also downward, inward, to deeper, faster quantum and classical processes in cytoskeletal microtubules inside neurons. Evidence shows self-similar patterns of conductive resonances repeating in terahertz, gigahertz, megahertz, kilohertz and hertz frequency ranges in microtubules. These conductive resonances apparently originate in terahertz quantum dipole oscillations and optical interactions among pi electron resonance clouds of aromatic amino acid rings of tryptophan, phenylalanine and tyrosine within each tubulin, the component subunit of microtubules, and the brain's most abundant protein. Evidence from cultured neuronal networks also now shows that gigahertz and megahertz oscillations in dendritic-somatic microtubules regulate specific firings of distal axonal branches, causally modulating membrane and synaptic activities. The brain should be viewed as a scale-invariant hierarchy, with quantum and classical processes critical to consciousness and cognition originating in microtubules inside neurons.

A Dynamic Mechanistic Model of Perceptual Binding

The brain’s ability to create a unified conscious representation of an object by integrating information from multiple perception pathways is called perceptual binding. Binding is crucial for normal cognitive function. Some perceptual binding errors and disorders have been linked to certain neurological conditions, brain lesions, and conditions that give rise to illusory conjunctions. However, the mechanism of perceptual binding remains elusive. Here, I present a computational model of binding using two sets of coupled oscillatory processes that are assumed to occur in response to two different percepts. I use the model to study the dynamic behavior of coupled processes to characterize how these processes can modulate each other and reach a temporal synchrony. I identify different oscillatory dynamic regimes that depend on coupling mechanisms and parameter values. The model can also discriminate different combinations of initial inputs that are set by initial states of coupled processes. Decoding brain signals that are formed through perceptual binding is a challenging task, but my modeling results demonstrate how crosstalk between two systems of processes can possibly modulate their outputs. Therefore, my mechanistic model can help one gain a better understanding of how crosstalk between perception pathways can affect the dynamic behavior of the systems that involve perceptual binding.

Linking Brain Structure, Activity, and Cognitive Function through Computation

Understanding the human brain is a “Grand Challenge” for 21st century research. Computational approaches enable large and complex datasets to be addressed efficiently, supported by artificial neural networks, modeling and simulation. Dynamic generative multiscale models, which enable the investigation of causation across scales and are guided by principles and theories of brain function, are instrumental for linking brain structure and function. An example of a resource enabling such an integrated approach to neuroscientific discovery is the BigBrain, which spatially anchors tissue models and data across different scales and ensures that multiscale models are supported by the data, making the bridge to both basic neuroscience and medicine. Research at the intersection of neuroscience, computing and robotics has the potential to advance neuro-inspired technologies by taking advantage of a growing body of insights into perception, plasticity and learning. To render data, tools and methods, theories, basic principles and concepts interoperable, the Human Brain Project (HBP) has launched EBRAINS, a digital neuroscience research infrastructure, which brings together a transdisciplinary community of researchers united by the quest to understand the brain, with fascinating insights and perspectives for societal benefits.

Bi-Stable Perception: Self-Coordinating Brain Regions to Make-Up the Mind

Bi-stable perception is a strong instance of cognitive self-organization, providing a research model for how ‘the brain makes up its mind.’ The complexity of perceptual bistability prevents a simple attribution of functions to areas, because many cognitive processes, recruiting multiple brain regions, are simultaneously involved. The functional magnetic resonance imaging (fMRI) evidence suggests the activation of a large network of distant brain areas. Concurrently, electroencephalographic and magnetoencephalographic (MEEG) literature shows sub second oscillatory activity and phase synchrony on several frequency bands. Strongly represented are beta and gamma bands, often associated with neural/cognitive integration processes. The spatial extension and short duration of brain activities suggests the need for a fast, large-scale neural coordination mechanism. To address the range of temporo-spatial scales involved, we systematize the current knowledge from mathematical models, cognitive sciences and neuroscience at large, from single-cell- to system-level research, including evidence from human and non-human primates. Surprisingly, despite evidence spanning through different organization levels, models, and experimental approaches, the scarcity of integrative studies is evident. In a final section of the review we dwell on the reasons behind such scarcity and on the need of integration in order to achieve a real understanding of the complexities underlying bi-stable perception processes.

No Evidence for Entrainment: Endogenous Gamma Oscillations and Rhythmic Flicker Responses Coexist in Visual Cortex

Over the past decades, numerous studies have linked cortical gamma oscillations (∼30-100 Hz) to neurocomputational mechanisms. Their functional relevance, however, is still passionately debated. Here, we asked whether endogenous gamma oscillations in the human brain can be entrained by a rhythmic photic drive >50 Hz. Such a noninvasive modulation of endogenous brain rhythms would allow conclusions about their causal involvement in neurocognition. To this end, we systematically investigated oscillatory responses to a rapid sinusoidal flicker in the absence and presence of endogenous gamma oscillations using magnetoencephalography (MEG) in combination with a high-frequency projector. The photic drive produced a robust response over visual cortex to stimulation frequencies of up to 80 Hz. Strong, endogenous gamma oscillations were induced using moving grating stimuli as repeatedly done in previous research. When superimposing the flicker and the gratings, there was no evidence for phase or frequency entrainment of the endogenous gamma oscillations by the photic drive. Unexpectedly, we did not observe an amplification of the flicker response around participants' individual gamma frequencies (IGFs); rather, the magnitude of the response decreased monotonically with increasing frequency. Source reconstruction suggests that the flicker response and the gamma oscillations were produced by separate, coexistent generators in visual cortex. The presented findings challenge the notion that cortical gamma oscillations can be entrained by rhythmic visual stimulation. Instead, the mechanism generating endogenous gamma oscillations seems to be resilient to external perturbation.SIGNIFICANCE STATEMENT We aimed to investigate to what extent ongoing, high-frequency oscillations in the gamma-band (30-100 Hz) in the human brain can be entrained by a visual flicker. Gamma oscillations have long been suggested to coordinate neuronal firing and enable interregional communication. Our results demonstrate that rhythmic visual stimulation cannot hijack the dynamics of ongoing gamma oscillations; rather, the flicker response and the endogenous gamma oscillations coexist in different visual areas. Therefore, while a visual flicker evokes a strong neuronal response even at high frequencies in the gamma-band, it does not entrain endogenous gamma oscillations in visual cortex. This has important implications for interpreting studies investigating the causal and neuroprotective effects of rhythmic sensory stimulation in the gamma-band.

Visual Remapping

Our visual system is fundamentally retinotopic. When viewing a stable scene, each eye movement shifts object features and locations on the retina. Thus, sensory representations must be updated, or remapped, across saccades to align presaccadic and postsaccadic inputs. The earliest remapping studies focused on anticipatory, presaccadic shifts of neuronal spatial receptive fields. Over time, it has become clear that there are multiple forms of remapping and that different forms of remapping may be mediated by different neural mechanisms. This review attempts to organize the various forms of remapping into a functional taxonomy based on experimental data and ongoing debates about forward versus convergent remapping, presaccadic versus postsaccadic remapping, and spatial versus attentional remapping. We integrate findings from primate neurophysiological, human neuroimaging and behavioral, and computational modeling studies. We conclude by discussing persistent open questions related to remapping, with specific attention to binding of spatial and featural information during remapping and speculations about remapping's functional significance. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

The visual cortex produces gamma band echo in response to broadband visual flicker

The aim of this study is to uncover the network dynamics of the human visual cortex by driving it with a broadband random visual flicker. We here applied a broadband flicker (1-720 Hz) while measuring the MEG and then estimated the temporal response function (TRF) between the visual input and the MEG response. This TRF revealed an early response in the 40-60 Hz gamma range as well as in the 8-12 Hz alpha band. While the gamma band response is novel, the latter has been termed the alpha band perceptual echo. The gamma echo preceded the alpha perceptual echo. The dominant frequency of the gamma echo was subject-specific thereby reflecting the individual dynamical properties of the early visual cortex. To understand the neuronal mechanisms generating the gamma echo, we implemented a pyramidal-interneuron gamma (PING) model that produces gamma oscillations in the presence of constant input currents. Applying a broadband input current mimicking the visual stimulation allowed us to estimate TRF between the input current and the population response (akin to the local field potentials). The TRF revealed a gamma echo that was similar to the one we observed in the MEG data. Our results suggest that the visual gamma echo can be explained by the dynamics of the PING model even in the absence of sustained gamma oscillations.

The Hierarchical Evolution in Human Vision Modeling

Computational models of primate vision took a significant advance with David Marr's tripartite separation of the vision enterprise into the problem formulation, algorithm, and neural implementation; however, many subsequent parallel developments in robotics and modeling greatly refined the algorithm descriptions into very distinct levels that complement each other. This review traces the time course of these developments and shows how the current perspective evolved to have its alternative internal hierarchical organization.

Is Neuromorphic MNIST Neuromorphic? Analyzing the Discriminative Power of Neuromorphic Datasets in the Time Domain

A major characteristic of spiking neural networks (SNNs) over conventional artificial neural networks (ANNs) is their ability to spike, enabling them to use spike timing for coding and efficient computing. In this paper, we assess if neuromorphic datasets recorded from static images are able to evaluate the ability of SNNs to use spike timings in their calculations. We have analyzed N-MNIST, N-Caltech101 and DvsGesture along these lines, but focus our study on N-MNIST. First we evaluate if additional information is encoded in the time domain in a neuromorphic dataset. We show that an ANN trained with backpropagation on frame-based versions of N-MNIST and N-Caltech101 images achieve 99.23 and 78.01% accuracy. These are comparable to the state of the art-showing that an algorithm that purely works on spatial data can classify these datasets. Second we compare N-MNIST and DvsGesture on two STDP algorithms, RD-STDP, that can classify only spatial data, and STDP-tempotron that classifies spatiotemporal data. We demonstrate that RD-STDP performs very well on N-MNIST, while STDP-tempotron performs better on DvsGesture. Since DvsGesture has a temporal dimension, it requires STDP-tempotron, while N-MNIST can be adequately classified by an algorithm that works on spatial data alone. This shows that precise spike timings are not important in N-MNIST. N-MNIST does not, therefore, highlight the ability of SNNs to classify temporal data. The conclusions of this paper open the question-what dataset can evaluate SNN ability to classify temporal data?

Network mechanism for insect olfaction

Early olfactory pathway responses to the presentation of an odor exhibit remarkably similar dynamical behavior across phyla from insects to mammals, and frequently involve transitions among quiescence, collective network oscillations, and asynchronous firing. We hypothesize that the time scales of fast excitation and fast and slow inhibition present in these networks may be the essential element underlying this similar behavior, and design an idealized, conductance-based integrate-and-fire model to verify this hypothesis via numerical simulations. To better understand the mathematical structure underlying the common dynamical behavior across species, we derive a firing-rate model and use it to extract a slow passage through a saddle-node-on-an-invariant-circle bifurcation structure. We expect this bifurcation structure to provide new insights into the understanding of the dynamical behavior of neuronal assemblies and that a similar structure can be found in other sensory systems.

Resonator Networks, 1: An Efficient Solution for Factoring High-Dimensional, Distributed Representations of Data Structures

The ability to encode and manipulate data structures with distributed neural representations could qualitatively enhance the capabilities of traditional neural networks by supporting rule-based symbolic reasoning, a central property of cognition. Here we show how this may be accomplished within the framework of Vector Symbolic Architectures (VSAs) (Plate, 1991; Gayler, 1998; Kanerva, 1996), whereby data structures are encoded by combining high-dimensional vectors with operations that together form an algebra on the space of distributed representations. In particular, we propose an efficient solution to a hard combinatorial search problem that arises when decoding elements of a VSA data structure: the factorization of products of multiple codevectors. Our proposed algorithm, called a resonator network, is a new type of recurrent neural network that interleaves VSA multiplication operations and pattern completion. We show in two examples-parsing of a tree-like data structure and parsing of a visual scene-how the factorization problem arises and how the resonator network can solve it. More broadly, resonator networks open the possibility of applying VSAs to myriad artificial intelligence problems in real-world domains. The companion article in this issue (Kent, Frady, Sommer, & Olshausen, 2020) presents a rigorous analysis and evaluation of the performance of resonator networks, showing it outperforms alternative approaches.

Negative emotions in the target speaker's voice enhance speech recognition under "cocktail-party" environments

Under a "cocktail-party" environment with simultaneous multiple talkers, recognition of target speech is effectively improved by a number of perceptually unmasking cues. It remains unclear whether emotions embedded in the target-speaker's voice can either improve speech perception alone or interact with other cues facilitating speech perception against a masker background. This study used two target-speaker voices with different emotional valences to examine whether recognition of target speech is modulated by the emotional valence when the target speech and the maskers were perceptually co-located or separated. The results showed that both the speech recognition against the masker background and the separation-induced unmasking effect were higher for the target speaker with a negatively emotional voice than for the target speaker with a positively emotional voice. Moreover, when the negative voice was fear conditioned, the target-speech recognition was further improved against speech informational masking. These results suggested that the emotionally vocal unmasking cue interacts significantly with the perceived spatial-separation unmasking cue, facilitating the unmasking effect against a masking background. Thus, emotional features embedded in the target-speaker's vocal timbre are also useful for unmasking the target speech in "cocktail-party" environments.

A Compositional Neural Architecture for Language

Hierarchical structure and compositionality imbue human language with unparalleled expressive power and set it apart from other perception–action systems. However, neither formal nor neurobiological models account for how these defining computational properties might arise in a physiological system. I attempt to reconcile hierarchy and compositionality with principles from cell assembly computation in neuroscience; the result is an emerging theory of how the brain could convert distributed perceptual representations into hierarchical structures across multiple timescales while representing interpretable incremental stages of (de)compositional meaning. The model's architecture—a multidimensional coordinate system based on neurophysiological models of sensory processing—proposes that a manifold of neural trajectories encodes sensory, motor, and abstract linguistic states. Gain modulation, including inhibition, tunes the path in the manifold in accordance with behavior and is how latent structure is inferred. As a consequence, predictive information about upcoming sensory input during production and comprehension is available without a separate operation. The proposed processing mechanism is synthesized from current models of neural entrainment to speech, concepts from systems neuroscience and category theory, and a symbolic-connectionist computational model that uses time and rhythm to structure information. I build on evidence from cognitive neuroscience and computational modeling that suggests a formal and mechanistic alignment between structure building and neural oscillations, and moves toward unifying basic insights from linguistics and psycholinguistics with the currency of neural computation.

A neurodynamic model of the interaction between color perception and color memory

The memory color effect and Spanish castle illusion have been taken as evidence of the cognitive penetrability of vision. In the same manner, the successful decoding of color-related brain signals in functional neuroimaging studies suggests the retrieval of memory colors associated with a perceived gray object. Here, we offer an alternative account of these findings based on the design principles of adaptive resonance theory (ART). In ART, conscious perception is a consequence of a resonant state. Resonance emerges in a recurrent cortical circuit when a bottom-up spatial pattern agrees with the top-down expectation. When they do not agree, a special control mechanism is activated that resets the network and clears off erroneous expectation, thus allowing the bottom-up activity to always dominate in perception. We developed a color ART circuit and evaluated its behavior in computer simulations. The model helps to explain how traces of erroneous expectations about incoming color are eventually removed from the color perception, although their transient effect may be visible in behavioral responses or in brain imaging. Our results suggest that the color ART circuit, as a predictive computational system, is almost never penetrable, because it is equipped with computational mechanisms designed to constrain the impact of the top-down predictions on ongoing perceptual processing.

The temporally-integrated causality landscape: A theoretical framework for consciousness and meaning

Theoretical approaches to understanding consciousness have begun to converge upon areas of general agreement, yet substantive differences remain. Here, I introduce a new theoretical framework for the emergence of consciousness from the functional integration of the thalamocortical system: the Temporally-Integrated Causality Landscape (TICL). TICL presents a novel perspective which addresses important phenomenological characteristics of consciousness that other frameworks, such as IIT, do not. First, the TICL is based upon the observation that conscious experiences are temporally continuous, not discrete. Secondly, the TICL establishes a thalamocortical basis for the point-of-view. According to TICL, consciousness is composed of contents that arise from neuronal subsystems that have meaning from the point-of-view of the larger, integrated system in which they are nested. Meaningful contents emerge from the subsystems because they exhibit a level of temporally-integrated causality (TIC) that is distinguishable from that of the larger system.

Improving the Izhikevich Model Based on Rat Basolateral Amygdala and Hippocampus Neurons, and Recognizing Their Possible Firing Patterns

Introduction

Identifying the potential firing patterns following different brain regions under normal and abnormal conditions increases our understanding of events at the level of neural interactions in the brain. Furthermore, it is important to be capable of modeling the potential neural activities to build precise artificial neural networks. The Izhikevich model is one of the simplest biologically-plausible models, i.e. capable of capturing most recognized firing patterns of neurons. This property makes the model efficient in simulating the large-scale networks of neurons. Improving the Izhikevich model for adapting with the neuronal activity of rat brain with great accuracy would make the model effective for future neural network implementations.

Methods

Data sampling from two brain regions, the HIP and BLA, was performed by the extracellular recordings of male Wistar rats, and spike sorting was conducted by Plexon offline sorter. Further analyses were performed through NeuroExplorer and MATLAB. To optimize the Izhikevich model parameters, a genetic algorithm was used. In this algorithm, optimization tools, like crossover and mutation, provide the basis for generating model parameters populations. The process of comparison in each iteration leads to the survival of better populations until achieving the optimum solution.

Results

In the present study, the possible firing patterns of the real single neurons of the HIP and BLA were identified. Additionally, an improved Izhikevich model was achieved. Accordingly, the real neuronal spiking pattern of these regions' neurons and the corresponding cases of the Izhikevich neuron spiking pattern were adjusted with great accuracy.

Conclusion

This study was conducted to elevate our knowledge of neural interactions in different structures of the brain and accelerate the quality of future large-scale neural networks simulations, as well as reducing the modeling complexity. This aim was achievable by performing the improved Izhikevich model, and inserting only the plausible firing patterns and eliminating unrealistic ones.

Network architectures supporting learnability

Human learners acquire complex interconnected networks of relational knowledge. The capacity for such learning naturally depends on two factors: the architecture (or informational structure) of the knowledge network itself and the architecture of the computational unit-the brain-that encodes and processes the information. That is, learning is reliant on integrated network architectures at two levels: the epistemic and the computational, or the conceptual and the neural. Motivated by a wish to understand conventional human knowledge, here, we discuss emerging work assessing network constraints on the learnability of relational knowledge, and theories from statistical physics that instantiate the principles of thermodynamics and information theory to offer an explanatory model for such constraints. We then highlight similarities between those constraints on the learnability of relational networks, at one level, and the physical constraints on the development of interconnected patterns in neural systems, at another level, both leading to hierarchically modular networks. To support our discussion of these similarities, we employ an operational distinction between the modeller (e.g. the human brain), the model (e.g. a single human's knowledge) and the modelled (e.g. the information present in our experiences). We then turn to a philosophical discussion of whether and how we can extend our observations to a claim regarding explanation and mechanism for knowledge acquisition. What relation between hierarchical networks, at the conceptual and neural levels, best facilitate learning? Are the architectures of optimally learnable networks a topological reflection of the architectures of comparably developed neural networks? Finally, we contribute to a unified approach to hierarchies and levels in biological networks by proposing several epistemological norms for analysing the computational brain and social epistemes, and for developing pedagogical principles conducive to curious thought. This article is part of the theme issue 'Unifying the essential concepts of biological networks: biological insights and philosophical foundations'.

Spatial congruency bias in identifying objects is triggered by retinal position congruence: Examination using the Ternus-Pikler illusion

When two different objects are sequentially presented at the same location, the viewer tends to misjudge them as identical (spatial congruency bias). The present study examined whether the spatial congruency bias would involve not only retinotopic but also non-retinotopic processing using the Ternus-Pikler illusion. In the experiments, two objects (central and peripheral) appeared in an initial frame. The target object was presented in the central area of the display, while the peripheral object was either on the left or right side of the target object. In the second frame, the target object was again presented in the central area, and the peripheral object was on the opposite side. Two kinds of inter-stimulus intervals were used. In the no-blank condition, the target object was perceived as stationary, and the peripheral object appeared to move to the opposite side. However, in the long-blank condition, the two objects were perceived to move together. Participants judged whether the target objects in the two frames were identical. As a result, the spatial congruency bias occurred irrespective of the ISI conditions. Our findings suggest that the spatial congruency bias is mainly based on retinotopic processing.

Acetylcholine Mediates Dynamic Switching Between Information Coding Schemes in Neuronal Networks

Rate coding and phase coding are the two major coding modes seen in the brain. For these two modes, network dynamics must either have a wide distribution of frequencies for rate coding, or a narrow one to achieve stability in phase dynamics for phase coding. Acetylcholine (ACh) is a potent regulator of neural excitability. Acting through the muscarinic receptor, ACh reduces the magnitude of the potassium M-current, a hyperpolarizing current that builds up as neurons fire. The M-current contributes to several excitability features of neurons, becoming a major player in facilitating the transition between Type 1 (integrator) and Type 2 (resonator) excitability. In this paper we argue that this transition enables a dynamic switch between rate coding and phase coding as levels of ACh release change. When a network is in a high ACh state variations in synaptic inputs will lead to a wider distribution of firing rates across the network and this distribution will reflect the network structure or pattern of external input to the network. When ACh is low, network frequencies become narrowly distributed and the structure of a network or pattern of external inputs will be represented through phase relationships between firing neurons. This work provides insights into how modulation of neuronal features influences network dynamics and information processing across brain states.

From cells to sensations: A window to the physics of mind

Principles of methods for studying particles and fields that cannot be sensed by third-person observers by routine methods can be used to understand the physics of first-person properties of mind. Accordingly, whenever a system exhibits disparate features at multiple levels, unique combination of constraints offered by them direct us towards a solution that will be the first principle of that system. Using this method, it was possible to arrive at a third-person observable solution-point of brain-mind interface. Examination of this location identified a set of unique features that can allow an associatively learned (cue) stimulus to spark hallucinations that form units of first-person internal (inner) sensations reminiscent of stimuli from the associatively learned second item in timescales of milliseconds. It allows us to peep into a virtual space of mind where different modifications and integrations of units of internal sensations generate their different net conformations ranging from perception to an inner sense of hidden relationships that form a hypothesis. Since sparking of inner sensations of the late arriving (when far away) or non-arriving (when hidden) features of items started providing survival advantage, the focus of evolution might have been to optimize this property. Hence, the circuity that generates it can be considered as the primary circuitry of the system. The solution provides several testable predictions. By taking readers through the process of deriving the solution and by explaining how it interconnects disparate findings, it is hoped that the factors determining the physics of mind will become evident.

Theory of Event Coding (TEC) V2.0: Representing and controlling perception and action

This article provides an update of the Theory of Event Coding (TEC), which claims that perception and action are identical processes operating on the same codes - event files consisting of integrated networks of sensorimotor feature codes. The original version of the theory emphasized its representational underpinnings, but recent theoretical developments provide the basis for a more integrated view consisting of both the codes that are shared between perception and action in the control processes operating on these codes. Four developments are discussed in more detail: The degree to which the integration and retrieval of event files depends on current goals, how metacontrol states impact the handling of event files, how feature binding relates to event learning, and how the integration of non-social events relates to the integration of social events. Case examples using various versions of the Simon task are used to explain how the new version of TEC explains interactions between perception and action in non-social and social situations.

Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs

Humans can infer concepts from image pairs and apply those in the physical world in a completely different setting, enabling tasks like IKEA assembly from diagrams. If robots could represent and infer high-level concepts, then it would notably improve their ability to understand our intent and to transfer tasks between different environments. To that end, we introduce a computational framework that replicates aspects of human concept learning. Concepts are represented as programs on a computer architecture consisting of a visual perception system, working memory, and action controller. The instruction set of this cognitive computer has commands for parsing a visual scene, directing gaze and attention, imagining new objects, manipulating the contents of a visual working memory, and controlling arm movement. Inferring a concept corresponds to inducing a program that can transform the input to the output. Some concepts require the use of imagination and recursion. Previously learned concepts simplify the learning of subsequent, more elaborate concepts and create a hierarchy of abstractions. We demonstrate how a robot can use these abstractions to interpret novel concepts presented to it as schematic images and then apply those concepts in very different situations. By bringing cognitive science ideas on mental imagery, perceptual symbols, embodied cognition, and deictic mechanisms into the realm of machine learning, our work brings us closer to the goal of building robots that have interpretable representations and common sense.

A new approach to solving the feature-binding problem in primate vision

We discuss a recently proposed approach to solve the classic feature-binding problem in primate vision that uses neural dynamics known to be present within the visual cortex. Broadly, the feature-binding problem in the visual context concerns not only how a hierarchy of features such as edges and objects within a scene are represented, but also the hierarchical relationships between these features at every spatial scale across the visual field. This is necessary for the visual brain to be able to make sense of its visuospatial world. Solving this problem is an important step towards the development of artificial general intelligence. In neural network simulation studies, it has been found that neurons encoding the binding relations between visual features, known as binding neurons, emerge during visual training when key properties of the visual cortex are incorporated into the models. These biological network properties include (i) bottom-up, lateral and top-down synaptic connections, (ii) spiking neuronal dynamics, (iii) spike timing-dependent plasticity, and (iv) a random distribution of axonal transmission delays (of the order of several milliseconds) in the propagation of spikes between neurons. After training the network on a set of visual stimuli, modelling studies have reported observing the gradual emergence of polychronization through successive layers of the network, in which subpopulations of neurons have learned to emit their spikes in regularly repeating spatio-temporal patterns in response to specific visual stimuli. Such a subpopulation of neurons is known as a polychronous neuronal group (PNG). Some neurons embedded within these PNGs receive convergent inputs from neurons representing lower- and higher-level visual features, and thus appear to encode the hierarchical binding relationship between features. Neural activity with this kind of spatio-temporal structure robustly emerges in the higher network layers even when neurons in the input layer represent visual stimuli with spike timings that are randomized according to a Poisson distribution. The resulting hierarchical representation of visual scenes in such models, including the representation of hierarchical binding relations between lower- and higher-level visual features, is consistent with the hierarchical phenomenology or subjective experience of primate vision and is distinct from approaches interested in segmenting a visual scene into a finite set of objects.

An oscillatory neural network model that demonstrates the benefits of multisensory learning

Since the world consists of objects that stimulate multiple senses, it is advantageous for a vertebrate to integrate all the sensory information available. However, the precise mechanisms governing the temporal dynamics of multisensory processing are not well understood. We develop a computational modeling approach to investigate these mechanisms. We present an oscillatory neural network model for multisensory learning based on sparse spatio-temporal encoding. Recently published results in cognitive science show that multisensory integration produces greater and more efficient learning. We apply our computational model to qualitatively replicate these results. We vary learning protocols and system dynamics, and measure the rate at which our model learns to distinguish superposed presentations of multisensory objects. We show that the use of multiple channels accelerates learning and recall by up to 80%. When a sensory channel becomes disabled, the performance degradation is less than that experienced during the presentation of non-congruent stimuli. This research furthers our understanding of fundamental brain processes, paving the way for multiple advances including the building of machines with more human-like capabilities.

Adaptive scales of integration and response latencies in a critically-balanced model of the primary visual cortex

The primary visual cortex (V1) integrates information over scales in visual space, which have been shown to vary, in an input-dependent manner, as a function of contrast and other visual parameters. Which algorithms the brain uses to achieve this feat are largely unknown and an open problem in visual neuroscience. We demonstrate that a simple dynamical mechanism can account for this contrast-dependent scale of integration in visuotopic space as well as connect this property to two other stimulus-dependent features of V1: extents of lateral integration on the cortical surface and response latencies.

Resonance with subthreshold oscillatory drive organizes activity and optimizes learning in neural networks

Networks of neurons need to reliably encode and replay patterns and sequences of activity. In the brain, sequences of spatially coding neurons are replayed in both the forward and reverse direction in time with respect to their order in recent experience. As of yet there is no network-level or biophysical mechanism known that can produce both modes of replay within the same network. Here we propose that resonance, a property of neurons, paired with subthreshold oscillations in neural input facilitate network-level learning of fixed and sequential activity patterns and lead to both forward and reverse replay.

Network oscillations across and within brain areas are critical for learning and performance of memory tasks. While a large amount of work has focused on the generation of neural oscillations, their effect on neuronal populations’ spiking activity and information encoding is less known. Here, we use computational modeling to demonstrate that a shift in resonance responses can interact with oscillating input to ensure that networks of neurons properly encode new information represented in external inputs to the weights of recurrent synaptic connections. Using a neuronal network model, we find that due to an input current-dependent shift in their resonance response, individual neurons in a network will arrange their phases of firing to represent varying strengths of their respective inputs. As networks encode information, neurons fire more synchronously, and this effect limits the extent to which further “learning” (in the form of changes in synaptic strength) can occur. We also demonstrate that sequential patterns of neuronal firing can be accurately stored in the network; these sequences are later reproduced without external input (in the context of subthreshold oscillations) in both the forward and reverse directions (as has been observed following learning in vivo). To test whether a similar mechanism could act in vivo, we show that periodic stimulation of hippocampal neurons coordinates network activity and functional connectivity in a frequency-dependent manner. We conclude that resonance with subthreshold oscillations provides a plausible network-level mechanism to accurately encode and retrieve information without overstrengthening connections between neurons.

Development of novel tasks for studying view-invariant object recognition in rodents: Sensitivity to scopolamine

The capacity to recognize objects from different view-points or angles, referred to as view-invariance, is an essential process that humans engage in daily. Currently, the ability to investigate the neurobiological underpinnings of this phenomenon is limited, as few ethologically valid view-invariant object recognition tasks exist for rodents. Here, we report two complementary, novel view-invariant object recognition tasks in which rodents physically interact with three-dimensional objects. Prior to experimentation, rats and mice were given extensive experience with a set of 'pre-exposure' objects. In a variant of the spontaneous object recognition task, novelty preference for pre-exposed or new objects was assessed at various angles of rotation (45°, 90° or 180°); unlike control rodents, for whom the objects were novel, rats and mice tested with pre-exposed objects did not discriminate between rotated and un-rotated objects in the choice phase, indicating substantial view-invariant object recognition. Secondly, using automated operant touchscreen chambers, rats were tested on pre-exposed or novel objects in a pairwise discrimination task, where the rewarded stimulus (S+) was rotated (180°) once rats had reached acquisition criterion; rats tested with pre-exposed objects re-acquired the pairwise discrimination following S+ rotation more effectively than those tested with new objects. Systemic scopolamine impaired performance on both tasks, suggesting involvement of acetylcholine at muscarinic receptors in view-invariant object processing. These tasks present novel means of studying the behavioral and neural bases of view-invariant object recognition in rodents.

The temporal dynamics involved in object representation updating to predict change

The synchronization of cortically disparate neural assemblies at frequencies in the gamma-band range (30-70Hz) is considered to be involved in the perceptual organization of the environment. In support of this Elliott (2014) demonstrated improved detection of a target stimulus when this target was primed in a matrix that flickered at specific frequencies in the gamma-band range, each found to be separated by regular intervals which correspond with a 6.69Hz period. This can be explained in terms of the interaction of the stimulus (and stimulus-induced) rhythm with a slow endogenous theta rhythm. When the interaction is in phase between these rhythms and target presentation time is slightly ahead of the priming stimulus presentation, improved detection times are recorded indicating an anticipatory response. However, when these rhythms are out of phase and the target is presented during or slightly after priming stimulus presentation, improved responding also occurs, suggesting a retroactive response is facilitated. Research in the auditory domain supports these findings (Aksentijevic et al., 2011). The conclusions of this research suggest that synchronization of neural assemblies contributes to the temporal code necessary to facilitate representation updating in order to respond to a dynamic environment and anticipate the logical next event.

Neural reuse of action perception circuits for language, concepts and communication

Neurocognitive and neurolinguistics theories make explicit statements relating specialized cognitive and linguistic processes to specific brain loci. These linking hypotheses are in need of neurobiological justification and explanation. Recent mathematical models of human language mechanisms constrained by fundamental neuroscience principles and established knowledge about comparative neuroanatomy offer explanations for where, when and how language is processed in the human brain. In these models, network structure and connectivity along with action- and perception-induced correlation of neuronal activity co-determine neurocognitive mechanisms. Language learning leads to the formation of action perception circuits (APCs) with specific distributions across cortical areas. Cognitive and linguistic processes such as speech production, comprehension, verbal working memory and prediction are modelled by activity dynamics in these APCs, and combinatorial and communicative-interactive knowledge is organized in the dynamics within, and connections between APCs. The network models and, in particular, the concept of distributionally-specific circuits, can account for some previously not well understood facts about the cortical 'hubs' for semantic processing and the motor system's role in language understanding and speech sound recognition. A review of experimental data evaluates predictions of the APC model and alternative theories, also providing detailed discussion of some seemingly contradictory findings. Throughout, recent disputes about the role of mirror neurons and grounded cognition in language and communication are assessed critically.

Enhanced Working Memory Binding by Direct Electrical Stimulation of the Parietal Cortex

Recent works evince the critical role of visual short-term memory (STM) binding deficits as a clinical and preclinical marker of Alzheimer’s disease (AD). These studies suggest a potential role of posterior brain regions in both the neurocognitive deficits of Alzheimer’s patients and STM binding in general. Thereupon, we surmised that stimulation of the posterior parietal cortex (PPC) might be a successful approach to tackle working memory deficits in this condition, especially at early stages. To date, no causal evidence exists of the role of the parietal cortex in STM binding. A unique approach to assess this issue is afforded by single-subject direct intracranial electrical stimulation of specific brain regions during a relevant cognitive task. Electrical stimulation has been used both for clinical purposes and to causally probe brain mechanisms. Previous evidence of electrical currents spreading through white matter along well defined functional circuits indicates that visual working memory mechanisms are subserved by a specific widely distributed network. Here, we stimulated the parietal cortex of a subject with intracranial electrodes as he performed the visual STM task. We compared the ensuing results to those from a non-stimulated condition and to the performance of a matched control group. In brief, direct stimulation of the parietal cortex induced a selective improvement in STM. These results, together with previous studies, provide very preliminary but promising ground to examine behavioral changes upon parietal stimulation in AD. We discuss our results regarding: (a) the usefulness of the task to target prodromal stages of AD; (b) the role of a posterior network in STM binding and in AD; and (c) the potential opportunity to improve STM binding through brain stimulation.

Emergence of local synchronization in neuronal networks with adaptive couplings

Local synchronization, both prolonged and transient, of oscillatory neuronal behavior in cortical networks plays a fundamental role in many aspects of perception and cognition. Here we study networks of Hindmarsh-Rose neurons with a new type of adaptive coupling, and show that these networks naturally produce both permanent and transient synchronization of local clusters of neurons. These deterministic systems exhibit complex dynamics with 1/f^η power spectra, which appears to be a consequence of a novel form of self-organized criticality.