A modular architecture for transparent computation in recurrent neural networks

Computation is classically studied in terms of automata, formal languages and algorithms; yet, the relation between neural dynamics and symbolic representations and operations is still unclear in traditional eliminative connectionism. Therefore, we suggest a unique perspective on this central issue, to which we would like to refer as transparent connectionism, by proposing accounts of how symbolic computation can be implemented in neural substrates. In this study we first introduce a new model of dynamics on a symbolic space, the versatile shift, showing that it supports the real-time simulation of a range of automata. We then show that the Gödelization of versatile shifts defines nonlinear dynamical automata, dynamical systems evolving on a vectorial space. Finally, we present a mapping between nonlinear dynamical automata and recurrent artificial neural networks. The mapping defines an architecture characterized by its granular modularity, where data, symbolic operations and their control are not only distinguishable in activation space, but also spatially localizable in the network itself, while maintaining a distributed encoding of symbolic representations. The resulting networks simulate automata in real-time and are programmed directly, in the absence of network training. To discuss the unique characteristics of the architecture and their consequences, we present two examples: (i) the design of a Central Pattern Generator from a finite-state locomotive controller, and (ii) the creation of a network simulating a system of interactive automata that supports the parsing of garden-path sentences as investigated in psycholinguistics experiments.

Statistical Frequency-Dependent Analysis of Trial-to-Trial Variability in Single Time Series by Recurrence Plots

For decades, research in neuroscience has supported the hypothesis that brain dynamics exhibits recurrent metastable states connected by transients, which together encode fundamental neural information processing. To understand the system's dynamics it is important to detect such recurrence domains, but it is challenging to extract them from experimental neuroscience datasets due to the large trial-to-trial variability. The proposed methodology extracts recurrent metastable states in univariate time series by transforming datasets into their time-frequency representations and computing recurrence plots based on instantaneous spectral power values in various frequency bands. Additionally, a new statistical inference analysis compares different trial recurrence plots with corresponding surrogates to obtain statistically significant recurrent structures. This combination of methods is validated by applying it to two artificial datasets. In a final study of visually-evoked Local Field Potentials in partially anesthetized ferrets, the methodology is able to reveal recurrence structures of neural responses with trial-to-trial variability. Focusing on different frequency bands, the δ-band activity is much less recurrent than α-band activity. Moreover, α-activity is susceptible to pre-stimuli, while δ-activity is much less sensitive to pre-stimuli. This difference in recurrence structures in different frequency bands indicates diverse underlying information processing steps in the brain.

Metastable dynamics in heterogeneous neural fields

We present numerical simulations of metastable states in heterogeneous neural fields that are connected along heteroclinic orbits. Such trajectories are possible representations of transient neural activity as observed, for example, in the electroencephalogram. Based on previous theoretical findings on learning algorithms for neural fields, we directly construct synaptic weight kernels from Lotka-Volterra neural population dynamics without supervised training approaches. We deliver a MATLAB neural field toolbox validated by two examples of one- and two-dimensional neural fields. We demonstrate trial-to-trial variability and distributed representations in our simulations which might therefore be regarded as a proof-of-concept for more advanced neural field models of metastable dynamics in neurophysiological data.

Detecting event-related recurrences by symbolic analysis: applications to human language processing

Quasi-stationarity is ubiquitous in complex dynamical systems. In brain dynamics, there is ample evidence that event-related potentials (ERPs) reflect such quasi-stationary states. In order to detect them from time series, several segmentation techniques have been proposed. In this study, we elaborate a recent approach for detecting quasi-stationary states as recurrence domains by means of recurrence analysis and subsequent symbolization methods. We address two pertinent problems of contemporary recurrence analysis: optimizing the size of recurrence neighbourhoods and identifying symbols from different realizations for sequence alignment. As possible solutions for these problems, we suggest a maximum entropy criterion and a Hausdorff clustering algorithm. The resulting recurrence domains for single-subject ERPs are obtained as partition cells reflecting quasi-stationary brain states.

Order effects in dynamic semantics

In their target article, Wang and Busemeyer (2013) discuss question order effects in terms of incompatible projectors on a Hilbert space. In a similar vein, Blutner recently presented an orthoalgebraic query language essentially relying on dynamic update semantics. Here, I shall comment on some interesting analogies between the different variants of dynamic semantics and generalized quantum theory to illustrate other kinds of order effects in human cognition, such as belief revision, the resolution of anaphors, and default reasoning that result from the crucial non-commutativity of mental operations upon the belief state of a cognitive agent.

A biophysical observation model for field potentials of networks of leaky integrate-and-fire neurons

We present a biophysical approach for the coupling of neural network activity as resulting from proper dipole currents of cortical pyramidal neurons to the electric field in extracellular fluid. Starting from a reduced three-compartment model of a single pyramidal neuron, we derive an observation model for dendritic dipole currents in extracellular space and thereby for the dendritic field potential (DFP) that contributes to the local field potential (LFP) of a neural population. This work aligns and satisfies the widespread dipole assumption that is motivated by the "open-field" configuration of the DFP around cortical pyramidal cells. Our reduced three-compartment scheme allows to derive networks of leaky integrate-and-fire (LIF) models, which facilitates comparison with existing neural network and observation models. In particular, by means of numerical simulations we compare our approach with an ad hoc model by Mazzoni et al. (2008), and conclude that our biophysically motivated approach yields substantial improvement.

Dimensional reduction for the inverse problem of neural field theory

Inverse problems in computational neuroscience comprise the determination of synaptic weight matrices or kernels for neural networks or neural fields respectively. Here, we reduce multi-dimensional inverse problems to inverse problems in lower dimensions which can be solved in an easier way or even explicitly through kernel construction. In particular, we discuss a range of embedding techniques and analyze their properties. We study the Amari equation as a particular example of a neural field theory. We obtain a solution of the full 2D or 3D problem by embedding 0D or 1D kernels into the domain of the Amari equation using a suitable path parametrization and basis transformations. Pulses are interconnected at branching points via path gluing. As instructive examples we construct logical gates, such as the persistent XOR and binary addition in neural fields. In addition, we compare results of inversion by dimensional reduction with a recently proposed global inversion scheme for neural fields based on Tikhonov–Hebbian learning. The results show that stable construction of complex distributed processes is possible via neural field dynamics. This is an important first step to study the properties of such constructions and to analyze natural or artificial realizations of neural field architectures.

Unifying syntactic theory and sentence processing difficulty through a connectionist minimalist parser

Syntactic theory provides a rich array of representational assumptions about linguistic knowledge and processes. Such detailed and independently motivated constraints on grammatical knowledge ought to play a role in sentence comprehension. However most grammar-based explanations of processing difficulty in the literature have attempted to use grammatical representations and processes per se to explain processing difficulty. They did not take into account that the description of higher cognition in mind and brain encompasses two levels: on the one hand, at the macrolevel, symbolic computation is performed, and on the other hand, at the microlevel, computation is achieved through processes within a dynamical system. One critical question is therefore how linguistic theory and dynamical systems can be unified to provide an explanation for processing effects. Here, we present such a unification for a particular account to syntactic theory: namely a parser for Stabler's Minimalist Grammars, in the framework of Smolensky's Integrated Connectionist/Symbolic architectures. In simulations we demonstrate that the connectionist minimalist parser produces predictions which mirror global empirical findings from psycholinguistic research.

Towards dynamical system models of language-related brain potentials

Event-related brain potentials (ERP) are important neural correlates of cognitive processes. In the domain of language processing, the N400 and P600 reflect lexical-semantic integration and syntactic processing problems, respectively. We suggest an interpretation of these markers in terms of dynamical system theory and present two nonlinear dynamical models for syntactic computations where different processing strategies correspond to functionally different regions in the system's phase space.

Language processing with dynamic fields

We construct a mapping from complex recursive linguistic data structures to spherical wave functions using Smolensky's filler/role bindings and tensor product representations. Syntactic language processing is then described by the transient evolution of these spherical patterns whose amplitudes are governed by nonlinear order parameter equations. Implications of the model in terms of brain wave dynamics are indicated.

Diagnosis and repair of negative polarity constructions in the light of symbolic resonance analysis

In a post hoc analysis, we investigate differences in event-related potentials of two studies (Drenhaus et al., 2004, Drenhaus et al., to appear, Saddy et al., 2004a and Saddy et al., 2004b) by using the symbolic resonance analysis (Beim Graben & Kurths, 2003). The studies under discussion, examined the failure to license a negative polarity item (NPI) in German: Saddy et al. (2004a) reported an N400 component when the NPI was not accurately licensed by negation; Drenhaus et al., 2004 and Drenhaus et al., to appear considered additionally the influence of constituency of the licensor in NPI constructions. A biphasic N400-P600 response was found for the two induced violations (the lack of licensor and the inaccessibility of negation in a relative clause). The symbolic resonance analysis (SRA) revealed an effect in the P600 time window for the data in Saddy et al., which was not found by using the averaging technique. The SRA of the ERPs in Drenhaus et al., showed that the P600 components are distinguishable concerning the amplitude and latency. It was smaller and earlier in the condition where the licensor is inaccessible, compared to the condition without negation in the string. Our findings suggest that the failure in licensing NPIs is not exclusively related to semantic integration costs (N400). The elicited P600 components reflect differences in syntactic processing. Our results confirm and replicate the effects of the traditional voltage average analysis and show that the SRA is a useful tool to reveal and pull apart ERP differences which are not evident using the traditional voltage average analysis.

Finding needles in haystacks: Symbolic resonance analysis of event-related potentials unveils different processing demands

Previous ERP studies have found an N400-P600 pattern in sentences in which the number of arguments does not match the number of arguments that the verb can take. In the present study, we elaborate on this question by investigating whether the case of the mismatching object argument in German (accusative/direct object versus dative/indirect object) affects processing differently. In general, both types of mismatches elicited a biphasic N400-P600 response in the ERP. However, traditional voltage average analysis was unable to reveal differences between the two mismatching conditions, that is, between a mismatching accusative versus dative. Therefore, we employed a recently developed method on ERP data analysis, the symbolic resonance analysis (SRA), where EEG epochs are symbolically encoded in sequences of three symbols depending on a given parameter, the encoding threshold. We found a larger proportion of threshold crossing events with negative polarity in the N400 time window for a mismatching dative argument compared to a mismatching accusative argument. By contrast, the proportion of threshold crossing events with positive polarity was smaller for dative in the P600 time window. We argue that this difference is due to the phenomenon of "free dative" in German. This result also shows that the SRA provides a useful tool for revealing ERP differences that cannot be discovered using the traditional voltage average analysis.

Detecting subthreshold events in noisy data by symbolic dynamics

We show that a symmetric threshold crossing detector can be described by a symbolic dynamics of a static three-symbol encoding which is highly efficient to detect subthreshold events in noisy nonstationary data. After computing instantaneous word statistics and running cylinder entropies, we introduce a mean-field transformation of the three-symbol dynamics considered as a Potts-spin lattice onto a distribution of two symbols. This transformed word statistics enables one to derive an estimator of the signal-to-noise ratio (SNR). Subthreshold events are then proven by a prominent peak of the SNR estimator as a function of the noise intensity.