Recognizing sequences of sequences

Outlier responses reflect sensitivity to statistical structure in the human brain

We constantly look for patterns in the environment that allow us to learn its key regularities. These regularities are fundamental in enabling us to make predictions about what is likely to happen next. The physiological study of regularity extraction has focused primarily on repetitive sequence-based rules within the sensory environment, or on stimulus-outcome associations in the context of reward-based decision-making. Here we ask whether we implicitly encode non-sequential stochastic regularities, and detect violations therein. We addressed this question using a novel experimental design and both behavioural and magnetoencephalographic (MEG) metrics associated with responses to pure-tone sounds with frequencies sampled from a Gaussian distribution. We observed that sounds in the tail of the distribution evoked a larger response than those that fell at the centre. This response resembled the mismatch negativity (MMN) evoked by surprising or unlikely events in traditional oddball paradigms. Crucially, responses to physically identical outliers were greater when the distribution was narrower. These results show that humans implicitly keep track of the uncertainty induced by apparently random distributions of sensory events. Source reconstruction suggested that the statistical-context-sensitive responses arose in a temporo-parietal network, areas that have been associated with attention orientation to unexpected events. Our results demonstrate a very early neurophysiological marker of the brain's ability to implicitly encode complex statistical structure in the environment. We suggest that this sensitivity provides a computational basis for our ability to make perceptual inferences in noisy environments and to make decisions in an uncertain world.

Survival crucially depends on our ability to extract information from the environment. This ability relies on learning about regularities that enable us to make predictions about what is likely to happen next. Sensitivity to violations of these regularities is necessary for timely reactions and adaptive responses to unexpected, or odd, events. Prior work on speech acquisition and artificial grammar learning has provided important behavioural evidence that humans are able to learn statistical regularities, but it still falls considerably short of providing a biological understanding for how these processes might take place in the brain. The neurophysiological study of regularity extraction has so far been limited, to either sequence-based rules or to simple change-detection paradigms, and thus the neurobiological mechanisms that underpin statistical learning remain unknown. Here we provide both behavioural and neurophysiological evidence to show that humans keep track of the uncertainty in apparently random distributions of events. Our work demonstrates that an early neurophysiological signal underlies the fundamental human ability of learning and making inferences in an uncertain world.

Associating spontaneous with evoked activity in a neural mass model of visual cortex

Spontaneous activity of the brain at rest frequently has been considered a mere backdrop to the salient activity evoked by external stimuli or tasks. However, the resting state of the brain consumes most of its energy budget, which suggests a far more important role. An intriguing hint comes from experimental observations of spontaneous activity patterns, which closely resemble those evoked by visual stimulation with oriented gratings, except that cortex appeared to cycle between different orientation maps. Moreover, patterns similar to those evoked by the behaviorally most relevant horizontal and vertical orientations occurred more often than those corresponding to oblique angles. We hypothesize that this kind of spontaneous activity develops at least to some degree autonomously, providing a dynamical reservoir of cortical states, which are then associated with visual stimuli through learning. To test this hypothesis, we use a biologically inspired neural mass model to simulate a patch of cat visual cortex. Spontaneous transitions between orientation states were induced by modest modifications of the neural connectivity, establishing a stable heteroclinic channel. Significantly, the experimentally observed greater frequency of states representing the behaviorally important horizontal and vertical orientations emerged spontaneously from these simulations. We then applied bar-shaped inputs to the model cortex and used Hebbian learning rules to modify the corresponding synaptic strengths. After unsupervised learning, different bar inputs reliably and exclusively evoked their associated orientation state; whereas in the absence of input, the model cortex resumed its spontaneous cycling. We conclude that the experimentally observed similarities between spontaneous and evoked activity in visual cortex can be explained as the outcome of a learning process that associates external stimuli with a preexisting reservoir of autonomous neural activity states. Our findings hence demonstrate how cortical connectivity can link the maintenance of spontaneous activity in the brain mechanistically to its core cognitive functions.

Active inference and agency: optimal control without cost functions

This paper describes a variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty. We show that optimal control can be cast as active inference. In active inference, both action and posterior beliefs about hidden states minimise a free energy bound on the negative log-likelihood of observed states, under a generative model. In this setting, reward or cost functions are absorbed into prior beliefs about state transitions and terminal states. Effectively, this converts optimal control into a pure inference problem, enabling the application of standard Bayesian filtering techniques. We then consider optimal trajectories that rest on posterior beliefs about hidden states in the future. Crucially, this entails modelling control as a hidden state that endows the generative model with a representation of agency. This leads to a distinction between models with and without inference on hidden control states; namely, agency-free and agency-based models, respectively.

Temporal context in speech processing and attentional stream selection: a behavioral and neural perspective

The human capacity for processing speech is remarkable, especially given that information in speech unfolds over multiple time scales concurrently. Similarly notable is our ability to filter out of extraneous sounds and focus our attention on one conversation, epitomized by the 'Cocktail Party' effect. Yet, the neural mechanisms underlying on-line speech decoding and attentional stream selection are not well understood. We review findings from behavioral and neurophysiological investigations that underscore the importance of the temporal structure of speech for achieving these perceptual feats. We discuss the hypothesis that entrainment of ambient neuronal oscillations to speech's temporal structure, across multiple time-scales, serves to facilitate its decoding and underlies the selection of an attended speech stream over other competing input. In this regard, speech decoding and attentional stream selection are examples of 'Active Sensing', emphasizing an interaction between proactive and predictive top-down modulation of neuronal dynamics and bottom-up sensory input.

Dysfunction of the auditory thalamus in developmental dyslexia

Developmental dyslexia, a severe and persistent reading and spelling impairment, is characterized by difficulties in processing speech sounds (i.e., phonemes). Here, we test the hypothesis that these phonological difficulties are associated with a dysfunction of the auditory sensory thalamus, the medial geniculate body (MGB). By using functional MRI, we found that, in dyslexic adults, the MGB responded abnormally when the task required attending to phonemes compared with other speech features. No other structure in the auditory pathway showed distinct functional neural patterns between the two tasks for dyslexic and control participants. Furthermore, MGB activity correlated with dyslexia diagnostic scores, indicating that the task modulation of the MGB is critical for performance in dyslexics. These results suggest that deficits in dyslexia are associated with a failure of the neural mechanism that dynamically tunes MGB according to predictions from cortical areas to optimize speech processing. This view on task-related MGB dysfunction in dyslexics has the potential to reconcile influential theories of dyslexia within a predictive coding framework of brain function.

Recognizing recurrent neural networks (rRNN): Bayesian inference for recurrent neural networks

Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made computationally more powerful by its fusion with Bayesian inference techniques for nonlinear dynamical systems. In this scheme, we use an RNN as a generative model of dynamic input caused by the environment, e.g. of speech or kinematics. Given this generative RNN model, we derive Bayesian update equations that can decode its output. Critically, these updates define a 'recognizing RNN' (rRNN), in which neurons compute and exchange prediction and prediction error messages. The rRNN has several desirable features that a conventional RNN does not have, e.g. fast decoding of dynamic stimuli and robustness to initial conditions and noise. Furthermore, it implements a predictive coding scheme for dynamic inputs. We suggest that the Bayesian inversion of RNNs may be useful both as a model of brain function and as a machine learning tool. We illustrate the use of the rRNN by an application to the online decoding (i.e. recognition) of human kinematics.

The role of prediction in social neuroscience

Research has shown that the brain is constantly making predictions about future events. Theories of prediction in perception, action and learning suggest that the brain serves to reduce the discrepancies between expectation and actual experience, i.e., by reducing the prediction error. Forward models of action and perception propose the generation of a predictive internal representation of the expected sensory outcome, which is matched to the actual sensory feedback. Shared neural representations have been found when experiencing one's own and observing other's actions, rewards, errors, and emotions such as fear and pain. These general principles of the “predictive brain” are well established and have already begun to be applied to social aspects of cognition. The application and relevance of these predictive principles to social cognition are discussed in this article. Evidence is presented to argue that simple non-social cognitive processes can be extended to explain complex cognitive processes required for social interaction, with common neural activity seen for both social and non-social cognitions. A number of studies are included which demonstrate that bottom-up sensory input and top-down expectancies can be modulated by social information. The concept of competing social forward models and a partially distinct category of social prediction errors are introduced. The evolutionary implications of a “social predictive brain” are also mentioned, along with the implications on psychopathology. The review presents a number of testable hypotheses and novel comparisons that aim to stimulate further discussion and integration between currently disparate fields of research, with regard to computational models, behavioral and neurophysiological data. This promotes a relatively new platform for inquiry in social neuroscience with implications in social learning, theory of mind, empathy, the evolution of the social brain, and potential strategies for treating social cognitive deficits.

Waking and dreaming consciousness: neurobiological and functional considerations

This paper presents a theoretical review of rapid eye movement sleep with a special focus on pontine-geniculate-occipital waves and what they might tell us about the functional anatomy of sleep and consciousness. In particular, we review established ideas about the nature and purpose of sleep in terms of protoconsciousness and free energy minimization. By combining these theoretical perspectives, we discover answers to some fundamental questions about sleep: for example, why is homeothermy suspended during sleep? Why is sleep necessary? Why are we not surprised by our dreams? What is the role of synaptic regression in sleep? The imperatives for sleep that emerge also allow us to speculate about the functional role of PGO waves and make some empirical predictions that can, in principle, be tested using recent advances in the modeling of electrophysiological data.

Temporal predictive codes for spoken words in auditory cortex

Humans can recognize spoken words with unmatched speed and accuracy. Hearing the initial portion of a word such as "formu…" is sufficient for the brain to identify "formula" from the thousands of other words that partially match. Two alternative computational accounts propose that partially matching words (1) inhibit each other until a single word is selected ("formula" inhibits "formal" by lexical competition) or (2) are used to predict upcoming speech sounds more accurately (segment prediction error is minimal after sequences like "formu…"). To distinguish these theories we taught participants novel words (e.g., "formubo") that sound like existing words ("formula") on two successive days. Computational simulations show that knowing "formubo" increases lexical competition when hearing "formu…", but reduces segment prediction error. Conversely, when the sounds in "formula" and "formubo" diverge, the reverse is observed. The time course of magnetoencephalographic brain responses in the superior temporal gyrus (STG) is uniquely consistent with a segment prediction account. We propose a predictive coding model of spoken word recognition in which STG neurons represent the difference between predicted and heard speech sounds. This prediction error signal explains the efficiency of human word recognition and simulates neural responses in auditory regions.

A dynamical pattern recognition model of γ activity in auditory cortex

This paper describes a dynamical process which serves both as a model of temporal pattern recognition in the brain and as a forward model of neuroimaging data. This process is considered at two separate levels of analysis: the algorithmic and implementation levels. At an algorithmic level, recognition is based on the use of Occurrence Time features. Using a speech digit database we show that for noisy recognition environments, these features rival standard cepstral coefficient features. At an implementation level, the model is defined using a Weakly Coupled Oscillator (WCO) framework and uses a transient synchronization mechanism to signal a recognition event. In a second set of experiments, we use the strength of the synchronization event to predict the high gamma (75–150 Hz) activity produced by the brain in response to word versus non-word stimuli. Quantitative model fits allow us to make inferences about parameters governing pattern recognition dynamics in the brain.

Evidence for a hierarchy of predictions and prediction errors in human cortex

According to hierarchical predictive coding models, the cortex constantly generates predictions of incoming stimuli at multiple levels of processing. Responses to auditory mismatches and omissions are interpreted as reflecting the prediction error when these predictions are violated. An alternative interpretation, however, is that neurons passively adapt to repeated stimuli. We separated these alternative interpretations by designing a hierarchical auditory novelty paradigm and recording human EEG and magnetoencephalographic (MEG) responses to mismatching or omitted stimuli. In the crucial condition, participants listened to frequent series of four identical tones followed by a fifth different tone, which generates a mismatch response. Because this response itself is frequent and expected, the hierarchical predictive coding hypothesis suggests that it should be cancelled out by a higher-order prediction. Three consequences ensue. First, the mismatch response should be larger when it is unexpected than when it is expected. Second, a perfectly monotonic sequence of five identical tones should now elicit a higher-order novelty response. Third, omitting the fifth tone should reveal the brain's hierarchical predictions. The rationale here is that, when a deviant tone is expected, its omission represents a violation of two expectations: a local prediction of a tone plus a hierarchically higher expectation of its deviancy. Thus, such an omission should induce a greater prediction error than when a standard tone is expected. Simultaneous EEE- magnetoencephalographic recordings verify those predictions and thus strongly support the predictive coding hypothesis. Higher-order predictions appear to be generated in multiple areas of frontal and associative cortices.

Free energy and dendritic self-organization

In this paper, we pursue recent observations that, through selective dendritic filtering, single neurons respond to specific sequences of presynaptic inputs. We try to provide a principled and mechanistic account of this selectivity by applying a recent free-energy principle to a dendrite that is immersed in its neuropil or environment. We assume that neurons self-organize to minimize a variational free-energy bound on the self-information or surprise of presynaptic inputs that are sampled. We model this as a selective pruning of dendritic spines that are expressed on a dendritic branch. This pruning occurs when postsynaptic gain falls below a threshold. Crucially, postsynaptic gain is itself optimized with respect to free energy. Pruning suppresses free energy as the dendrite selects presynaptic signals that conform to its expectations, specified by a generative model implicit in its intracellular kinetics. Not only does this provide a principled account of how neurons organize and selectively sample the myriad of potential presynaptic inputs they are exposed to, but it also connects the optimization of elemental neuronal (dendritic) processing to generic (surprise or evidence-based) schemes in statistics and machine learning, such as Bayesian model selection and automatic relevance determination.

Predictive coding and pitch processing in the auditory cortex

In this work, we show that electrophysiological responses during pitch perception are best explained by distributed activity in a hierarchy of cortical sources and, crucially, that the effective connectivity between these sources is modulated with pitch strength. Local field potentials were recorded in two subjects from primary auditory cortex and adjacent auditory cortical areas along the axis of Heschl's gyrus (HG) while they listened to stimuli of varying pitch strength. Dynamic causal modeling was used to compare system architectures that might explain the recorded activity. The data show that representation of pitch requires an interaction between nonprimary and primary auditory cortex along HG that is consistent with the principle of predictive coding.

Time scale hierarchies in the functional organization of complex behaviors

Traditional approaches to cognitive modelling generally portray cognitive events in terms of ‘discrete’ states (point attractor dynamics) rather than in terms of processes, thereby neglecting the time structure of cognition. In contrast, more recent approaches explicitly address this temporal dimension, but typically provide no entry points into cognitive categorization of events and experiences. With the aim to incorporate both these aspects, we propose a framework for functional architectures. Our approach is grounded in the notion that arbitrary complex (human) behaviour is decomposable into functional modes (elementary units), which we conceptualize as low-dimensional dynamical objects (structured flows on manifolds). The ensemble of modes at an agent’s disposal constitutes his/her functional repertoire. The modes may be subjected to additional dynamics (termed operational signals), in particular, instantaneous inputs, and a mechanism that sequentially selects a mode so that it temporarily dominates the functional dynamics. The inputs and selection mechanisms act on faster and slower time scales then that inherent to the modes, respectively. The dynamics across the three time scales are coupled via feedback, rendering the entire architecture autonomous. We illustrate the functional architecture in the context of serial behaviour, namely cursive handwriting. Subsequently, we investigate the possibility of recovering the contributions of functional modes and operational signals from the output, which appears to be possible only when examining the output phase flow (i.e., not from trajectories in phase space or time).

In most established approaches to cognitive modelling, cognitive events are treated as ‘discrete’ states, thus passing by the continuous nature of cognitive processes. In contrast, some novel approaches explicitly acknowledge cognition’s temporal structure but provides no entry points into cognitive categorization of events and experiences. We attempt to incorporate both aspects in a new framework, which departs from the established idea that complex (human) behaviour is made up of elementary functional ‘building blocks’, referred to as modes. We model these as mathematical objects that are inherently dynamic (i.e., account for change over time). A mechanism sequentially selects the modes required and binds them together to compose complex behaviours. These modes may be subjected to brief inputs. The ensemble of these three ingredients, which influence one another and operate on different time scales, constitutes a functional architecture. We illustrate the architecture via cursive handwriting simulations, and investigate the possibility of recovering the contributions of the architecture from the written word. This appears possible only when focussing on the dynamic modes.

Facilitated auditory detection for speech sounds

If it is well known that knowledge facilitates higher cognitive functions, such as visual and auditory word recognition, little is known about the influence of knowledge on detection, particularly in the auditory modality. Our study tested the influence of phonological and lexical knowledge on auditory detection. Words, pseudo-words, and complex non-phonological sounds, energetically matched as closely as possible, were presented at a range of presentation levels from sub-threshold to clearly audible. The participants performed a detection task (Experiments 1 and 2) that was followed by a two alternative forced-choice recognition task in Experiment 2. The results of this second task in Experiment 2 suggest a correct recognition of words in the absence of detection with a subjective threshold approach. In the detection task of both experiments, phonological stimuli (words and pseudo-words) were better detected than non-phonological stimuli (complex sounds), presented close to the auditory threshold. This finding suggests an advantage of speech for signal detection. An additional advantage of words over pseudo-words was observed in Experiment 2, suggesting that lexical knowledge could also improve auditory detection when listeners had to recognize the stimulus in a subsequent task. Two simulations of detection performance performed on the sound signals confirmed that the advantage of speech over non-speech processing could not be attributed to energetic differences in the stimuli.

Action understanding and active inference

We have suggested that the mirror-neuron system might be usefully understood as implementing Bayes-optimal perception of actions emitted by oneself or others. To substantiate this claim, we present neuronal simulations that show the same representations can prescribe motor behavior and encode motor intentions during action-observation. These simulations are based on the free-energy formulation of active inference, which is formally related to predictive coding. In this scheme, (generalised) states of the world are represented as trajectories. When these states include motor trajectories they implicitly entail intentions (future motor states). Optimizing the representation of these intentions enables predictive coding in a prospective sense. Crucially, the same generative models used to make predictions can be deployed to predict the actions of self or others by simply changing the bias or precision (i.e. attention) afforded to proprioceptive signals. We illustrate these points using simulations of handwriting to illustrate neuronally plausible generation and recognition of itinerant (wandering) motor trajectories. We then use the same simulations to produce synthetic electrophysiological responses to violations of intentional expectations. Our results affirm that a Bayes-optimal approach provides a principled framework, which accommodates current thinking about the mirror-neuron system. Furthermore, it endorses the general formulation of action as active inference.

Attention, uncertainty, and free-energy

We suggested recently that attention can be understood as inferring the level of uncertainty or precision during hierarchical perception. In this paper, we try to substantiate this claim using neuronal simulations of directed spatial attention and biased competition. These simulations assume that neuronal activity encodes a probabilistic representation of the world that optimizes free-energy in a Bayesian fashion. Because free-energy bounds surprise or the (negative) log-evidence for internal models of the world, this optimization can be regarded as evidence accumulation or (generalized) predictive coding. Crucially, both predictions about the state of the world generating sensory data and the precision of those data have to be optimized. Here, we show that if the precision depends on the states, one can explain many aspects of attention. We illustrate this in the context of the Posner paradigm, using the simulations to generate both psychophysical and electrophysiological responses. These simulated responses are consistent with attentional bias or gating, competition for attentional resources, attentional capture and associated speed-accuracy trade-offs. Furthermore, if we present both attended and non-attended stimuli simultaneously, biased competition for neuronal representation emerges as a principled and straightforward property of Bayes-optimal perception.

Neural syntax: cell assemblies, synapsembles, and readers

A widely discussed hypothesis in neuroscience is that transiently active ensembles of neurons, known as "cell assemblies," underlie numerous operations of the brain, from encoding memories to reasoning. However, the mechanisms responsible for the formation and disbanding of cell assemblies and temporal evolution of cell assembly sequences are not well understood. I introduce and review three interconnected topics, which could facilitate progress in defining cell assemblies, identifying their neuronal organization, and revealing causal relationships between assembly organization and behavior. First, I hypothesize that cell assemblies are best understood in light of their output product, as detected by "reader-actuator" mechanisms. Second, I suggest that the hierarchical organization of cell assemblies may be regarded as a neural syntax. Third, constituents of the neural syntax are linked together by dynamically changing constellations of synaptic weights ("synapsembles"). The existing support for this tripartite framework is reviewed and strategies for experimental testing of its predictions are discussed.

Dynamical principles of emotion-cognition interaction: mathematical images of mental disorders

The key contribution of this work is to introduce a mathematical framework to understand self-organized dynamics in the brain that can explain certain aspects of itinerant behavior. Specifically, we introduce a model based upon the coupling of generalized Lotka-Volterra systems. This coupling is based upon competition for common resources. The system can be regarded as a normal or canonical form for any distributed system that shows self-organized dynamics that entail winnerless competition. Crucially, we will show that some of the fundamental instabilities that arise in these coupled systems are remarkably similar to endogenous activity seen in the brain (using EEG and fMRI). Furthermore, by changing a small subset of the system's parameters we can produce bifurcations and metastable sequential dynamics changing, which bear a remarkable similarity to pathological brain states seen in psychiatry. In what follows, we will consider the coupling of two macroscopic modes of brain activity, which, in a purely descriptive fashion, we will label as cognitive and emotional modes. Our aim is to examine the dynamical structures that emerge when coupling these two modes and relate them tentatively to brain activity in normal and non-normal states.

The relation of ongoing brain activity, evoked neural responses, and cognition

Ongoing brain activity has been observed since the earliest neurophysiological recordings and is found over a wide range of temporal and spatial scales. It is characterized by remarkably large spontaneous modulations. Here, we review evidence for the functional role of these ongoing activity fluctuations and argue that they constitute an essential property of the neural architecture underlying cognition. The role of spontaneous activity fluctuations is probably best understood when considering both their spatiotemporal structure and their functional impact on cognition. We first briefly argue against a "segregationist" view on ongoing activity, both in time and space, which would selectively associate certain frequency bands or levels of spatial organization with specific functional roles. Instead, we emphasize the functional importance of the full range, from differentiation to integration, of intrinsic activity within a hierarchical spatiotemporal structure. We then highlight the flexibility and context-sensitivity of intrinsic functional connectivity that suggest its involvement in functionally relevant information processing. This role in information processing is pursued by reviewing how ongoing brain activity interacts with afferent and efferent information exchange of the brain with its environment. We focus on the relationship between the variability of ongoing and evoked brain activity, and review recent reports that tie ongoing brain activity fluctuations to variability in human perception and behavior. Finally, these observations are discussed within the framework of the free-energy principle which - applied to human brain function - provides a theoretical account for a non-random, coordinated interaction of ongoing and evoked activity in perception and behavior.

On consciousness, resting state fMRI, and neurodynamics

BACKGROUND

During the last years, functional magnetic resonance imaging (fMRI) of the brain has been introduced as a new tool to measure consciousness, both in a clinical setting and in a basic neurocognitive research. Moreover, advanced mathematical methods and theories have arrived the field of fMRI (e.g. computational neuroimaging), and functional and structural brain connectivity can now be assessed non-invasively.

RESULTS

The present work deals with a pluralistic approach to "consciousness'', where we connect theory and tools from three quite different disciplines: (1) philosophy of mind (emergentism and global workspace theory), (2) functional neuroimaging acquisitions, and (3) theory of deterministic and statistical neurodynamics - in particular the Wilson-Cowan model and stochastic resonance.

CONCLUSIONS

Based on recent experimental and theoretical work, we believe that the study of large-scale neuronal processes (activity fluctuations, state transitions) that goes on in the living human brain while examined with functional MRI during "resting state", can deepen our understanding of graded consciousness in a clinical setting, and clarify the concept of "consiousness" in neurocognitive and neurophilosophy research.

The anatomy of language: a review of 100 fMRI studies published in 2009

In this review of 100 fMRI studies of speech comprehension and production, published in 2009, activation is reported for: prelexical speech perception in bilateral superior temporal gyri; meaningful speech in middle and inferior temporal cortex; semantic retrieval in the left angular gyrus and pars orbitalis; and sentence comprehension in bilateral superior temporal sulci. For incomprehensible sentences, activation increases in four inferior frontal regions, posterior planum temporale, and ventral supramarginal gyrus. These effects are associated with the use of prior knowledge of semantic associations, word sequences, and articulation that predict the content of the sentence. Speech production activates the same set of regions as speech comprehension but in addition, activation is reported for: word retrieval in left middle frontal cortex; articulatory planning in the left anterior insula; the initiation and execution of speech in left putamen, pre-SMA, SMA, and motor cortex; and for suppressing unintended responses in the anterior cingulate and bilateral head of caudate nuclei. Anatomical and functional connectivity studies are now required to identify the processing pathways that integrate these areas to support language.

How the human brain recognizes speech in the context of changing speakers

We understand speech from different speakers with ease, whereas artificial speech recognition systems struggle with this task. It is unclear how the human brain solves this problem. The conventional view is that speech message recognition and speaker identification are two separate functions and that message processing takes place predominantly in the left hemisphere, whereas processing of speaker-specific information is located in the right hemisphere. Here, we distinguish the contribution of specific cortical regions, to speech recognition and speaker information processing, by controlled manipulation of task and resynthesized speaker parameters. Two functional magnetic resonance imaging studies provide evidence for a dynamic speech-processing network that questions the conventional view. We found that speech recognition regions in left posterior superior temporal gyrus/superior temporal sulcus (STG/STS) also encode speaker-related vocal tract parameters, which are reflected in the amplitude peaks of the speech spectrum, along with the speech message. Right posterior STG/STS activated specifically more to a speaker-related vocal tract parameter change during a speech recognition task compared with a voice recognition task. Left and right posterior STG/STS were functionally connected. Additionally, we found that speaker-related glottal fold parameters (e.g., pitch), which are not reflected in the amplitude peaks of the speech spectrum, are processed in areas immediately adjacent to primary auditory cortex, i.e., in areas in the auditory hierarchy earlier than STG/STS. Our results point to a network account of speech recognition, in which information about the speech message and the speaker's vocal tract are combined to solve the difficult task of understanding speech from different speakers.