Transitional Probabilities Are Prioritized over Stimulus/Pattern Probabilities in Auditory Deviance Detection: Memory Basis for Predictive Sound Processing

Sequential maturation of stimulus-specific adaptation in the mouse lemniscal auditory system

Stimulus-specific adaptation (SSA), the reduction of neural activity to a common stimulus that does not generalize to other, rare stimuli, is an essential property of our brain. Although well characterized in adults, it is still unknown how it develops during adolescence and what neuronal circuits are involved. Using in vivo electrophysiology and optogenetics in the lemniscal pathway of the mouse auditory system, we observed SSA to be stable from postnatal day 20 (P20) in the inferior colliculus, to develop until P30 in the auditory thalamus and even later in the primary auditory cortex (A1). We found this maturation process to be experience-dependent in A1 but not in thalamus and to be related to alterations in deep but not input layers of A1. We also identified corticothalamic projections to be implicated in thalamic SSA development. Together, our results reveal different circuits underlying the sequential SSA maturation and provide a unique perspective to understand predictive coding and surprise across sensory systems.

A multimodal cortical network of sensory expectation violation revealed by fMRI

The brain is subjected to multi-modal sensory information in an environment governed by statistical dependencies. Mismatch responses (MMRs), classically recorded with EEG, have provided valuable insights into the brain's processing of regularities and the generation of corresponding sensory predictions. Only few studies allow for comparisons of MMRs across multiple modalities in a simultaneous sensory stream and their corresponding cross-modal context sensitivity remains unknown. Here, we used a tri-modal version of the roving stimulus paradigm in fMRI to elicit MMRs in the auditory, somatosensory and visual modality. Participants (N = 29) were simultaneously presented with sequences of low and high intensity stimuli in each of the three senses while actively observing the tri-modal input stream and occasionally reporting the intensity of the previous stimulus in a prompted modality. The sequences were based on a probabilistic model, defining transition probabilities such that, for each modality, stimuli were more likely to repeat (p = .825) than change (p = .175) and stimulus intensities were equiprobable (p = .5). Moreover, each transition was conditional on the configuration of the other two modalities comprising global (cross-modal) predictive properties of the sequences. We identified a shared mismatch network of modality general inferior frontal and temporo-parietal areas as well as sensory areas, where the connectivity (psychophysiological interaction) between these regions was modulated during mismatch processing. Further, we found deviant responses within the network to be modulated by local stimulus repetition, which suggests highly comparable processing of expectation violation across modalities. Moreover, hierarchically higher regions of the mismatch network in the temporo-parietal area around the intraparietal sulcus were identified to signal cross-modal expectation violation. With the consistency of MMRs across audition, somatosensation and vision, our study provides insights into a shared cortical network of uni- and multi-modal expectation violation in response to sequence regularities.

Auditory learning of recurrent tone sequences is present in the newborn's brain

The seemingly effortless ability of our auditory system to rapidly detect new events in a dynamic environment is crucial for survival. Whether the underlying brain processes are innate is unknown. To answer this question, electroencephalography was recorded while regularly patterned (REG) versus random (RAND) tone sequences were presented to sleeping neonates. Regular relative to random sequences elicited differential neural responses after only a single repetition of the pattern indicating the existence of an innate capacity of the auditory system to detect auditory sequential regularities. We show that the newborn auditory system accumulates evidence only somewhat longer than the minimum amount determined by the ideal Bayesian observer model (the prediction from a variable-order Markov chain model) before detecting a repeating pattern. Thus, newborns can quickly form representations for regular features of the sound input, preparing the way for learning the contingencies of the environment.

Markov chains as a proxy for the predictive memory representations underlying mismatch negativity

Events not conforming to a regularity inherent to a sequence of events elicit prediction error signals of the brain such as the Mismatch Negativity (MMN) and impair behavioral task performance. Events conforming to a regularity lead to attenuation of brain activity such as stimulus-specific adaptation (SSA) and behavioral benefits. Such findings are usually explained by theories stating that the information processing system predicts the forthcoming event of the sequence via detected sequential regularities. A mathematical model that is widely used to describe, to analyze and to generate event sequences are Markov chains: They contain a set of possible events and a set of probabilities for transitions between these events (transition matrix) that allow to predict the next event on the basis of the current event and the transition probabilities. The accuracy of such a prediction depends on the distribution of the transition probabilities. We argue that Markov chains also have useful applications when studying cognitive brain functions. The transition matrix can be regarded as a proxy for generative memory representations that the brain uses to predict the next event. We assume that detected regularities in a sequence of events correspond to (a subset of) the entries in the transition matrix. We apply this idea to the Mismatch Negativity (MMN) research and examine three types of MMN paradigms: classical oddball paradigms emphasizing sound probabilities, between-sound regularity paradigms manipulating transition probabilities between adjacent sounds, and action-sound coupling paradigms in which sounds are associated with actions and their intended effects. We show that the Markovian view on MMN yields theoretically relevant insights into the brain processes underlying MMN and stimulates experimental designs to study the brain's processing of event sequences.

Prerequisites of language acquisition in the newborn brain

Learning to decode and produce speech is one of the most demanding tasks faced by infants. Nevertheless, infants typically utter their first words within a year, and phrases soon follow. Here we review cognitive abilities of newborn infants that promote language acquisition, focusing primarily on studies tapping neural activity. The results of these studies indicate that infants possess core adult auditory abilities already at birth, including statistical learning and rule extraction from variable speech input. Thus, the neonatal brain is ready to categorize sounds, detect word boundaries, learn words, and separate speech streams: in short, to acquire language quickly and efficiently from everyday linguistic input.

Why mismatch negativity continues to hold potential in probing altered brain function in schizophrenia

The brain potential known as mismatch negativity (MMN) is one of the most studied indices of altered brain function in schizophrenia. This review looks at what has been learned about MMN in schizophrenia over the last three decades and why the level of interest and activity in this field of research remains strong. A diligent consideration of available evidence suggests that MMN can serve as a biomarker in schizophrenia, but perhaps not the kind of biomarker that early research supposed. This review concludes that MMN measurement is likely to be most useful as a monitoring and response biomarker enabling tracking of an underlying pathology and efficacy of interventions, respectively. The role of, and challenges presented by, pre-clinical models is discussed as well as the merits of different methodologies that can be brought to bear in pursuing a deeper understanding of pathophysiology that might explain smaller MMN in schizophrenia.

Relevance to the higher order structure may govern auditory statistical learning in neonates

Hearing is one of the earliest senses to develop and is quite mature by birth. Contemporary theories assume that regularities in sound are exploited by the brain to create internal models of the environment. Through statistical learning, internal models extrapolate from patterns to predictions about subsequent experience. In adults, altered brain responses to sound enable us to infer the existence and properties of these models. In this study, brain potentials were used to determine whether newborns exhibit context-dependent modulations of a brain response that can be used to infer the existence and properties of internal models. Results are indicative of significant context-dependence in the responsivity to sound in newborns. When common and rare sounds continue in stable probabilities over a very long period, neonates respond to all sounds equivalently (no differentiation). However, when the same common and rare sounds at the same probabilities alternate over time, the neonate responses show clear differentiations. The context-dependence is consistent with the possibility that the neonate brain produces more precise internal models that discriminate between contexts when there is an emergent structure to be discovered but appears to adopt broader models when discrimination delivers little or no additional information about the environment.

The posterior auditory field is the chief generator of prediction error signals in the auditory cortex

The auditory cortex (AC) encompasses distinct fields subserving partly different aspects of sound processing. One essential function of the AC is the detection of unpredicted sounds, as revealed by differential neural activity to predictable and unpredictable sounds. According to the predictive coding framework, this effect can be explained by repetition suppression and/or prediction error signaling. The present study investigates functional specialization of the rat AC fields in repetition suppression and prediction error by combining a tone frequency oddball paradigm (involving high-probable standard and low-probable deviant tones) with two different control sequences (many-standards and cascade). Tones in the control sequences were comparable to deviant events with respect to neural adaptation but were not violating a regularity. Therefore, a difference in the neural activity between deviant and control tones indicates a prediction error effect, whereas a difference between control and standard tones indicates a repetition suppression effect. Single-unit recordings revealed by far the largest prediction error effects for the posterior auditory field, while the primary auditory cortex, the anterior auditory field, the ventral auditory field, and the suprarhinal auditory field were dominated by repetition suppression effects. Statistically significant repetition suppression effects occurred in all AC fields, whereas prediction error effects were less robust in the primary auditory cortex and the anterior auditory field. Results indicate that the non-lemniscal, posterior auditory field is more engaged in context-dependent processing underlying deviance-detection than the other AC fields, which are more sensitive to stimulus-dependent effects underlying differential degrees of neural adaptation.

Musical rhythm effects on visual attention are non-rhythmical: evidence against metrical entrainment

The idea that external rhythms synchronize attention cross-modally has attracted much interest and scientific inquiry. Yet, whether associated attentional modulations are indeed rhythmical in that they spring from and map onto an underlying meter has not been clearly established. Here we tested this idea while addressing the shortcomings of previous work associated with confounding (i) metricality and regularity, (ii) rhythmic and temporal expectations or (iii) global and local temporal effects. We designed sound sequences that varied orthogonally (high/low) in metricality and regularity and presented them as task-irrelevant auditory background in four separate blocks. The participants' task was to detect rare visual targets occurring at a silent metrically aligned or misaligned temporal position. We found that target timing was irrelevant for reaction times and visual event-related potentials. High background regularity and to a lesser extent metricality facilitated target processing across metrically aligned and misaligned positions. Additionally, high regularity modulated auditory background frequencies in the EEG recorded over occipital cortex. We conclude that external rhythms, rather than synchronizing attention cross-modally, confer general, nontemporal benefits. Their predictability conserves processing resources that then benefit stimulus representations in other modalities.

Neuroplasticity in the phonological system: The PMN and the N400 as markers for the perception of non-native phonemic contrasts by late second language learners

Second language (L2) learners frequently encounter persistent difficulty in perceiving certain non-native sound contrasts, i.e., a phenomenon called "phonological deafness". However, if extensive L2 experience leads to neuroplastic changes in the phonological system, then the capacity to discriminate non-native phonemic contrasts should progressively improve. Such perceptual changes should be attested by modifications at the neurophysiological level. We designed an EEG experiment in which the listeners' perceptual capacities to discriminate second language phonemic contrasts influence the processing of lexical-semantic violations. Semantic congruency of critical words in a sentence context was driven by a phonemic contrast that was unique to the L2, English (e.g.,/ɪ/-/i:/, ship - sheep). Twenty-eight young adult native speakers of French with intermediate proficiency in English listened to sentences that contained either a semantically congruent or incongruent critical word (e.g., The anchor of theship/*sheepwas let down) while EEG was recorded. Three ERP effects were found to relate to increasing L2 proficiency: (1) a left frontal auditory N100 effect, (2) a smaller fronto-central phonological mismatch negativity (PMN) effect and (3) a semantic N400 effect. No effect of proficiency was found on oscillatory markers. The current findings suggest that neuronal plasticity in the human brain allows for the late acquisition of even hard-wired linguistic features such as the discrimination of phonemic contrasts in a second language. This is the first time that behavioral and neurophysiological evidence for the critical role of neural plasticity underlying L2 phonological processing and its interdependence with semantic processing has been provided. Our data strongly support the idea that pieces of information from different levels of linguistic processing (e.g., phonological, semantic) strongly interact and influence each other during online language processing.

Retrieving the structure of probabilistic sequences of auditory stimuli from EEG data

Using a new probabilistic approach we model the relationship between sequences of auditory stimuli generated by stochastic chains and the electroencephalographic (EEG) data acquired while 19 participants were exposed to those stimuli. The structure of the chains generating the stimuli are characterized by rooted and labeled trees whose leaves, henceforth called contexts, represent the sequences of past stimuli governing the choice of the next stimulus. A classical conjecture claims that the brain assigns probabilistic models to samples of stimuli. If this is true, then the context tree generating the sequence of stimuli should be encoded in the brain activity. Using an innovative statistical procedure we show that this context tree can effectively be extracted from the EEG data, thus giving support to the classical conjecture.

Interactional synchrony: signals, mechanisms and benefits

Many group-living animals, humans included, occasionally synchronize their behavior with that of conspecifics. Social psychology and neuroscience have attempted to explain this phenomenon. Here we sought to integrate results around three themes: the stimuli, the mechanisms and the benefits of interactional synchrony. As regards stimuli, we asked what characteristics, apart from temporal regularity, prompt synchronization and found that stimulus modality and complexity are important. The high temporal resolution of the auditory system and the relevance of socio-emotional information endow auditory, multimodal, emotional and somewhat variable and adaptive sequences with particular synchronizing power. Looking at the mechanisms revealed that traditional perspectives emphasizing beat-based representations of others' signals conflict with more recent work investigating the perception of temporal regularity. Timing processes supported by striato-cortical loops represent any kind of repetitive interval sequence fairly automatically. Additionally, socio-emotional processes supported by posterior superior temporal cortex help endow such sequences with value motivating the extent of synchronizing. Synchronizing benefits arise from an increased predictability of incoming signals and include many positive outcomes ranging from basic information processing at the individual level to the bonding of dyads and larger groups.

Prediction error signaling explains neuronal mismatch responses in the medial prefrontal cortex

The mismatch negativity (MMN) is a key biomarker of automatic deviance detection thought to emerge from 2 cortical sources. First, the auditory cortex (AC) encodes spectral regularities and reports frequency-specific deviances. Then, more abstract representations in the prefrontal cortex (PFC) allow to detect contextual changes of potential behavioral relevance. However, the precise location and time asynchronies between neuronal correlates underlying this frontotemporal network remain unclear and elusive. Our study presented auditory oddball paradigms along with "no-repetition" controls to record mismatch responses in neuronal spiking activity and local field potentials at the rat medial PFC. Whereas mismatch responses in the auditory system are mainly induced by stimulus-dependent effects, we found that auditory responsiveness in the PFC was driven by unpredictability, yielding context-dependent, comparatively delayed, more robust and longer-lasting mismatch responses mostly comprised of prediction error signaling activity. This characteristically different composition discarded that mismatch responses in the PFC could be simply inherited or amplified downstream from the auditory system. Conversely, it is more plausible for the PFC to exert top-down influences on the AC, since the PFC exhibited flexible and potent predictive processing, capable of suppressing redundant input more efficiently than the AC. Remarkably, the time course of the mismatch responses we observed in the spiking activity and local field potentials of the AC and the PFC combined coincided with the time course of the large-scale MMN-like signals reported in the rat brain, thereby linking the microscopic, mesoscopic, and macroscopic levels of automatic deviance detection.

Functional connectivity of the frontotemporal network in preattentive detection of abstract changes: Perturbs and observes with transcranial magnetic stimulation and event-related optical signal

Current theories of automatic or preattentive change detection suggest a regularity or prediction violation mechanism involving functional connectivity between the inferior frontal cortex (IFC) and the superior temporal cortex (STC). By disrupting the IFC function with transcranial magnetic stimulation (TMS) and recording the later STC mismatch response with event-related optical signal (EROS), previous study demonstrated a causal IFC-to-STC functional connection in detecting a pitch or physical change. However, physical change detection can be achieved by memory comparison of the physical features and may not necessarily involve regularity/rule extraction and prediction. The current study investigated the IFC-STC functional connectivity in detecting rule violation (i.e., an abstract change). Frequent standard tone pairs with a constant relative pitch difference, but varying pitches, were presented to establish a pitch interval rule. This abstract rule was violated by deviants with reduced relative pitch intervals. The EROS STC mismatch response to the deviants was abolished by the TMS applied at the IFC 80 ms after deviance onset, but preserved in the spatial (TMS on vertex), auditory (TMS sound), and temporal (200 ms after deviance onset) control conditions. These results demonstrate the IFC-STC connection in preattentive abstract change detection and support the regularity or prediction violation account.

Encoding of deterministic and stochastic auditory rules in the human brain: The mismatch negativity mechanism does not reflect basic probability

Regularities in a sequence of sounds can be automatically encoded in a predictive model by the auditory system. When a sound deviates from the one predicted by the model, a mismatch negativity (MMN) is elicited, which is taken to reflect a prediction error at a particular level of the model hierarchy. Although there are many studies on deterministic regularities, only a few have investigated the brain's ability to encode non-deterministic regularities. We studied a simple stochastic regularity: two tone pitches (standards, each occurring on 45% of trials); this regularity was occasionally violated by another tone pitch (deviant, occurring on 10% of trials). We found MMN when the deviant's pitch was outside those of the standards, but not when it was between them. Importantly, when we alternated the occurrence of the same two standards, making them deterministic, the deviant elicited MMN, even when its pitch was between those of the standards. Thus, although the MMN system is extremely powerful in establishing even quite complex deterministic regularities, it fails with a simple stochastic regularity. We argue that the MMN system does not know basic probability.

Making Sense of Mismatch Negativity

Evoked potentials provide valuable insight into brain processes that are integral to our ability to interact effectively and efficiently in the world. The mismatch negativity (MMN) component of the evoked potential has proven highly informative on the ways in which sensitivity to regularity contributes to perception and cognition. This review offers a compendium of research on MMN with a view to scaffolding an appreciation for its use as a tool to explore the way regularities contribute to predictions about the sensory environment over many timescales. In compiling this work, interest in MMN as an index of sensory encoding and memory are addressed, as well as attention. Perspectives on the possible underlying computational processes are reviewed as well as recent observations that invite consideration of how MMN relates to how we learn, what we learn, and why.

Dynamic effects of habituation and novelty detection on newborn event-related potentials

Newborns habituate to repeated auditory stimuli, and discriminate syllables, generating opportunities for early language learning. This study investigated trial-by-trial changes in newborn electrophysiological responses to auditory speech syllables as an index of habituation and novelty detection. Auditory event-related potentials (ERPs) were recorded from 16 term newborn infants, aged 1-3 days, in response to monosyllabic speech syllables presented during habituation and novelty detection tasks. Multilevel models demonstrated that newborns habituated to repeated auditory syllables, as ERP amplitude attenuated for a late-latency component over successive trials. Subsequently, during the novelty detection task, early- and late-latency component amplitudes decreased over successive trials for novel syllables only, indicating encoding of the novel speech syllable. We conclude that newborns dynamically encoded novel syllables over relatively short time periods, as indicated by a systematic change in response patterns with increased exposure. These results have important implications for understanding early precursors of learning and memory in newborns.

The Neuronal Basis of Predictive Coding Along the Auditory Pathway: From the Subcortical Roots to Cortical Deviance Detection

In this review, we attempt to integrate the empirical evidence regarding stimulus-specific adaptation (SSA) and mismatch negativity (MMN) under a predictive coding perspective (also known as Bayesian or hierarchical-inference model). We propose a renewed methodology for SSA study, which enables a further decomposition of deviance detection into repetition suppression and prediction error, thanks to the use of two controls previously introduced in MMN research: the many-standards and the cascade sequences. Focusing on data obtained with cellular recordings, we explain how deviance detection and prediction error are generated throughout hierarchical levels of processing, following two vectors of increasing computational complexity and abstraction along the auditory neuraxis: from subcortical toward cortical stations and from lemniscal toward nonlemniscal divisions. Then, we delve into the particular characteristics and contributions of subcortical and cortical structures to this generative mechanism of hierarchical inference, analyzing what is known about the role of neuromodulation and local microcircuitry in the emergence of mismatch signals. Finally, we describe how SSA and MMN are occurring at similar time frame and cortical locations, and both are affected by the manipulation of N-methyl- D-aspartate receptors. We conclude that there is enough empirical evidence to consider SSA and MMN, respectively, as the microscopic and macroscopic manifestations of the same physiological mechanism of deviance detection in the auditory cortex. Hence, the development of a common theoretical framework for SSA and MMN is all the more recommendable for future studies. In this regard, we suggest a shared nomenclature based on the predictive coding interpretation of deviance detection.

Brain signatures of a multiscale process of sequence learning in humans

Extracting the temporal structure of sequences of events is crucial for perception, decision-making, and language processing. Here, we investigate the mechanisms by which the brain acquires knowledge of sequences and the possibility that successive brain responses reflect the progressive extraction of sequence statistics at different timescales. We measured brain activity using magnetoencephalography in humans exposed to auditory sequences with various statistical regularities, and we modeled this activity as theoretical surprise levels using several learning models. Successive brain waves related to different types of statistical inferences. Early post-stimulus brain waves denoted a sensitivity to a simple statistic, the frequency of items estimated over a long timescale (habituation). Mid-latency and late brain waves conformed qualitatively and quantitatively to the computational properties of a more complex inference: the learning of recent transition probabilities. Our findings thus support the existence of multiple computational systems for sequence processing involving statistical inferences at multiple scales.

Neurons along the auditory pathway exhibit a hierarchical organization of prediction error

Perception is characterized by a reciprocal exchange of predictions and prediction error signals between neural regions. However, the relationship between such sensory mismatch responses and hierarchical predictive processing has not yet been demonstrated at the neuronal level in the auditory pathway. We recorded single-neuron activity from different auditory centers in anaesthetized rats and awake mice while animals were played a sequence of sounds, designed to separate the responses due to prediction error from those due to adaptation effects. Here we report that prediction error is organized hierarchically along the central auditory pathway. These prediction error signals are detectable in subcortical regions and increase as the signals move towards auditory cortex, which in turn demonstrates a large-scale mismatch potential. Finally, the predictive activity of single auditory neurons underlies automatic deviance detection at subcortical levels of processing. These results demonstrate that prediction error is a fundamental component of singly auditory neuron responses.

Great Expectations: Is there Evidence for Predictive Coding in Auditory Cortex?

Predictive coding is possibly one of the most influential, comprehensive, and controversial theories of neural function. While proponents praise its explanatory potential, critics object that key tenets of the theory are untested or even untestable. The present article critically examines existing evidence for predictive coding in the auditory modality. Specifically, we identify five key assumptions of the theory and evaluate each in the light of animal, human and modeling studies of auditory pattern processing. For the first two assumptions - that neural responses are shaped by expectations and that these expectations are hierarchically organized - animal and human studies provide compelling evidence. The anticipatory, predictive nature of these expectations also enjoys empirical support, especially from studies on unexpected stimulus omission. However, for the existence of separate error and prediction neurons, a key assumption of the theory, evidence is lacking. More work exists on the proposed oscillatory signatures of predictive coding, and on the relation between attention and precision. However, results on these latter two assumptions are mixed or contradictory. Looking to the future, more collaboration between human and animal studies, aided by model-based analyses will be needed to test specific assumptions and implementations of predictive coding - and, as such, help determine whether this popular grand theory can fulfill its expectations.

Computational Models of Auditory Scene Analysis: A Review

Auditory scene analysis (ASA) refers to the process (es) of parsing the complex acoustic input into auditory perceptual objects representing either physical sources or temporal sound patterns, such as melodies, which contributed to the sound waves reaching the ears. A number of new computational models accounting for some of the perceptual phenomena of ASA have been published recently. Here we provide a theoretically motivated review of these computational models, aiming to relate their guiding principles to the central issues of the theoretical framework of ASA. Specifically, we ask how they achieve the grouping and separation of sound elements and whether they implement some form of competition between alternative interpretations of the sound input. We consider the extent to which they include predictive processes, as important current theories suggest that perception is inherently predictive, and also how they have been evaluated. We conclude that current computational models of ASA are fragmentary in the sense that rather than providing general competing interpretations of ASA, they focus on assessing the utility of specific processes (or algorithms) for finding the causes of the complex acoustic signal. This leaves open the possibility for integrating complementary aspects of the models into a more comprehensive theory of ASA.