Constructing the hierarchy of predictive auditory sequences in the marmoset brain

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain

Predictive coding theory suggests the brain anticipates sensory information using prior knowledge. While this theory has been extensively researched within individual sensory modalities, evidence for predictive processing across sensory modalities is limited. Here, we examine how crossmodal knowledge is represented and learned in the brain, by identifying the hierarchical networks underlying crossmodal predictions when information of one sensory modality leads to a prediction in another modality. We record electroencephalogram (EEG) during a crossmodal audiovisual local-global oddball paradigm, in which the predictability of transitions between tones and images are manipulated at both the stimulus and sequence levels. To dissect the complex predictive signals in our EEG data, we employed a model-fitting approach to untangle neural interactions across modalities and hierarchies. The model-fitting result demonstrates that audiovisual integration occurs at both the levels of individual stimulus interactions and multi-stimulus sequences. Furthermore, we identify the spatio-spectro-temporal signatures of prediction-error signals across hierarchies and modalities, and reveal that auditory and visual prediction errors are rapidly redirected to the central-parietal electrodes during learning through alpha-band interactions. Our study suggests a crossmodal predictive coding mechanism where unimodal predictions are processed by distributed brain networks to form crossmodal knowledge.

Erroneous predictive coding across brain hierarchies in a non-human primate model of autism spectrum disorder

In autism spectrum disorder (ASD), atypical sensory experiences are often associated with irregularities in predictive coding, which proposes that the brain creates hierarchical sensory models via a bidirectional process of predictions and prediction errors. However, it remains unclear how these irregularities manifest across different functional hierarchies in the brain. To address this, we study a marmoset model of ASD induced by valproic acid (VPA) treatment. We record high-density electrocorticography (ECoG) during an auditory task with two layers of temporal control, and applied a quantitative model to quantify the integrity of predictive coding across two distinct hierarchies. Our results demonstrate a persistent pattern of sensory hypersensitivity and unstable predictions across two brain hierarchies in VPA-treated animals, and reveal the associated spatio-spectro-temporal neural signatures. Despite the regular occurrence of imprecise predictions in VPA-treated animals, we observe diverse configurations of underestimation or overestimation of sensory regularities within the hierarchies. Our results demonstrate the coexistence of the two primary Bayesian accounts of ASD: overly-precise sensory observations and weak prior beliefs, and offer a potential multi-layered biomarker for ASD, which could enhance our understanding of its diverse symptoms.

Multiple talker processing in autistic adult listeners

Accommodating talker variability is a complex and multi-layered cognitive process. It involves shifting attention to the vocal characteristics of the talker as well as the linguistic content of their speech. Due to an interdependence between voice and phonological processing, multi-talker environments typically incur additional processing costs compared to single-talker environments. A failure or inability to efficiently distribute attention over multiple acoustic cues in the speech signal may have detrimental language learning consequences. Yet, no studies have examined effects of multi-talker processing in populations with atypical perceptual, social and language processing for communication, including autistic people. Employing a classic word-monitoring task, we investigated effects of talker variability in Australian English autistic (n = 24) and non-autistic (n = 28) adults. Listeners responded to target words (e.g., apple, duck, corn) in randomised sequences of words. Half of the sequences were spoken by a single talker and the other half by multiple talkers. Results revealed that autistic participants' sensitivity scores to accurately-spotted target words did not differ to those of non-autistic participants, regardless of whether they were spoken by a single or multiple talkers. As expected, the non-autistic group showed the well-established processing cost associated with talker variability (e.g., slower response times). Remarkably, autistic listeners' response times did not differ across single- or multi-talker conditions, indicating they did not show perceptual processing costs when accommodating talker variability. The present findings have implications for theories of autistic perception and speech and language processing.

Distributed representations of prediction error signals across the cortical hierarchy are synergistic

A relevant question concerning inter-areal communication in the cortex is whether these interactions are synergistic. Synergy refers to the complementary effect of multiple brain signals conveying more information than the sum of each isolated signal. Redundancy, on the other hand, refers to the common information shared between brain signals. Here, we dissociated cortical interactions encoding complementary information (synergy) from those sharing common information (redundancy) during prediction error (PE) processing. We analyzed auditory and frontal electrocorticography (ECoG) signals in five common awake marmosets performing two distinct auditory oddball tasks and investigated to what extent event-related potentials (ERP) and broadband (BB) dynamics encoded synergistic and redundant information about PE processing. The information conveyed by ERPs and BB signals was synergistic even at lower stages of the hierarchy in the auditory cortex and between auditory and frontal regions. Using a brain-constrained neural network, we simulated the synergy and redundancy observed in the experimental results and demonstrated that the emergence of synergy between auditory and frontal regions requires the presence of strong, long-distance, feedback, and feedforward connections. These results indicate that distributed representations of PE signals across the cortical hierarchy can be highly synergistic.

Neural correlates of novelty detection in the primary auditory cortex of behaving monkeys

The neural mechanisms underlying novelty detection are not well understood, especially in relation to behavior. Here, we present single-unit responses from the primary auditory cortex (A1) from two monkeys trained to detect deviant tones amid repetitive ones. Results show that monkeys can detect deviant sounds, and there is a strong correlation between late neuronal responses (250-350 ms after deviant onset) and the monkeys' perceptual decisions. The magnitude and timing of both neuronal and behavioral responses are increased by larger frequency differences between the deviant and standard tones and by increasing the number of standard tones preceding the deviant. This suggests that A1 neurons encode novelty detection in behaving monkeys, influenced by stimulus relevance and expectations. This study provides evidence supporting aspects of predictive coding in the sensory cortex.

Auditory prediction errors in sound frequency and duration generated different cortical activation patterns in the human brain: an ECoG study

Sound frequency and duration are essential auditory components. The brain perceives deviations from the preceding sound context as prediction errors, allowing efficient reactions to the environment. Additionally, prediction error response to duration change is reduced in the initial stages of psychotic disorders. To compare the spatiotemporal profiles of responses to prediction errors, we conducted a human electrocorticography study with special attention to high gamma power in 13 participants who completed both frequency and duration oddball tasks. Remarkable activation in the bilateral superior temporal gyri in both the frequency and duration oddball tasks were observed, suggesting their association with prediction errors. However, the response to deviant stimuli in duration oddball task exhibited a second peak, which resulted in a bimodal response. Furthermore, deviant stimuli in frequency oddball task elicited a significant response in the inferior frontal gyrus that was not observed in duration oddball task. These spatiotemporal differences within the Parasylvian cortical network could account for our efficient reactions to changes in sound properties. The findings of this study may contribute to unveiling auditory processing and elucidating the pathophysiology of psychiatric disorders.

Relationship between schematic and dynamic expectations of melodic patterns in music perception

Prediction is fundamental in music listening. Two types of expectations have been proposed: schematic expectations, which arise from knowledge of tonal regularities (e.g., harmony and key) acquired through long-term plasticity and learning, and dynamic expectations, which arise from short-term regularity representations (e.g., rhythmic patterns and melodic contours) extracted from ongoing musical contexts. Although both expectations are indispensable in music listening, how they interact with each other in music prediction remains unclear. The present study examined the relationship between schematic and dynamic expectations in music processing using event-related potentials (ERPs). At the ending note of the melodies, the schematic expectation was violated by presenting a note with music-syntactic irregular (i.e., outof- key note), while the dynamic expectation was violated by presenting a contour deviant based on online statistical learning of melodic patterns. Schematic and dynamic expectations were manipulated to predict the same note. ERPs were recorded for the music-syntactic irregularity and the contour deviant, which occurred independently or simultaneously. The results showed that the music-syntactic irregularity elicited an early right anterior negativity (ERAN), reflecting the prediction error in the schematic expectation, while the contour deviant elicited a mismatch negativity (MMN), reflecting the prediction error in the dynamic expectation. Both components occurred within a similar latency range. Moreover, the ERP amplitude was multiplicatively increased when the irregularity and deviance occurred simultaneously. These findings suggest that schematic and dynamic expectations function concurrently in an interactive manner when both expectations predict the same note.

Auditory prediction and prediction error responses evoked through a novel cascade roving paradigm: a human ECoG study

Auditory sensory processing is assumed to occur in a hierarchical structure including the primary auditory cortex (A1), superior temporal gyrus, and frontal areas. These areas are postulated to generate predictions for incoming stimuli, creating an internal model of the surrounding environment. Previous studies on mismatch negativity have indicated the involvement of the superior temporal gyrus in this processing, whereas reports have been mixed regarding the contribution of the frontal cortex. We designed a novel auditory paradigm, the "cascade roving" paradigm, which incorporated complex structures (cascade sequences) into a roving paradigm. We analyzed electrocorticography data from six patients with refractory epilepsy who passively listened to this novel auditory paradigm and detected responses to deviants mainly in the superior temporal gyrus and inferior frontal gyrus. Notably, the inferior frontal gyrus exhibited broader distribution and sustained duration of deviant-elicited responses, seemingly differing in spatio-temporal characteristics from the prediction error responses observed in the superior temporal gyrus, compared with conventional oddball paradigms performed on the same participants. Moreover, we observed that the deviant responses were enhanced through stimulus repetition in the high-gamma range mainly in the superior temporal gyrus. These features of the novel paradigm may aid in our understanding of auditory predictive coding.

Multiple Concurrent Predictions Inform Prediction Error in the Human Auditory Pathway

The key assumption of the predictive coding framework is that internal representations are used to generate predictions on how the sensory input will look like in the immediate future. These predictions are tested against the actual input by the so-called prediction error units, which encode the residuals of the predictions. What happens to prediction errors, however, if predictions drawn by different stages of the sensory hierarchy contradict each other? To answer this question, we conducted two fMRI experiments while female and male human participants listened to sequences of sounds: pure tones in the first experiment and frequency-modulated sweeps in the second experiment. In both experiments, we used repetition to induce predictions based on stimulus statistics (stats-informed predictions) and abstract rules disclosed in the task instructions to induce an orthogonal set of (task-informed) predictions. We tested three alternative scenarios: neural responses in the auditory sensory pathway encode prediction error with respect to (1) the stats-informed predictions, (2) the task-informed predictions, or (3) a combination of both. Results showed that neural populations in all recorded regions (bilateral inferior colliculus, medial geniculate body, and primary and secondary auditory cortices) encode prediction error with respect to a combination of the two orthogonal sets of predictions. The findings suggest that predictive coding exploits the non-linear architecture of the auditory pathway for the transmission of predictions. Such non-linear transmission of predictions might be crucial for the predictive coding of complex auditory signals like speech.Significance Statement Sensory systems exploit our subjective expectations to make sense of an overwhelming influx of sensory signals. It is still unclear how expectations at each stage of the processing pipeline are used to predict the representations at the other stages. The current view is that this transmission is hierarchical and linear. Here we measured fMRI responses in auditory cortex, sensory thalamus, and midbrain while we induced two sets of mutually inconsistent expectations on the sensory input, each putatively encoded at a different stage. We show that responses at all stages are concurrently shaped by both sets of expectations. The results challenge the hypothesis that expectations are transmitted linearly and provide for a normative explanation of the non-linear physiology of the corticofugal sensory system.

Brainstem fMRI signaling of surprise across different types of deviant stimuli

Detection of deviant stimuli is crucial to orient and adapt our behavior. Previous work shows that deviant stimuli elicit phasic activation of the locus coeruleus (LC), which releases noradrenaline and controls central arousal. However, it is unclear whether the detection of behaviorally relevant deviant stimuli selectively triggers LC responses or other neuromodulatory systems (dopamine, serotonin, and acetylcholine). We combine human functional MRI (fMRI) recordings optimized for brainstem imaging with pupillometry to perform a mapping of deviant-related responses in subcortical structures. Participants have to detect deviant items in a "local-global" paradigm that distinguishes between deviance based on the stimulus probability and the sequence structure. fMRI responses to deviant stimuli are distributed in many cortical areas. Both types of deviance elicit responses in the pupil, LC, and other neuromodulatory systems. Our results reveal that the detection of task-relevant deviant items recruits the same multiple subcortical systems across computationally different types of deviance.

Short-term neuronal and synaptic plasticity act in synergy for deviance detection in spiking networks

Sensory areas of cortex respond more strongly to infrequent stimuli when these violate previously established regularities, a phenomenon known as deviance detection (DD). Previous modeling work has mainly attempted to explain DD on the basis of synaptic plasticity. However, a large fraction of cortical neurons also exhibit firing rate adaptation, an underexplored potential mechanism. Here, we investigate DD in a spiking neuronal network model with two types of short-term plasticity, fast synaptic short-term depression (STD) and slower threshold adaptation (TA). We probe the model with an oddball stimulation paradigm and assess DD by evaluating the network responses. We find that TA is sufficient to elicit DD. It achieves this by habituating neurons near the stimulation site that respond earliest to the frequently presented standard stimulus (local fatigue), which diminishes the response and promotes the recovery (global fatigue) of the wider network. Further, we find a synergy effect between STD and TA, where they interact with each other to achieve greater DD than the sum of their individual effects. We show that this synergy is caused by the local fatigue added by STD, which inhibits the global response to the frequently presented stimulus, allowing greater recovery of TA-mediated global fatigue and making the network more responsive to the deviant stimulus. Finally, we show that the magnitude of DD strongly depends on the timescale of stimulation. We conclude that highly predictable information can be encoded in strong local fatigue, which allows greater global recovery and subsequent heightened sensitivity for DD.

Spatiotemporal dynamics across visual cortical laminae support a predictive coding framework for interpreting mismatch responses

Context modulates neocortical processing of sensory data. Unexpected visual stimuli elicit large responses in primary visual cortex (V1)-a phenomenon known as deviance detection (DD) at the neural level, or "mismatch negativity" (MMN) when measured with EEG. It remains unclear how visual DD/MMN signals emerge across cortical layers, in temporal relation to the onset of deviant stimuli, and with respect to brain oscillations. Here we employed a visual "oddball" sequence-a classic paradigm for studying aberrant DD/MMN in neuropsychiatric populations-and recorded local field potentials in V1 of awake mice with 16-channel multielectrode arrays. Multiunit activity and current source density profiles showed that although basic adaptation to redundant stimuli was present early (50 ms) in layer 4 responses, DD emerged later (150-230 ms) in supragranular layers (L2/3). This DD signal coincided with increased delta/theta (2-7 Hz) and high-gamma (70-80 Hz) oscillations in L2/3 and decreased beta oscillations (26-36 Hz) in L1. These results clarify the neocortical dynamics elicited during an oddball paradigm at a microcircuit level. They are consistent with a predictive coding framework, which posits that predictive suppression is present in cortical feed-back circuits, which synapse in L1, whereas "prediction errors" engage cortical feed-forward processing streams, which emanate from L2/3.

Neuronal responses to omitted tones in the auditory brain: A neuronal correlate for predictive coding

Prediction provides key advantages for survival, and cognitive studies have demonstrated that the brain computes multilevel predictions. Evidence for predictions remains elusive at the neuronal level because of the complexity of separating neural activity into predictions and stimulus responses. We overcome this challenge by recording from single neurons from cortical and subcortical auditory regions in anesthetized and awake preparations, during unexpected stimulus omissions interspersed in a regular sequence of tones. We find a subset of neurons that responds reliably to omitted tones. In awake animals, omission responses are similar to anesthetized animals, but larger and more frequent, indicating that the arousal and attentional state levels affect the degree to which predictions are neuronally represented. Omission-sensitive neurons also responded to frequency deviants, with their omission responses getting emphasized in the awake state. Because omission responses occur in the absence of sensory input, they provide solid and empirical evidence for the implementation of a predictive process.

A vocalization-processing network in marmosets

Vocalizations play an important role in the daily life of primates and likely form the basis of human language. Functional imaging studies have demonstrated that listening to voices activates a fronto-temporal voice perception network in human participants. Here, we acquired whole-brain ultrahigh-field (9.4 T) fMRI in awake marmosets (Callithrix jacchus) and demonstrate that these small, highly vocal New World primates possess a similar fronto-temporal network, including subcortical regions, that is activated by the presentation of conspecific vocalizations. The findings suggest that the human voice perception network has evolved from an ancestral vocalization-processing network that predates the separation of New and Old World primates.

Repetition plasticity in primary auditory cortex occurs across long timescales for spectrotemporally randomized pure-tones

Repetition plasticity is a ubiquitous property of sensory systems in which repetitive sensation causes either a decrease ("repetition suppression", i.e. "adaptation") or increase ("repetition enhancement", i.e. "facilitation") in the amplitude of neural responses. Timescales of repetition plasticity for sensory neurons typically span milliseconds to tens of seconds, with longer durations for cortical vs subcortical regions. Here, we used 2-photon (2P) imaging to study repetition plasticity in mouse primary auditory cortex (A1) layer 2/3 (L2/3) during the presentation of spectrotemporally randomized pure-tone frequencies. Our study revealed subpopulations of neurons with repetition plasticity for equiprobable frequencies spaced minutes apart over a 20-minute period. We found both repetition suppression and enhancement in individual neurons and on average across populations. Each neuron tended to show repetition plasticity for 1-2 pure-tone frequencies near the neuron's best frequency. Moreover, we found correlated changes in neural response amplitude and latency across stimulus repetitions. Together, our results highlight cortical specialization for pattern recognition over long timescales in complex acoustic sequences.

Hierarchical Deviant Processing in Auditory Cortex of Awake Mice

Detecting patterns, and noticing unexpected pattern changes, in the environment is a vital aspect of sensory processing. Adaptation and prediction error responses are two components of neural processing related to these tasks, and previous studies in the auditory system in rodents show that these two components are partially dissociable in terms of the topography and latency of neural responses to sensory deviants. However, many previous studies have focused on repetitions of single stimuli, such as pure tones, which have limited ecological validity. In this study, we tested whether the auditory cortical activity shows adaptation to repetition of more complex sound patterns (bisyllabic pairs). Specifically, we compared neural responses to violations of sequences based on single stimulus probability only, against responses to more complex violations based on stimulus order. We employed an auditory oddball paradigm and monitored the auditory cortex (ACtx) activity of awake mice (N=8) using wide-field calcium imaging. We found that cortical responses were sensitive both to single stimulus probabilities and to more global stimulus patterns, as mismatch signals were elicited following both substitution deviants and transposition deviants. Notably, A2 area elicited larger mismatch signaling to those deviants than primary ACtx (A1), which suggests a hierarchical gradient of prediction error signaling in the auditory cortex. Such a hierarchical gradient was observed for late but not early peaks of calcium transients to deviants, suggesting that the late part of the deviant response may reflect prediction error signaling in response to more complex sensory pattern violations.

A quantitative model reveals a frequency ordering of prediction and prediction-error signals in the human brain

The human brain is proposed to harbor a hierarchical predictive coding neuronal network underlying perception, cognition, and action. In support of this theory, feedforward signals for prediction error have been reported. However, the identification of feedback prediction signals has been elusive due to their causal entanglement with prediction-error signals. Here, we use a quantitative model to decompose these signals in electroencephalography during an auditory task, and identify their spatio-spectral-temporal signatures across two functional hierarchies. Two prediction signals are identified in the period prior to the sensory input: a low-level signal representing the tone-to-tone transition in the high beta frequency band, and a high-level signal for the multi-tone sequence structure in the low beta band. Subsequently, prediction-error signals dependent on the prior predictions are found in the gamma band. Our findings reveal a frequency ordering of prediction signals and their hierarchical interactions with prediction-error signals supporting predictive coding theory.

A computational framework can extract spatio-spectro-temporal neural signatures corresponding to hierarchical prediction and prediction errors in a local-global auditory task.