Probing the lifetimes of auditory novelty detection processes

Sensory Target Detection at Local and Global Timescales Reveals a Hierarchy of Supramodal Dynamics in the Human Cortex

To ensure survival in a dynamic environment, the human neocortex monitors input streams from different sensory organs for important sensory events. Which principles govern whether different senses share common or modality-specific brain networks for sensory target detection? We examined whether complex targets evoke sustained supramodal activity while simple targets rely on modality-specific networks with short-lived supramodal contributions. In a series of hierarchical multisensory target detection studies (n = 77, of either sex) using EEG, we applied a temporal cross-decoding approach to dissociate supramodal and modality-specific cortical dynamics elicited by rule-based global and feature-based local sensory deviations within and between the visual, somatosensory, and auditory modality. Our data show that each sense implements a cortical hierarchy orchestrating supramodal target detection responses, which operate at local and global timescales in successive processing stages. Across different sensory modalities, simple feature-based sensory deviations presented in temporal vicinity to a monotonous input stream triggered a mismatch negativity-like local signal which decayed quickly and early, whereas complex rule-based targets tracked across time evoked a P3b-like global neural response which generalized across a late time window. Converging results from temporal cross-modality decoding analyses across different datasets, we reveal that global neural responses are sustained in a supramodal higher-order network, whereas local neural responses canonically thought to rely on modality-specific regions evolve into short-lived supramodal activity. Together, our findings demonstrate that cortical organization largely follows a gradient in which short-lived modality-specific as well as supramodal processes dominate local responses, whereas higher-order processes encode temporally extended abstract supramodal information fed forward from modality-specific cortices.SIGNIFICANCE STATEMENT Each sense supports a cortical hierarchy of processes tracking deviant sensory events at multiple timescales. Conflicting evidence produced a lively debate around which of these processes are supramodal. Here, we manipulated the temporal complexity of auditory, tactile, and visual targets to determine whether cortical local and global ERP responses to sensory targets share cortical dynamics between the senses. Using temporal cross-decoding, we found that temporally complex targets elicit a supramodal sustained response. Conversely, local responses to temporally confined targets typically considered modality-specific rely on early short-lived supramodal activation. Our finding provides evidence for a supramodal gradient supporting sensory target detection in the cortex, with implications for multiple fields in which these responses are studied (e.g., predictive coding, consciousness, and attention).

Linking the neural basis of distributional statistical learning with transitional statistical learning: The paradox of attention

Statistical learning, the process of tracking distributional information and discovering embedded patterns, is traditionally regarded as a form of implicit learning. However, recent studies proposed that both implicit (attention-independent) and explicit (attention-dependent) learning systems are involved in statistical learning. To understand the role of attention in statistical learning, the current study investigates the cortical processing of distributional patterns in speech across local and global contexts. We then ask how these cortical responses relate to statistical learning behavior in a word segmentation task. We found Event-Related Potential (ERP) evidence of pre-attentive processing of both the local (mismatching negativity) and global distributional information (late discriminative negativity). However, as speech elements became less frequent and more surprising, some participants showed an involuntary attentional shift, reflected in a P3a response. Individuals who displayed attentive neural tracking of distributional information showed faster learning in a speech statistical learning task. These results suggest that an involuntary attentional shift might play a facilitatory, but not essential, role in statistical learning.

Involuntary oculomotor inhibition markers of saliency and deviance in response to auditory sequences

Our eyes move constantly but are often inhibited momentarily in response to external stimuli (oculomotor inhibition [OMI]), depending on the stimulus saliency, anticipation, and attention. Previous studies have shown prolonged OMI for auditory oddballs; however, they required counting the oddballs, possibly reflecting voluntary attention. Here, we investigated whether the “passive” OMI response to auditory deviants can provide a quantitative measure of deviance strength (pitch difference) and studied its dependence on the inter-trial interval (ITI). Participants fixated centrally and passively listened to repeated short sequences of pure tones that contained a deviant tone either regularly or with 20% probability (oddballs). In an “active” control experiment, participants counted the deviant or the standard. As in previous studies, the results showed prolonged microsaccade inhibition and increased pupil dilation following the rare deviant tone. Earlier inhibition onset was found in proportion to the pitch deviance (the saliency effect), and a later release was found for oddballs, but only for ITI <2.5 seconds. The active control experiment showed similar results when counting the deviant but longer OMI for the standard when counting it. Taken together, these results suggest that OMI provides involuntary markers of saliency and deviance, which can be obtained without the participant's response.

Auditory Event-Related "Global Effect" Predicts Recovery of Overt Consciousness

Objective: To explore whether the presence of an event-related potential (ERP) "global effect" (GE+)-that corresponds to a correlate of conscious processing in the local-global auditory task-predicts behaviorally overt consciousness recovery in a large cohort of patients suffering from disorders of consciousness (DOC). Methods: We conducted a prospective study on all DOC patients evaluated during the 2009-2018 period. Behavioral examination included Coma Recovery Scale-Revised (CRS-R) scores and bedside high-density EEG recordings. Consciousness recovery was evaluated at 6 months by a structured phone interview. The predictive value of a GE+ was calculated both on survivors and on all patients. Results: A total of 236 patients with a documented outcome and technically valid EEG recordings could be included. Among them, 66 patients had a GE+ status (28%). Presence of GE+ predicted behaviorally overt consciousness recovery in survivors with high specificity (Sp = 84%) and high positive predictive value (PPV = 80%) but with low sensitivity (Se = 35%) and low negative predictive value (NPV = 42%). Positive likelihood ratio (LR+) of GE+ was superior to LR+ of initial clinical status and of ERP effect indexing unconscious auditory processing [local effect (LE)]. Interpretation: Our results demonstrate that the presence of a bedside ERP GE+ is highly predictive of behaviorally overt consciousness recovery in DOC patients, regardless of the delay, of behavioral status, and of the etiology of brain dysfunction. However, the absence of this effect is not a reliable predictor of negative outcome. This study provides Class III evidence that the presence of an ERP "global effect" predicts consciousness recovery in DOC patients.

A Minimum Temporal Window for Direction Detection of Frequency-Modulated Sweeps: A Magnetoencephalography Study

The ability to rapidly encode the direction of frequency contour contained in frequency-modulated (FM) sweeps is essential for speech processing, music appreciation, and conspecific communications. Psychophysical evidence points to a common temporal window threshold for human listeners in processing rapid changes in frequency glides. No neural evidence has been provided for the existence of a cortical temporal window threshold underlying the encoding of rapid transitions in frequency glides. The present magnetoencephalography study used the cortical mismatch negativity activity (MMNm) to investigate the minimum temporal window required for detecting different magnitudes of directional changes in frequency-modulated sweeps. A deviant oddball paradigm was used in which directional upward or downward frequency sweep serves as the standard and the same type of sweep with the opposite direction serves as its deviant. Stimuli consisted of unidirectional linear frequency-sweep complexes that swept across speech-relevant frequency bands in durations of 10, 20, 40, 80, 160, and 320 ms (with corresponding rates of 50, 25, 12.5, 6.2, 3.1, 1.5 oct/s). The data revealed significant magnetic mismatch field responses across all sweep durations, with slower-rate sweeps eliciting larger MMNm responses. A greater temporally related enhancement in MMNm response was obtained for rising but not falling frequency sweep contours. A hemispheric asymmetry in the MMNm response pattern was observed corresponding to the directionality of frequency sweeps. Contrary to psychophysical findings, we report a temporal window as short as 10 ms sufficient to elicit a robust MMNm response to a directional change in speech-relevant frequency contours. The results suggest that auditory cortex requires extremely brief temporal window to implicitly differentiate a dynamic change in frequency of linguistically relevant pitch contours. That the brain is extremely sensitive to fine spectral changes contained in speech-relevant glides provides cortical evidence for the ecological importance of FM sweeps in speech processing.

Neural Signal to Violations of Abstract Rules Using Speech-Like Stimuli

As the evidence of predictive processes playing a role in a wide variety of cognitive domains increases, the brain as a predictive machine becomes a central idea in neuroscience. In auditory processing, a considerable amount of progress has been made using variations of the Oddball design, but most of the existing work seems restricted to predictions based on physical features or conditional rules linking successive stimuli. To characterize the predictive capacity of the brain to abstract rules, we present here two experiments that use speech-like stimuli to overcome limitations and avoid common confounds. Pseudowords were presented in isolation, intermixed with infrequent deviants that contained unexpected phoneme sequences. As hypothesized, the occurrence of unexpected sequences of phonemes reliably elicited an early prediction error signal. These prediction error signals do not seemed to be modulated by attentional manipulations due to different task instructions, suggesting that the predictions are deployed even when the task at hand does not volitionally involve error detection. In contrast, the amount of syllables congruent with a standard pseudoword presented before the point of deviance exerted a strong modulation. Prediction error's amplitude doubled when two congruent syllables were presented instead of one, despite keeping local transitional probabilities constant. This suggests that auditory predictions can be built integrating information beyond the immediate past. In sum, the results presented here further contribute to the understanding of the predictive capabilities of the human auditory system when facing complex stimuli and abstract rules.

Brain signatures of a multiscale process of sequence learning in humans

Extracting the temporal structure of sequences of events is crucial for perception, decision-making, and language processing. Here, we investigate the mechanisms by which the brain acquires knowledge of sequences and the possibility that successive brain responses reflect the progressive extraction of sequence statistics at different timescales. We measured brain activity using magnetoencephalography in humans exposed to auditory sequences with various statistical regularities, and we modeled this activity as theoretical surprise levels using several learning models. Successive brain waves related to different types of statistical inferences. Early post-stimulus brain waves denoted a sensitivity to a simple statistic, the frequency of items estimated over a long timescale (habituation). Mid-latency and late brain waves conformed qualitatively and quantitatively to the computational properties of a more complex inference: the learning of recent transition probabilities. Our findings thus support the existence of multiple computational systems for sequence processing involving statistical inferences at multiple scales.

Unconscious semantic processing of polysemous words is not automatic

Semantic processing of visually presented words can be identified both on behavioral and neurophysiological evidence. One of the major discoveries of the last decades is the demonstration that these signatures of semantic processing, initially observed for consciously perceived words, can also be detected for masked words inaccessible to conscious reports. In this context, the distinction between conscious and unconscious verbal semantic processing constitutes a challenging scientific issue. A prominent view considered that while conscious representations are subject to executive control, unconscious ones would operate automatically in a modular way, independent from control and top-down influences. Recent findings challenged this view by revealing that endogenous attention and task-setting can have a strong influence on unconscious processing. However, one of the major arguments supporting the automaticity of unconscious semantic processing still stands, stemming from a seminal observation reported by Marcel in 1980 about polysemous words. In the present study we reexamined this evidence. We present a combination of behavioral and event-related-potentials (ERPs) results that refute this view by showing that the current conscious semantic context has a major and similar influence on the semantic processing of both visible and masked polysemous words. In a classical lexical decision task, a polysemous word was preceded by a word which defined the current semantic context. Crucially, this context was associated with only one of the two meanings of the polysemous word, and was followed by a word/pseudo-word target. Behavioral and electrophysiological evidence of semantic priming of target words by masked polysemous words was strongly dependent on the conscious context. Moreover, we describe a new type of influence related to the response-code used to answer for target words in the lexical decision task: unconscious semantic priming constrained by the conscious context was present both in behavior and ERPs exclusively when right-handed subjects were instructed to respond to words with their right hand. The strong and respective influences of conscious context and response-code on semantic processing of masked polysemous words demonstrate that unconscious verbal semantic representations are not automatic.

Assessing consciousness with auditory event-related potential during coma recovery: a case study

To observe changes in mismatch negativity (MMN) and P300 in a patient transitioning from a vegetative state (VS) to a minimally conscious state (MCS). One patient with intracerebral hemorrhage and an 8-month disease course was evaluated as being in the VS using the Coma Recovery Scale-Revised. Two weeks after the patient was admitted to the hospital, another evaluation was performed, and the patient was determined to be in an MCS. Using the Oddball paradigm, pure tone and name stimuli were presented to the patient to study event-related potentials (ERPs). A 15-week clinical follow-up was carried out, and four ERP examinations were performed at 2 days and 2, 6, and 15 weeks after admission. One healthy individual was assessed as a control participant. MMN and P300 were elicited in all four ERP examinations. MMN and P300 may occur earlier than believed in patients in persistent VS and MCS; their predictive values for prognosis need to be further confirmed by follow-up studies on a large clinical sample.

Disruption of hierarchical predictive coding during sleep

When presented with an auditory sequence, the brain acts as a predictive-coding device that extracts regularities in the transition probabilities between sounds and detects unexpected deviations from these regularities. Does such prediction require conscious vigilance, or does it continue to unfold automatically in the sleeping brain? The mismatch negativity and P300 components of the auditory event-related potential, reflecting two steps of auditory novelty detection, have been inconsistently observed in the various sleep stages. To clarify whether these steps remain during sleep, we recorded simultaneous electroencephalographic and magnetoencephalographic signals during wakefulness and during sleep in normal subjects listening to a hierarchical auditory paradigm including short-term (local) and long-term (global) regularities. The global response, reflected in the P300, vanished during sleep, in line with the hypothesis that it is a correlate of high-level conscious error detection. The local mismatch response remained across all sleep stages (N1, N2, and REM sleep), but with an incomplete structure; compared with wakefulness, a specific peak reflecting prediction error vanished during sleep. Those results indicate that sleep leaves initial auditory processing and passive sensory response adaptation intact, but specifically disrupts both short-term and long-term auditory predictive coding.

How does the extraction of local and global auditory regularities vary with context?

How does the human brain extract regularities from its environment? There is evidence that short range or ‘local’ regularities (within seconds) are automatically detected by the brain while long range or ‘global’ regularities (over tens of seconds or more) require conscious awareness. In the present experiment, we asked whether participants' attention was needed to acquire such auditory regularities, to detect their violation or both. We designed a paradigm in which participants listened to predictable sounds. Subjects could be distracted by a visual task at two moments: when they were first exposed to a regularity or when they detected violations of this regularity. MEG recordings revealed that early brain responses (100–130 ms) to violations of short range regularities were unaffected by visual distraction and driven essentially by local transitional probabilities. Based on global workspace theory and prior results, we expected that visual distraction would eliminate the long range global effect, but unexpectedly, we found the contrary, i.e. late brain responses (300–600 ms) to violations of long range regularities on audio-visual trials but not on auditory only trials. Further analyses showed that, in fact, visual distraction was incomplete and that auditory and visual stimuli interfered in both directions. Our results show that conscious, attentive subjects can learn the long range dependencies present in auditory stimuli even while performing a visual task on synchronous visual stimuli. Furthermore, they acquire a complex regularity and end up making different predictions for the very same stimulus depending on the context (i.e. absence or presence of visual stimuli). These results suggest that while short-range regularity detection is driven by local transitional probabilities between stimuli, the human brain detects and stores long-range regularities in a highly flexible, context dependent manner.

Event-Related Potential, Time-frequency, and Functional Connectivity Facets of Local and Global Auditory Novelty Processing: An Intracranial Study in Humans

Auditory novelty detection has been associated with different cognitive processes. Bekinschtein et al. (2009) developed an experimental paradigm to dissociate these processes, using local and global novelty, which were associated, respectively, with automatic versus strategic perceptual processing. They have mostly been studied using event-related potentials (ERPs), but local spiking activity as indexed by gamma (60-120 Hz) power and interactions between brain regions as indexed by modulations in beta-band (13-25 Hz) power and functional connectivity have not been explored. We thus recorded 9 epileptic patients with intracranial electrodes to compare the precise dynamics of the responses to local and global novelty. Local novelty triggered an early response observed as an intracranial mismatch negativity (MMN) contemporary with a strong power increase in the gamma band and an increase in connectivity in the beta band. Importantly, all these responses were strictly confined to the temporal auditory cortex. In contrast, global novelty gave rise to a late ERP response distributed across brain areas, contemporary with a sustained power decrease in the beta band (13-25 Hz) and an increase in connectivity in the alpha band (8-13 Hz) within the frontal lobe. We discuss these multi-facet signatures in terms of conscious access to perceptual information.

A hierarchy of cortical responses to sequence violations in three-month-old infants

The adult human brain quickly adapts to regular temporal sequences, and emits a sequence of novelty responses when these regularities are violated. These novelty responses have been interpreted as error signals that reflect the difference between the incoming signal and predictions generated at multiple cortical levels. Do infants already possess such a hierarchy of violation-detection mechanisms? Using high-density recordings of event-related potentials during an auditory local-global violation paradigm, we show that three-month-old infants process novelty in temporal sequences at two distinct levels. Violations of local expectancies, such as perceiving a deviant vowel "a" after repeated presentation of another vowel i-i-i, elicited an early auditory mismatch response. Conversely, violations of global expectancies, such as hearing the rare sequence a-a-a-a instead of the frequent sequence a-a-a-i, modulated this early mismatch response and led to a late frontal negative slow wave, whose cortical sources included the left inferior frontal region. These results suggest that the infant brain already possesses two dissociable systems for temporal sequence learning.

Electrophysiological analysis of the role of novelty in the von Restorff effect

Items that are distinctive with respect to their context tend to be recalled better than nondistinctive items, a finding known as the von Restorff effect. The goal of this study was to elucidate the role of novelty in this effect. In two experiments, participants performed a dual task in which they had to study words presented visually while to-be ignored sounds were played over earphones. Sounds could be either standard or novel, and words could be presented in standard or novel font. Sounds were presented either simultaneously with the words (Experiment 1) or preceding them (Experiment 2). Electrophysiological correlates of novelty processing, the N2b and P3a ERP components, were recorded while the words were studied. It was seen that cued recall was better for words presented in novel fonts than for words in a standard font (the von Restorff effect). Words presented while novel sounds were played were remembered worse (Experiment 1) or equally well (Experiment 2) than those combined with standard sounds. Words presented in novel fonts elicited enhanced N2b, P3a, P3b, and N400 components; however, none of these components were specifically larger for subsequently recalled novel-font words. A larger N2b was found for recalled than for nonrecalled words, but this effect was not specific for words presented in novel font. We hypothesized that if novelty was beneficial for memory processing, the N2-P3 complex would be more enhanced for novel words that were later recalled than for those not recalled. The data showed otherwise. This suggests that novelty processing, as indexed by the N2-P3 novelty components, is not the main cause of the von Restorff effect.

A neuronal model of predictive coding accounting for the mismatch negativity

The mismatch negativity (MMN) is thought to index the activation of specialized neural networks for active prediction and deviance detection. However, a detailed neuronal model of the neurobiological mechanisms underlying the MMN is still lacking, and its computational foundations remain debated. We propose here a detailed neuronal model of auditory cortex, based on predictive coding, that accounts for the critical features of MMN. The model is entirely composed of spiking excitatory and inhibitory neurons interconnected in a layered cortical architecture with distinct input, predictive, and prediction error units. A spike-timing dependent learning rule, relying upon NMDA receptor synaptic transmission, allows the network to adjust its internal predictions and use a memory of the recent past inputs to anticipate on future stimuli based on transition statistics. We demonstrate that this simple architecture can account for the major empirical properties of the MMN. These include a frequency-dependent response to rare deviants, a response to unexpected repeats in alternating sequences (ABABAA…), a lack of consideration of the global sequence context, a response to sound omission, and a sensitivity of the MMN to NMDA receptor antagonists. Novel predictions are presented, and a new magnetoencephalography experiment in healthy human subjects is presented that validates our key hypothesis: the MMN results from active cortical prediction rather than passive synaptic habituation.

Neuroimaging and neuromodulation: complementary approaches for identifying the neuronal correlates of tinnitus

An inherent limitation of functional imaging studies is their correlational approach. More information about critical contributions of specific brain regions can be gained by focal transient perturbation of neural activity in specific regions with non-invasive focal brain stimulation methods. Functional imaging studies have revealed that tinnitus is related to alterations in neuronal activity of central auditory pathways. Modulation of neuronal activity in auditory cortical areas by repetitive transcranial magnetic stimulation (rTMS) can reduce tinnitus loudness and, if applied repeatedly, exerts therapeutic effects, confirming the relevance of auditory cortex activation for tinnitus generation and persistence. Measurements of oscillatory brain activity before and after rTMS demonstrate that the same stimulation protocol has different effects on brain activity in different patients, presumably related to interindividual differences in baseline activity in the clinically heterogeneous study cohort. In addition to alterations in auditory pathways, imaging techniques also indicate the involvement of non-auditory brain areas, such as the fronto-parietal "awareness" network and the non-tinnitus-specific distress network consisting of the anterior cingulate cortex, anterior insula, and amygdale. Involvement of the hippocampus and the parahippocampal region putatively reflects the relevance of memory mechanisms in the persistence of the phantom percept and the associated distress. Preliminary studies targeting the dorsolateral prefrontal cortex, the dorsal anterior cingulate cortex, and the parietal cortex with rTMS and with transcranial direct current stimulation confirm the relevance of the mentioned non-auditory networks. Available data indicate the important value added by brain stimulation as a complementary approach to neuroimaging for identifying the neuronal correlates of the various clinical aspects of tinnitus.

Evidence for a hierarchy of predictions and prediction errors in human cortex

According to hierarchical predictive coding models, the cortex constantly generates predictions of incoming stimuli at multiple levels of processing. Responses to auditory mismatches and omissions are interpreted as reflecting the prediction error when these predictions are violated. An alternative interpretation, however, is that neurons passively adapt to repeated stimuli. We separated these alternative interpretations by designing a hierarchical auditory novelty paradigm and recording human EEG and magnetoencephalographic (MEG) responses to mismatching or omitted stimuli. In the crucial condition, participants listened to frequent series of four identical tones followed by a fifth different tone, which generates a mismatch response. Because this response itself is frequent and expected, the hierarchical predictive coding hypothesis suggests that it should be cancelled out by a higher-order prediction. Three consequences ensue. First, the mismatch response should be larger when it is unexpected than when it is expected. Second, a perfectly monotonic sequence of five identical tones should now elicit a higher-order novelty response. Third, omitting the fifth tone should reveal the brain's hierarchical predictions. The rationale here is that, when a deviant tone is expected, its omission represents a violation of two expectations: a local prediction of a tone plus a hierarchically higher expectation of its deviancy. Thus, such an omission should induce a greater prediction error than when a standard tone is expected. Simultaneous EEE- magnetoencephalographic recordings verify those predictions and thus strongly support the predictive coding hypothesis. Higher-order predictions appear to be generated in multiple areas of frontal and associative cortices.

Processing of complex auditory patterns in musicians and nonmusicians

In the present study we investigated the capacity of the memory store underlying the mismatch negativity (MMN) response in musicians and nonmusicians for complex tone patterns. While previous studies have focused either on the kind of information that can be encoded or on the decay of the memory trace over time, we studied capacity in terms of the length of tone sequences, i.e., the number of individual tones that can be fully encoded and maintained. By means of magnetoencephalography (MEG) we recorded MMN responses to deviant tones that could occur at any position of standard tone patterns composed of four, six or eight tones during passive, distracted listening. Whereas there was a reliable MMN response to deviant tones in the four-tone pattern in both musicians and nonmusicians, only some individuals showed MMN responses to the longer patterns. This finding of a reliable capacity of the short-term auditory store underlying the MMN response is in line with estimates of a three to five item capacity of the short-term memory trace from behavioural studies, although pitch and contour complexity covaried with sequence length, which might have led to an understatement of the reported capacity. Whereas there was a tendency for an enhancement of the pattern MMN in musicians compared to nonmusicians, a strong advantage for musicians could be shown in an accompanying behavioural task of detecting the deviants while attending to the stimuli for all pattern lengths, indicating that long-term musical training differentially affects the memory capacity of auditory short-term memory for complex tone patterns with and without attention. Also, a left-hemispheric lateralization of MMN responses in the six-tone pattern suggests that additional networks that help structuring the patterns in the temporal domain might be recruited for demanding auditory processing in the pitch domain.