Interrelation of attention and prediction in visual processing: Effects of task-relevance and stimulus probability

Belief Updating during Social Interactions: Neural Dynamics and Causal Role of Dorsomedial Prefrontal Cortex

In competitive interactions, humans have to flexibly update their beliefs about another person's intentions in order to adjust their own choice strategy, such as when believing that the other may exploit their cooperativeness. Here we investigate both the neural dynamics and the causal neural substrate of belief updating processes in humans. We used an adapted prisoner's dilemma game in which participants explicitly predicted the coplayer's actions, which allowed us to quantify the prediction error between expected and actual behavior. First, in an EEG experiment, we found a stronger medial frontal negativity (MFN) for negative than positive prediction errors, suggesting that this medial frontal ERP component may encode unexpected defection of the coplayer. The MFN also predicted subsequent belief updating after negative prediction errors. In a second experiment, we used transcranial magnetic stimulation (TMS) to investigate whether the dorsomedial prefrontal cortex (dmPFC) causally implements belief updating after unexpected outcomes. Our results show that dmPFC TMS impaired belief updating and strategic behavioral adjustments after negative prediction errors. Taken together, our findings reveal the time course of the use of prediction errors in social decisions and suggest that the dmPFC plays a crucial role in updating mental representations of others' intentions.

Attention to audiovisual speech shapes neural processing through feedback-feedforward loops between different nodes of the speech network

Selective attention-related top-down modulation plays a significant role in separating relevant speech from irrelevant background speech when vocal attributes separating concurrent speakers are small and continuously evolving. Electrophysiological studies have shown that such top-down modulation enhances neural tracking of attended speech. Yet, the specific cortical regions involved remain unclear due to the limited spatial resolution of most electrophysiological techniques. To overcome such limitations, we collected both electroencephalography (EEG) (high temporal resolution) and functional magnetic resonance imaging (fMRI) (high spatial resolution), while human participants selectively attended to speakers in audiovisual scenes containing overlapping cocktail party speech. To utilise the advantages of the respective techniques, we analysed neural tracking of speech using the EEG data and performed representational dissimilarity-based EEG-fMRI fusion. We observed that attention enhanced neural tracking and modulated EEG correlates throughout the latencies studied. Further, attention-related enhancement of neural tracking fluctuated in predictable temporal profiles. We discuss how such temporal dynamics could arise from a combination of interactions between attention and prediction as well as plastic properties of the auditory cortex. EEG-fMRI fusion revealed attention-related iterative feedforward-feedback loops between hierarchically organised nodes of the ventral auditory object related processing stream. Our findings support models where attention facilitates dynamic neural changes in the auditory cortex, ultimately aiding discrimination of relevant sounds from irrelevant ones while conserving neural resources.

Mechanisms of sustained perceptual entrainment after stimulus offset

Temporal alignment of neural activity to rhythmic stimulation has been suggested to result from a resonating internal neural oscillator mechanism, but can also be explained by interval-based temporal prediction. Here, we investigate behavioural and brain responses in the post-stimulation period to compare an oscillatory versus an interval-based account. Hickok et al.'s (2015) behavioural paradigm yielded results that relate to a neural oscillatory entrainment mechanism. We adapted the paradigm to an event-related potential (ERP) suitable design: a periodic sequence was followed, in half of the trials, by near-threshold targets embedded in noise. The targets were played in various phases in relation to the preceding sequences' period. Participants had to detect whether targets were played or not, and their EEG was recorded. Both behavioural results and the P300 component of the ERP were not only partially consistent with an oscillatory mechanism but also partially consistent with an interval-based attentional gain mechanism. Instead, data obtained in the post-entrainment period can best be explained with a combination of both mechanisms.

Event-Related Potentials Index Prediction Error Signalling During Perceptual Processing of Emotional Facial Expressions

Humans use socially relevant stimuli to guide perceptual processing of the surrounding environment, with emotional stimuli receiving preferential attention due to their social importance. Predictive coding theory asserts this cognitive process occurs efficiently by combining predictions about what is to be perceived with incoming sensory information, generating prediction errors that are then used to update future predictions. Recent evidence has identified differing neural activity that demonstrates how spatial and feature-based attention may interact with prediction, yet how emotion-guided attention may influence this relationship remains unknown. In the present study, participants viewed a display of two faces in which attention, prediction, and emotion were manipulated, and responded to a face expressing a specific emotion (anger or happiness). The N170 was found to be enhanced by unpredictable as opposed to predictable stimuli, indicating that it indexes general prediction error signalling processes. The N300 amplitudes were also enhanced by unpredictable stimuli, but they were also affected by the attentional status of angry but not happy faces, suggesting that there are differences in prediction error processes indexed by the N170 and N300. Overall, the findings suggest that the N170 and N300 both index violations of expectation for spatial manipulations of stimuli in accordance with prediction error responding processes.

Global attentional selection of visual features is not associated with selective modulation of posterior alpha-band activity

Attending to a single feature, such as color or motion, leads to global modulation of neural processing associated with the representation of the attended features. Alpha-band modulations are hypothesized to be a marker (and even a mechanism) of the modulation of neural processing. By adopting a previously used attentional shifting paradigm, we examined whether alpha-band dynamics are linked to sustained Feature-Based-Attentional (FBA) selection. For this purpose, we presented task-irrelevant flickering random dot kinematograms (RDKs) in the periphery that either did or did not share the to-be-attended color of centrally presented task-relevant RDKs and should thus be subject to global FBA selection. Steady-state visual evoked potentials (SSVEPs) and alpha-band activity associated with these task-irrelevant RDKs were analyzed to quantify FBA modulation. Overall, the SSVEP results replicated previous findings: relative to a pre-cue baseline, SSVEP amplitudes for peripheral RDKs were significantly enhanced when these RDKs shared the to-be-attended color of the central RDKs and were not modulated when they shared the centrally to-be-ignored color. Nevertheless, there were no differences in alpha-band amplitude modulations between signals recorded contralateral to the RDKs sharing the centrally attended color and RDKs sharing the centrally ignored color. Hence, alpha-band modulations seem not to index the sustained global selection of attended over unattended feature values within the same feature dimension.

The effect of perceptual expectation on processing gain, attention and the perceptual decision bias in children and adolescents with Autism Spectrum Disorder (ASD)

Perceptual expectations influence perception, attention and the perceptual decision bias during visuospatial orienting, which is impaired in individuals with Autism Spectrum Disorder (ASD). In this study, we investigated whether during visuospatial orienting, perceptual expectations in ASD differentially influence perception, attention and the perceptual decision bias relative to neurotypical controls (NT). Twenty-three children and adolescents with ASD and 23 NT completed a visuospatial orienting task, which compared the effect of a valid relative to an invalid perceptual expectation on target detection (cue validity effect). Group differences were calculated regarding the cue validity effect on neural correlates of processing gain (N1a amplitude) and attention (N1pc amplitude), the perceptual decision bias and mean reaction time (RT). In ASD relative to NT, findings showed a reduced processing gain for validly relative to invalidly cued targets and increased attentional response following invalidly relative to validly cued targets. Increased attention correlated with faster performance across groups. Increased processing correlated with a higher perceptual decision bias and faster mean RT in NT, but not in ASD. Results suggest that during visuospatial orienting, perceptual expectations in ASD may drive changes in sensory processing and stimulus-driven attention, which may differentially guide behavioural responses.

Impact of depressed state on attention and language processing during news broadcasts: EEG analysis and machine learning approach

While information enriches daily life, it can also sometimes have a negative impact, depending on an individual's mental state. We recorded electroencephalogram (EEG) signals from depressed and non-depressed individuals classified based on the Beck Depression Inventory-II score while they listened to news to clarify differences in their attention to affective information and the impact of attentional bias on language processing. Results showed that depressed individuals are characterized by delayed attention to positive news and require a more increased load on language processing. The feasibility of detecting a depressed state using these EEG characteristics was evaluated by classifying individuals as depressed and non-depressed individuals. The area under the curve in the models trained by the EEG features used was 0.73. This result shows that individuals' mental states may be assessed based on EEG measured during daily activities like listening to news.

Violated Expectations for Spatial and Feature Attributes of Visual Trajectories Modulate Event-Related Potential Amplitudes across the Visual Processing Hierarchy

During visual perception the brain must combine its predictions about what is to be perceived with incoming relevant information. The present study investigated how this process interacts with attention by using event-related potentials that index these cognitive mechanisms. Specifically, this study focused on examining how the amplitudes of the N170, N2pc, and N300 would be modulated by violations of expectations for spatial and featural attributes of visual stimuli. Participants viewed a series of shape stimuli in which a salient shape moved across a set of circular locations so that the trajectory of the shape implied the final position and shape of the stimulus. The final salient stimuli occurred in one of four possible outcomes: predictable position and shape, predictable position but unpredictable shape, unpredictable position but predictable shape, and unpredictable position and shape. The N170 was enhanced by unpredictable positions and shapes, whereas the N300 was enlarged only by unpredictable positions. The N2pc was not modulated by violations of expectation for shapes or positions. Additionally, it was observed post-hoc that the P1pc amplitude was increased by unpredictable shapes. These findings revealed that incorrect prediction increases neural activity. Furthermore, they suggest that prediction and attention interact differently in different stages of visual perception, depending on the type of attention being engaged: The N170 indexes initial prediction error signalling irrespective of the type of information (spatial or featural) in which error occurs, followed by the N300 as a marker of prediction updating involving reorientation of spatial attention.

Cognitive Neural Mechanism of Backward Inhibition and Deinhibition: A Review

Task switching is one of the typical paradigms to study cognitive control. When switching back to a recently inhibited task (e.g., “A” in an ABA sequence), the performance is often worse compared to a task without N-2 task repetitions (e.g., CBA). This difference is called the backward inhibitory effect (BI effect), which reflects the process of overcoming residual inhibition from a recently performed task (i.e., deinhibition). The neural mechanism of backward inhibition and deinhibition has received a lot of attention in the past decade. Multiple brain regions, including the frontal lobe, parietal, basal ganglia, and cerebellum, are activated during deinhibition. The event-related potentials (ERP) studies have shown that deinhibition process is reflected in the P1/N1 and P3 components, which might be related to early attention control, context updating, and response selection, respectively. Future research can use a variety of new paradigms to separate the neural mechanisms of BI and deinhibition.

P3b Does Not Reflect Perceived Contrasts

It has been shown that P3b is not a signature of perceptual awareness per se but is instead more closely associated with postperceptual processing (Cohen et al., 2020). Here, we seek to investigate whether human participants’ attentional states are different in the report and the no-report conditions. This difference in attentional states, if exists, may lead to degraded consciousness of the stimuli in the no-report condition, and it therefore remains unknown whether the disappearance of P3b is because of a lack of reportability or degraded consciousness. Results of our experiment 1 showed that participants did experience degraded contents of consciousness in the no-report condition. However, results of experiment 2 showed that the degraded contents of consciousness did not influence the amplitude of P3b. These findings strengthen the claim that P3b is not a signature of perceptual awareness but is associated with postperceptual processing.

Attention and prediction modulations in expected and unexpected visuospatial trajectories

Humans are constantly exposed to a rich tapestry of visual information in a potentially changing environment. To cope with the computational burden this engenders, our perceptual system must use prior context to simultaneously prioritise stimuli of importance and suppress irrelevant surroundings. This study investigated the influence of prediction and attention in visual perception by investigating event-related potentials (ERPs) often associated with these processes, N170 and N2pc for prediction and attention, respectively. A contextual trajectory paradigm was used which violated visual predictions and neglected to predetermine areas of spatial interest, to account for the potentially unpredictable nature of a real-life visual scene. Participants (N = 36) viewed a visual display of cued and non-cued shapes rotating in a five-step predictable trajectory, with the fifth and final position of either the cued or non-cued shape occurring in a predictable or unpredictable spatial location. To investigate the predictive coding theory of attention we used factors of attention and prediction, whereby attention was manipulated as either cued or non-cued conditions, and prediction manipulated in either predictable or unpredictable conditions. Results showed both enhanced N170 and N2pc amplitudes to unpredictable compared to predictable stimuli. Stimulus cueing status also increased N170 amplitude, but this did not interact with stimulus predictability. The N2pc amplitude was not affected by stimulus cueing status. In accordance with previous research these results suggest the N170 is in part a visual prediction error response with respect to higher-level visual processes, and furthermore the N2pc may index attention reorientation. The results demonstrate prior context influences the sensitivity of the N170 and N2pc electrophysiological responses. These findings add further support to the role of N170 as a prediction error signal and suggest that the N2pc may reflect attentional reorientation in response to unpredicted stimulus locations.

Preparatory delta phase response is correlated with naturalistic speech comprehension performance

While human speech comprehension is thought to be an active process that involves top-down predictions, it remains unclear how predictive information is used to prepare for the processing of upcoming speech information. We aimed to identify the neural signatures of the preparatory processing of upcoming speech. Participants selectively attended to one of two competing naturalistic, narrative speech streams, and a temporal response function (TRF) method was applied to derive event-related-like neural responses from electroencephalographic data. The phase responses to the attended speech at the delta band (1-4 Hz) were correlated with the comprehension performance of individual participants, with a latency of - 200-0 ms relative to the onset of speech amplitude envelope fluctuations over the fronto-central and left-lateralized parietal electrodes. The phase responses to the attended speech at the alpha band also correlated with comprehension performance but with a latency of 650-980 ms post-onset over the fronto-central electrodes. Distinct neural signatures were found for the attentional modulation, taking the form of TRF-based amplitude responses at a latency of 240-320 ms post-onset over the left-lateralized fronto-central and occipital electrodes. Our findings reveal how the brain gets prepared to process an upcoming speech in a continuous, naturalistic speech context.

Disentangling predictive processing in the brain: a meta-analytic study in favour of a predictive network

According to the predictive coding (PC) theory, the brain is constantly engaged in predicting its upcoming states and refining these predictions through error signals. Despite extensive research investigating the neural bases of this theory, to date no previous study has systematically attempted to define the neural mechanisms of predictive coding across studies and sensory channels, focussing on functional connectivity. In this study, we employ a coordinate-based meta-analytical approach to address this issue. We first use the Activation Likelihood Estimation (ALE) algorithm to detect spatial convergence across studies, related to prediction error and encoding. Overall, our ALE results suggest the ultimate role of the left inferior frontal gyrus and left insula in both processes. Moreover, we employ a meta-analytic connectivity method (Seed-Voxel Correlations Consensus). This technique reveals a large, bilateral predictive network, which resembles large-scale networks involved in task-driven attention and execution. In sum, we find that: (i) predictive processing seems to occur more in certain brain regions than others, when considering different sensory modalities at a time; (ii) there is no evidence, at the network level, for a distinction between error and prediction processing.

The N300: An Index for Predictive Coding of Complex Visual Objects and Scenes

Predictive coding models can simulate known perceptual or neuronal phenomena, but there have been fewer attempts to identify a reliable neural signature of predictive coding for complex stimuli. In a pair of studies, we test whether the N300 component of the event-related potential, occurring 250–350-ms poststimulus-onset, has the response properties expected for such a signature of perceptual hypothesis testing at the level of whole objects and scenes. We show that N300 amplitudes are smaller to representative (“good exemplars”) compared with less representative (“bad exemplars”) items from natural scene categories. Integrating these results with patterns observed for objects, we establish that, across a variety of visual stimuli, the N300 is responsive to statistical regularity, or the degree to which the input is “expected” (either explicitly or implicitly) based on prior knowledge, with statistically regular images evoking a reduced response. Moreover, we show that the measure exhibits context-dependency; that is, we find the N300 sensitivity to category representativeness when stimuli are congruent with, but not when they are incongruent with, a category pre-cue. Thus, we argue that the N300 is the best candidate to date for an index of perceptual hypotheses testing for complex visual objects and scenes.

Can expectation suppression be explained by reduced attention to predictable stimuli?

The expectation-suppression effect - reduced stimulus-evoked responses to expected stimuli - is widely considered to be an empirical hallmark of reduced prediction errors in the framework of predictive coding. Here we challenge this notion by proposing that that expectation suppression could be explained by a reduced attention effect. Specifically, we argue that reduced responses to predictable stimuli can also be explained by a reduced saliency-driven allocation of attention. We base our discussion mainly on findings in the visual cortex and propose that resolving this controversy requires the assessment of qualitative differences between the ways in which attention and surprise enhance brain responses.

No Evidence that Predictions and Attention Modulate the First Feedforward Sweep of Cortical Information Processing

Predictive coding models propose that predictions (stimulus likelihood) reduce sensory signals as early as primary visual cortex (V1), and that attention (stimulus relevance) can modulate these effects. Indeed, both prediction and attention have been shown to modulate V1 activity, albeit with fMRI, which has low temporal resolution. This leaves it unclear whether these effects reflect a modulation of the first feedforward sweep of visual information processing and/or later, feedback-related activity. In two experiments, we used electroencephalography and orthogonally manipulated spatial predictions and attention to address this issue. Although clear top-down biases were found, as reflected in pre-stimulus alpha-band activity, we found no evidence for top-down effects on the earliest visual cortical processing stage (<80 ms post-stimulus), as indexed by the amplitude of the C1 event-related potential component and multivariate pattern analyses. These findings indicate that initial visual afferent activity may be impenetrable to top-down influences by spatial prediction and attention.

Beat-based and Memory-based Temporal Expectations in Rhythm: Similar Perceptual Effects, Different Underlying Mechanisms

Predicting the timing of incoming information allows the brain to optimize information processing in dynamic environments. Behaviorally, temporal expectations have been shown to facilitate processing of events at expected time points, such as sounds that coincide with the beat in musical rhythm. Yet, temporal expectations can develop based on different forms of structure in the environment, not just the regularity afforded by a musical beat. Little is still known about how different types of temporal expectations are neurally implemented and affect performance. Here, we orthogonally manipulated the periodicity and predictability of rhythmic sequences to examine the mechanisms underlying beat-based and memory-based temporal expectations, respectively. Behaviorally and using EEG, we looked at the effects of beat-based and memory-based expectations on auditory processing when rhythms were task-relevant or task-irrelevant. At expected time points, both beat-based and memory-based expectations facilitated target detection and led to attenuation of P1 and N1 responses, even when expectations were task-irrelevant (unattended). For beat-based expectations, we additionally found reduced target detection and enhanced N1 responses for events at unexpected time points (e.g., off-beat), regardless of the presence of memory-based expectations or task relevance. This latter finding supports the notion that periodicity selectively induces rhythmic fluctuations in neural excitability and furthermore indicates that, although beat-based and memory-based expectations may similarly affect auditory processing of expected events, their underlying neural mechanisms may be different.

Nine-month-old infants update their predictive models of a changing environment

Humans generate internal models of their environment to predict events in the world. As the environments change, our brains adjust to these changes by updating their internal models. Here, we investigated whether and how 9-month-old infants differentially update their models to represent a dynamic environment. Infants observed a predictable sequence of stimuli, which were interrupted by two types of cues. Following the update cue, the pattern was altered, thus, infants were expected to update their predictions for the upcoming stimuli. Because the pattern remained the same after the no-update cue, no subsequent updating was required. Infants showed an amplified negative central (Nc) response when the predictable sequence was interrupted. Late components such as the PSW were also evoked in response to unexpected stimuli; however, we found no evidence for a differential response to the informational value of surprising cues at later stages of processing. Infants rather learned that surprising cues always signal a change in the environment that requires updating. Interestingly, infants responded with an amplified neural response to the absence of an expected change, suggesting a top-down modulation of early sensory processing in infants. Our findings corroborate emerging evidence showing that infants build predictive models early in life.

Attention promotes the neural encoding of prediction errors

The encoding of sensory information in the human brain is thought to be optimised by two principal processes: 'prediction' uses stored information to guide the interpretation of forthcoming sensory events, and 'attention' prioritizes these events according to their behavioural relevance. Despite the ubiquitous contributions of attention and prediction to various aspects of perception and cognition, it remains unknown how they interact to modulate information processing in the brain. A recent extension of predictive coding theory suggests that attention optimises the expected precision of predictions by modulating the synaptic gain of prediction error units. Because prediction errors code for the difference between predictions and sensory signals, this model would suggest that attention increases the selectivity for mismatch information in the neural response to a surprising stimulus. Alternative predictive coding models propose that attention increases the activity of prediction (or 'representation') neurons and would therefore suggest that attention and prediction synergistically modulate selectivity for 'feature information' in the brain. Here, we applied forward encoding models to neural activity recorded via electroencephalography (EEG) as human observers performed a simple visual task to test for the effect of attention on both mismatch and feature information in the neural response to surprising stimuli. Participants attended or ignored a periodic stream of gratings, the orientations of which could be either predictable, surprising, or unpredictable. We found that surprising stimuli evoked neural responses that were encoded according to the difference between predicted and observed stimulus features, and that attention facilitated the encoding of this type of information in the brain. These findings advance our understanding of how attention and prediction modulate information processing in the brain, as well as support the theory that attention optimises precision expectations during hierarchical inference by increasing the gain of prediction errors.

Preattentive and Predictive Processing of Visual Motion

Interaction with the environment requires fast and reliable sensory processing. The visual system is confronted with a continuous flow of high-dimensional input (e.g. orientation, color, motion). From a theoretical point of view, it would be advantageous if critical information was processed independent of attentional load, i.e. preattentively. Here, we hypothesized that visual motion is such a critical signal and aimed for a neural signature of its preattentive encoding. Furthermore, we were interested in the neural correlates of predictability of linear motion trajectories based on the presence or absence of preceding motion. We presented a visual oddball paradigm and studied event-related potentials (ERPs). Stimuli were linearly moving Gabor patches that disappeared behind an occluder. The difference between deviant and standard trials was a trajectory change which happened behind the occluder in deviant trials only, inducing a prediction error. As hypothesized, we found a visual mismatch negativity-component over parietal and occipital electrodes. In a further condition, trials without preceding motion were presented in which the patch just appeared from behind the occluder and, hence, was not predictable. We found larger ERP-components for unpredictable stimuli. In summary, our results provide evidence for a preattentive and predictive processing of linear trajectories of visual motion.

Attentional gain is modulated by probabilistic feature expectations in a spatial cueing task: ERP evidence

Several theoretical and empirical studies suggest that attention and perceptual expectations influence perception in an interactive manner, whereby attentional gain is enhanced for predicted stimuli. The current study assessed whether attention and perceptual expectations interface when they are fully orthogonal, i.e., each of them relates to different stimulus features. We used a spatial cueing task with block-wise spatial attention cues that directed attention to either left or right visual field, in which Gabor gratings of either predicted (more likely) or unpredicted (less likely) orientation were presented. The lateralised posterior N1pc component was additively influenced by attention and perceptual expectations. Bayesian analysis showed no reliable evidence for the interactive effect of attention and expectations on the N1pc amplitude. However, attention and perceptual expectations interactively influenced the frontally distributed anterior N1 component (N1a). The attention effect (i.e., enhanced N1a amplitude in the attended compared to the unattended condition) was observed only for the gratings of predicted orientation, but not in the unpredicted condition. These findings suggest that attention and perceptual expectations interactively influence visual processing within 200 ms after stimulus onset and such joint influence may lead to enhanced endogenous attentional control in the dorsal fronto-parietal attention network.