How previous experience shapes perception in different sensory modalities

Prediction error processing and sharpening of expected information across the face-processing hierarchy

The perception and neural processing of sensory information are strongly influenced by prior expectations. The integration of prior and sensory information can manifest through distinct underlying mechanisms: focusing on unexpected input, denoted as prediction error (PE) processing, or amplifying anticipated information via sharpened representation. In this study, we employed computational modeling using deep neural networks combined with representational similarity analyses of fMRI data to investigate these two processes during face perception. Participants were cued to see face images, some generated by morphing two faces, leading to ambiguity in face identity. We show that expected faces were identified faster and perception of ambiguous faces was shifted towards priors. Multivariate analyses uncovered evidence for PE processing across and beyond the face-processing hierarchy from the occipital face area (OFA), via the fusiform face area, to the anterior temporal lobe, and suggest sharpened representations in the OFA. Our findings support the proposition that the brain represents faces grounded in prior expectations.

Randomness impacts the building of specific priors, visual exploration, and perception in object recognition

Recognising objects is a vital skill on which humans heavily rely to respond quickly and adaptively to their environment. Yet, we lack a full understanding of the role visual information sampling plays in this process, and its relation to the individual's priors. To bridge this gap, the eye-movements of 18 adult participants were recorded during a free-viewing object-recognition task using Dots stimuli1. Participants viewed the stimuli in one of three orders: from most visible to least (Descending), least visible to most (Ascending), or in a randomised order (Random). This dictated the strength of their priors along the experiment. Visibility order influenced the participants' recognition performance and visual exploration. In addition, we found that while orders allowing for stronger priors generally led participants to visually sample more informative locations, this was not the case of Random participants. Indeed, they appeared to behave naïvely, and their use of specific object-related priors was fully impaired, while they maintained the ability to use general, task-related priors to guide their exploration. These findings have important implications for our understanding of perception, which appears to be influenced by complex cognitive processes, even at the basic level of visual sampling during object recognition.

An efficient Bayesian observer model of attractive and repulsive temporal context effects when perceiving multistable dot lattices

In multistable dot lattices, the orientation we perceive is attracted toward the orientation we perceived in the immediately preceding stimulus and repelled from the orientation for which most evidence was present previously (Van Geert, Moors, Haaf, & Wagemans, 2022). Theoretically-inspired models have been proposed to explain the co-occurrence of attractive and repulsive context effects in multistable dot lattice tasks, but these models artificially induced an influence of the previous trial on the current one without detailing the process underlying such an influence (Gepshtein & Kubovy, 2005; Schwiedrzik et al., 2014). We conducted a simulation study to test whether the observed attractive and repulsive context effects could be explained with an efficient Bayesian observer model (Wei & Stocker, 2015). This model assumes variable encoding precision of orientations in line with their frequency of occurrence (i.e., efficient encoding) and takes the dissimilarity between stimulus space and sensory space into account. An efficient Bayesian observer model including both a stimulus and a perceptual level was needed to explain the co-occurrence of both attractive and repulsive temporal context effects. Furthermore, this model could reproduce the empirically observed strong positive correlation between individuals' attractive and repulsive effects (Van Geert et al., 2022), by assuming a positive correlation between temporal integration constants at the stimulus and the perceptual level. To conclude, the study brings evidence that efficient encoding and likelihood repulsion on the stimulus level can explain the repulsive context effect, whereas perceptual prior attraction can explain the attractive temporal context effect when perceiving multistable dot lattices.

Modulation of prefrontal couplings by prior belief-related responses in ventromedial prefrontal cortex

Humans and other animals can maintain constant payoffs in an uncertain environment by steadily re-evaluating and flexibly adjusting current strategy, which largely depends on the interactions between the prefrontal cortex (PFC) and mediodorsal thalamus (MD). While the ventromedial PFC (vmPFC) represents the level of uncertainty (i.e., prior belief about external states), it remains unclear how the brain recruits the PFC-MD network to re-evaluate decision strategy based on the uncertainty. Here, we leverage non-linear dynamic causal modeling on fMRI data to test how prior belief-dependent activity in vmPFC gates the information flow in the PFC-MD network when individuals switch their decision strategy. We show that the prior belief-related responses in vmPFC had a modulatory influence on the connections from dorsolateral PFC (dlPFC) to both, lateral orbitofrontal (lOFC) and MD. Bayesian parameter averaging revealed that only the connection from the dlPFC to lOFC surpassed the significant threshold, which indicates that the weaker the prior belief, the less was the inhibitory influence of the vmPFC on the strength of effective connections from dlPFC to lOFC. These findings suggest that the vmPFC acts as a gatekeeper for the recruitment of processing resources to re-evaluate the decision strategy in situations of high uncertainty.

Stable perceptual phenotype of the magnitude of history biases even in the face of global task complexity

According to a Bayesian framework, visual perception requires active interpretation of noisy sensory signals in light of prior information. One such mechanism, serial dependence, is thought to promote perceptual stability by assimilating current percepts with recent stimulus history. Combining a delayed orientation-adjustment paradigm with predictable (study 1) or unpredictable (study 2) task structure, we test two key predictions of this account in a novel context: first, that serial dependence should persist even in variable environments, and, second, that, within a given observer and context, this behavioral bias should be stable from one occasion to the next. Relying on data of 41 human volunteers and two separate experimental sessions, we confirm both hypotheses. Group-level, attractive serial dependence remained strong even in the face of volatile settings with multiple, unpredictable types of tasks, and, despite considerable interindividual variability, within-subject patterns of attractive and repulsive stimulus-history biases were highly stable from one experimental session to the next. In line with the hypothesized functional role of serial dependence, we propose that, together with previous work, our findings suggest the existence of a more general individual-specific fingerprint with which the past shapes current perception. Congruent with the Bayesian account, interindividual differences may then result from differential weighting of sensory evidence and prior information.

Adaptation in the sensory cortex drives bistable switching during auditory stream segregation

Current theories of perception emphasize the role of neural adaptation, inhibitory competition, and noise as key components that lead to switches in perception. Supporting evidence comes from neurophysiological findings of specific neural signatures in modality-specific and supramodal brain areas that appear to be critical to switches in perception. We used functional magnetic resonance imaging to study brain activity around the time of switches in perception while participants listened to a bistable auditory stream segregation stimulus, which can be heard as one integrated stream of tones or two segregated streams of tones. The auditory thalamus showed more activity around the time of a switch from segregated to integrated compared to time periods of stable perception of integrated; in contrast, the rostral anterior cingulate cortex and the inferior parietal lobule showed more activity around the time of a switch from integrated to segregated compared to time periods of stable perception of segregated streams, consistent with prior findings of asymmetries in brain activity depending on the switch direction. In sound-responsive areas in the auditory cortex, neural activity increased in strength preceding switches in perception and declined in strength over time following switches in perception. Such dynamics in the auditory cortex are consistent with the role of adaptation proposed by computational models of visual and auditory bistable switching, whereby the strength of neural activity decreases following a switch in perception, which eventually destabilizes the current percept enough to lead to a switch to an alternative percept.

Same stimulus, same temporal context, different percept? Individual differences in hysteresis and adaptation when perceiving multistable dot lattices

How we perceptually organize a visual stimulus depends not only on the stimulus itself, but also on the temporal and spatial context in which the stimulus is presented and on the individual processing the stimulus and context. Earlier research found both attractive and repulsive context effects in perception: tendencies to organize visual input similarly to preceding context stimuli (i.e., hysteresis, attraction) co-exist with tendencies that repel the current percept from the organization that is most dominant in these contextual stimuli (i.e., adaptation, repulsion). These processes have been studied mostly on a group level (e.g., Schwiedrzik et al., 2014). Using a Bayesian hierarchical model comparison approach, the present study (N = 75) investigated whether consistent individual differences exist in these attractive and repulsive temporal context effects, with multistable dot lattices as stimuli. In addition, the temporal stability of these individual differences in context effects was investigated, and it was studied how the strength of these effects related to the strength of individual biases for absolute orientations. The results demonstrate that large individual differences in the size of attractive and repulsive context effects exist. Furthermore, these individual differences are highly consistent across timepoints (one to two weeks apart). Although almost everyone showed both effects in the expected direction, not every single individual did. In sum, the study reveals differences in how individuals combine previous input and experience with current input in their perception, and more generally, this teaches us that different individuals can perceive identical stimuli differently, even within a similar context.

Resetting of Auditory and Visual Segregation Occurs After Transient Stimuli of the Same Modality

In the presence of a continually changing sensory environment, maintaining stable but flexible awareness is paramount, and requires continual organization of information. Determining which stimulus features belong together, and which are separate is therefore one of the primary tasks of the sensory systems. Unknown is whether there is a global or sensory-specific mechanism that regulates the final perceptual outcome of this streaming process. To test the extent of modality independence in perceptual control, an auditory streaming experiment, and a visual moving-plaid experiment were performed. Both were designed to evoke alternating perception of an integrated or segregated percept. In both experiments, transient auditory and visual distractor stimuli were presented in separate blocks, such that the distractors did not overlap in frequency or space with the streaming or plaid stimuli, respectively, thus preventing peripheral interference. When a distractor was presented in the opposite modality as the bistable stimulus (visual distractors during auditory streaming or auditory distractors during visual streaming), the probability of percept switching was not significantly different than when no distractor was presented. Conversely, significant differences in switch probability were observed following within-modality distractors, but only when the pre-distractor percept was segregated. Due to the modality-specificity of the distractor-induced resetting, the results suggest that conscious perception is at least partially controlled by modality-specific processing. The fact that the distractors did not have peripheral overlap with the bistable stimuli indicates that the perceptual reset is due to interference at a locus in which stimuli of different frequencies and spatial locations are integrated.

What you hear first, is what you get: Initial metrical cue presentation modulates syllable detection in sentence processing

Auditory rhythms create powerful expectations for the listener. Rhythmic cues with the same temporal structure as subsequent sentences enhance processing compared with irregular or mismatched cues. In the present study, we focus on syllable detection following matched rhythmic cues. Cues were aligned with subsequent sentences at the syllable (low-level cue) or the accented syllable (high-level cue) level. A different group of participants performed the task without cues to provide a baseline. We hypothesized that unaccented syllable detection would be faster after low-level cues, and accented syllable detection would be faster after high-level cues. There was no difference in syllable detection depending on whether the sentence was preceded by a high-level or low-level cue. However, the results revealed a priming effect of the cue that participants heard first. Participants who heard a high-level cue first were faster to detect accented than unaccented syllables, and faster to detect accented syllables than participants who heard a low-level cue first. The low-level-first participants showed no difference between detection of accented and unaccented syllables. The baseline experiment confirmed that hearing a low-level cue first removed the benefit of the high-level grouping structure for accented syllables. These results suggest that the initially perceived rhythmic structure influenced subsequent cue perception and its influence on syllable detection. Results are discussed in terms of dynamic attending, temporal context effects, and implications for context effects in neural entrainment.

Transcranial Magnetic Stimulation-Induced Motor Cortex Activity Influences Visual Awareness Judgments

The influence of non-visual information on visual awareness judgments has recently gained substantial interest. Using single-pulse transcranial magnetic stimulation (TMS), we investigate the potential contribution of evidence from the motor system to judgment of visual awareness. We hypothesized that TMS-induced activity in the primary motor cortex (M1) would increase reported visual awareness as compared to the control condition. Additionally, we investigated whether TMS-induced motor-evoked potential (MEP) could measure accumulated evidence for stimulus perception. Following stimulus presentation and TMS, participants first rated their visual awareness verbally using the Perceptual Awareness Scale (PAS), after which they responded manually to a Gabor orientation identification task. Delivering TMS to M1 resulted in higher average awareness ratings as compared to the control condition, in both correct and incorrect identification task response trials, when the hand with which participants responded was contralateral to the stimulated hemisphere (TMS-response-congruent trials). This effect was accompanied by longer PAS response times (RTs), irrespective of the congruence between TMS and identification response. Moreover, longer identification RTs were observed in TMS-response-congruent trials in the M1 condition as compared to the control condition. Additionally, the amplitudes of MEPs were related to the awareness ratings when response congruence was taken into account. We argue that MEP can serve as an indirect measure of evidence accumulated for stimulus perception and that longer PAS RTs and higher amplitudes of MEPs in the M1 condition reflect integration of additional evidence with visual awareness judgment. In conclusion, we advocate that motor activity influences perceptual awareness judgments.

Neural and Behavioral Evidence for Frequency-Selective Context Effects in Rhythm Processing in Humans

When listening to music, people often perceive and move along with a periodic meter. However, the dynamics of mapping between meter perception and the acoustic cues to meter periodicities in the sensory input remain largely unknown. To capture these dynamics, we recorded the electroencephalography while nonmusician and musician participants listened to nonrepeating rhythmic sequences, where acoustic cues to meter frequencies either gradually decreased (from regular to degraded) or increased (from degraded to regular). The results revealed greater neural activity selectively elicited at meter frequencies when the sequence gradually changed from regular to degraded compared with the opposite. Importantly, this effect was unlikely to arise from overall gain, or low-level auditory processing, as revealed by physiological modeling. Moreover, the context effect was more pronounced in nonmusicians, who also demonstrated facilitated sensory-motor synchronization with the meter for sequences that started as regular. In contrast, musicians showed weaker effects of recent context in their neural responses and robust ability to move along with the meter irrespective of stimulus degradation. Together, our results demonstrate that brain activity elicited by rhythm does not only reflect passive tracking of stimulus features, but represents continuous integration of sensory input with recent context.

The perceptual relevance of balance, evenness, and entropy in musical rhythms

There is an uncountable number of different ways of characterizing almost any given real-world stimulus. This necessitates finding stimulus features that are perceptually relevant - that is, they have distinct and independent effects on the perception and cognition of the stimulus. Here, we provide a theoretical framework for empirically testing the perceptual relevance of stimulus features through their association with recognition, memory bias, and æsthetic evaluation. We deploy this framework in the auditory domain to explore the perceptual relevance of three recently developed mathematical characterizations of periodic temporal patterns: balance, evenness, and interonset interval entropy. By modelling recognition responses and liking ratings from 177 participants listening to a total of 1252 different musical rhythms, we obtain very strong evidence that all three features have distinct effects on the memory for, and the liking of, musical rhythms. Interonset interval entropy is a measure of the unpredictability of a rhythm derived from the distribution of its durations. Balance and evenness are both obtained from the discrete Fourier transform (DFT) of periodic patterns represented as points on the unit circle, and we introduce a teleological explanation for their perceptual relevance: the DFT coefficients representing balance and evenness are relatively robust to small random temporal perturbations and hence are coherent in noisy environments. This theory suggests further research to explore the meaning and relevance of robust coefficients such as these to the perception of patterns that are periodic in time and, possibly, space.

A challenge for predictive coding: Representational or experiential diversity?

To become a unifying theory of brain function, predictive processing (PP) must accommodate its rich representational diversity. Gilead et al. claim such diversity requires a multi-process theory, and thus is out of reach for PP, which postulates a universal canonical computation. We contend this argument and instead propose that PP fails to account for the experiential level of representations.

Visual detection of time-varying signals: Opposing biases and their timescales

Human visual perception is a complex, dynamic and fluctuating process. In addition to the incoming visual stimulus, it is affected by many other factors including temporal context, both external and internal to the observer. In this study we investigate the dynamic properties of psychophysical responses to a continuous stream of visual near-threshold detection tasks. We manipulate the incoming signals to have temporal structures with various characteristic timescales. Responses of human observers to these signals are analyzed using tools that highlight their dynamical features as well. Our experiments show two opposing biases that shape perceptual decision making simultaneously: positive recency, biasing towards repeated response; and adaptation, entailing an increased probability of changed response. While both these effects have been reported in previous work, our results shed new light on the timescales involved in these effects, and on their interplay with varying inputs. We find that positive recency is a short-term bias, inversely correlated with response time, suggesting it can be compensated by afterthought. Adaptation, in contrast, reflects trends over longer times possibly including multiple previous trials. Our entire dataset, which includes different input signal temporal structures, is consistent with a simple model with the two biases characterized by a fixed parameter set. These results suggest that perceptual biases are inherent features which are not flexible to tune to input signals.

Medial prefrontal cortex supports perceptual memory

Our visual environment constantly changes, yet we experience the world as a stable, unified whole. How is this stability achieved? It has been proposed that the brain preserves an implicit perceptual memory in sensory cortices [1] which stabilizes perception towards previously experienced states [2,3]. The role of higher-order areas, especially prefrontal cortex (PFC), in perceptual memory is less explored. Because PFC exhibits long neural time constants, invariance properties, and large receptive fields which may stabilize perception against time-varying inputs, it seems particularly suited to implement perceptual memory [4]. Support for this idea comes from a neuroimaging study reporting that dorsomedial PFC (dmPFC) correlates with perceptual memory [5]. But dmPFC also participates in decision making [6], so its contribution to perceptual memory could arise on a post-perceptual, decisional level [7]. To determine which role, if any, PFC plays in perceptual memory, we obtained direct intracranial recordings in six epilepsy patients while they performed sequential orientation judgements on ambiguous stimuli known to elicit perceptual memory [8]. We found that dmPFC activity in the high gamma frequency band (HGB, 70-150 Hz) correlates with perceptual memory. This effect is anatomically specific to dmPFC and functionally specific for memories of preceding percepts. Further, dmPFC appears to play a causal role, as a patient with a lesion in this area showed impaired perceptual memory. Thus, dmPFC integrates current sensory information with prior percepts, stabilizing visual experience against the perpetual variability of our surroundings.

Laws of concatenated perception: Vision goes for novelty, decisions for perseverance

Every instant of perception depends on a cascade of brain processes calibrated to the history of sensory and decisional events. In the present work, we show that human visual perception is constantly shaped by two contrasting forces exerted by sensory adaptation and past decisions. In a series of experiments, we used multilevel modeling and cross-validation approaches to investigate the impact of previous stimuli and decisions on behavioral reports during adjustment and forced-choice tasks. Our results revealed that each perceptual report is permeated by opposite biases from a hierarchy of serially dependent processes: Low-level adaptation repels perception away from previous stimuli, whereas decisional traces attract perceptual reports toward the recent past. In this hierarchy of serial dependence, "continuity fields" arise from the inertia of decisional templates and not from low-level sensory processes. This finding is consistent with a Two-process model of serial dependence in which the persistence of readout weights in a decision unit compensates for sensory adaptation, leading to attractive biases in sequential perception. We propose a unified account of serial dependence in which functionally distinct mechanisms, operating at different stages, promote the differentiation and integration of visual information over time.

Reference repulsion is not a perceptual illusion

Perceptual decisions are often influenced by contextual factors. For instance, when engaged in a visual discrimination task against a reference boundary, subjective reports about the judged stimulus feature are biased away from the boundary - a phenomenon termed reference repulsion. Until recently, this phenomenon has been thought to reflect a perceptual illusion regarding the appearance of the stimulus, but new evidence suggests that it may rather reflect a post-perceptual decision bias. To shed light on this issue, we examined whether and how orientation judgments affect perceptual appearance. In a first experiment, we confirmed that after judging a grating stimulus against a discrimination boundary, the subsequent reproduction response was indeed repelled from the boundary. To investigate the perceptual nature of this bias, in a second experiment we measured the perceived orientation of the grating stimulus more directly, in comparison to a reference stimulus visible at the same time. Although we did observe a small repulsive bias away from the boundary, this bias was explained by random trial-by-trial fluctuations in sensory representations together with classical stimulus adaptation effects and did not reflect a systematic bias due to the discrimination judgment. Overall, the current study indicates that discrimination judgments do not elicit a perceptual illusion and points towards a post-perceptual locus of reference repulsion.

Restoration and Efficiency of the Neural Processing of Continuous Speech Are Promoted by Prior Knowledge

Sufficiently noisy listening conditions can completely mask the acoustic signal of significant parts of a sentence, and yet listeners may still report the perception of hearing the masked speech. This occurs even when the speech signal is removed entirely, if the gap is filled with stationary noise, a phenomenon known as perceptual restoration. At the neural level, however, it is unclear the extent to which the neural representation of missing extended speech sequences is similar to the dynamic neural representation of ordinary continuous speech. Using auditory magnetoencephalography (MEG), we show that stimulus reconstruction, a technique developed for use with neural representations of ordinary speech, works also for the missing speech segments replaced by noise, even when spanning several phonemes and words. The reconstruction fidelity of the missing speech, up to 25% of what would be attained if present, depends however on listeners' familiarity with the missing segment. This same familiarity also speeds up the most prominent stage of the cortical processing of ordinary speech by approximately 5 ms. Both effects disappear when listeners have no or little prior experience with the speech segment. The results are consistent with adaptive expectation mechanisms that consolidate detailed representations about speech sounds as identifiable factors assisting automatic restoration over ecologically relevant timescales.

Suboptimality in perceptual decision making

Human perceptual decisions are often described as optimal. Critics of this view have argued that claims of optimality are overly flexible and lack explanatory power. Meanwhile, advocates for optimality have countered that such criticisms single out a few selected papers. To elucidate the issue of optimality in perceptual decision making, we review the extensive literature on suboptimal performance in perceptual tasks. We discuss eight different classes of suboptimal perceptual decisions, including improper placement, maintenance, and adjustment of perceptual criteria; inadequate tradeoff between speed and accuracy; inappropriate confidence ratings; misweightings in cue combination; and findings related to various perceptual illusions and biases. In addition, we discuss conceptual shortcomings of a focus on optimality, such as definitional difficulties and the limited value of optimality claims in and of themselves. We therefore advocate that the field drop its emphasis on whether observed behavior is optimal and instead concentrate on building and testing detailed observer models that explain behavior across a wide range of tasks. To facilitate this transition, we compile the proposed hypotheses regarding the origins of suboptimal perceptual decisions reviewed here. We argue that verifying, rejecting, and expanding these explanations for suboptimal behavior - rather than assessing optimality per se - should be among the major goals of the science of perceptual decision making.

Prior context in audition informs binding and shapes simple features

A perceptual phenomenon is reported, whereby prior acoustic context has a large, rapid and long-lasting effect on a basic auditory judgement. Pairs of tones were devised to include ambiguous transitions between frequency components, such that listeners were equally likely to report an upward or downward 'pitch' shift between tones. We show that presenting context tones before the ambiguous pair almost fully determines the perceived direction of shift. The context effect generalizes to a wide range of temporal and spectral scales, encompassing the characteristics of most realistic auditory scenes. Magnetoencephalographic recordings show that a relative reduction in neural responsivity is correlated to the behavioural effect. Finally, a computational model reproduces behavioural results, by implementing a simple constraint of continuity for binding successive sounds in a probabilistic manner. Contextual processing, mediated by ubiquitous neural mechanisms such as adaptation, may be crucial to track complex sound sources over time.

Perceptual Decision-Making: Picking the Low-Hanging Fruit?

How do we decide what we perceive? Obviously, we base our decisions on sensory evidence. However, a new and surprising study by Hagura et al. shows that our perceptual decisions are also biased by the action costs that are associated with our decisions.

Perceptual suppression of predicted natural images

Perception is shaped not only by current sensory inputs but also by expectations generated from past sensory experience. Humans viewing ambiguous stimuli in a stable visual environment are generally more likely to see the perceptual interpretation that matches their expectations, but it is less clear how expectations affect perception when the environment is changing predictably. We used statistical learning to teach observers arbitrary sequences of natural images and employed binocular rivalry to measure perceptual selection as a function of predictive context. In contrast to previous demonstrations of preferential selection of predicted images for conscious awareness, we found that recently acquired sequence predictions biased perceptual selection toward unexpected natural images and image categories. These perceptual biases were not associated with explicit recall of the learned image sequences. Our results show that exposure to arbitrary sequential structure in the environment impacts subsequent visual perceptual selection and awareness. Specifically, for natural image sequences, the visual system prioritizes what is surprising, or statistically informative, over what is expected, or statistically likely.