A Bayesian observer model constrained by efficient coding can explain 'anti-Bayesian' percepts

Jointly efficient encoding and decoding in neural populations

The efficient coding approach proposes that neural systems represent as much sensory information as biological constraints allow. It aims at formalizing encoding as a constrained optimal process. A different approach, that aims at formalizing decoding, proposes that neural systems instantiate a generative model of the sensory world. Here, we put forth a normative framework that characterizes neural systems as jointly optimizing encoding and decoding. It takes the form of a variational autoencoder: sensory stimuli are encoded in the noisy activity of neurons to be interpreted by a flexible decoder; encoding must allow for an accurate stimulus reconstruction from neural activity. Jointly, neural activity is required to represent the statistics of latent features which are mapped by the decoder into distributions over sensory stimuli; decoding correspondingly optimizes the accuracy of the generative model. This framework yields in a family of encoding-decoding models, which result in equally accurate generative models, indexed by a measure of the stimulus-induced deviation of neural activity from the marginal distribution over neural activity. Each member of this family predicts a specific relation between properties of the sensory neurons-such as the arrangement of the tuning curve means (preferred stimuli) and widths (degrees of selectivity) in the population-as a function of the statistics of the sensory world. Our approach thus generalizes the efficient coding approach. Notably, here, the form of the constraint on the optimization derives from the requirement of an accurate generative model, while it is arbitrary in efficient coding models. Moreover, solutions do not require the knowledge of the stimulus distribution, but are learned on the basis of data samples; the constraint further acts as regularizer, allowing the model to generalize beyond the training data. Finally, we characterize the family of models we obtain through alternate measures of performance, such as the error in stimulus reconstruction. We find that a range of models admits comparable performance; in particular, a population of sensory neurons with broad tuning curves as observed experimentally yields both low reconstruction stimulus error and an accurate generative model that generalizes robustly to unseen data.

A Bayesian observer model reveals a prior for natural daylights in hue perception

Incorporating statistical characteristics of stimuli in perceptual processing can be highly beneficial for reliable estimation from noisy sensory measurements but may generate perceptual bias. According to Bayesian inference, perceptual biases arise from the integration of internal priors with noisy sensory inputs. In this study, we used a Bayesian observer model to derive biases and priors in hue perception based on discrimination data for hue ensembles with varying levels of chromatic noise. Our results showed that discrimination thresholds for isoluminant stimuli with hue defined by azimuth angle in cone-opponent color space exhibited a bimodal pattern, with lowest thresholds near a non-cardinal blue-yellow axis that aligns closely with the variation of natural daylights. Perceptual biases showed zero crossings around this axis, indicating repulsion away from yellow and attraction towards blue. These biases could be explained by the Bayesian observer model through a non-uniform prior with a preference for blue. Our findings suggest that visual processing takes advantage of knowledge of the distribution of colors in natural environments for hue perception.

Reward prediction error neurons implement an efficient code for reward

We use efficient coding principles borrowed from sensory neuroscience to derive the optimal neural population to encode a reward distribution. We show that the responses of dopaminergic reward prediction error neurons in mouse and macaque are similar to those of the efficient code in the following ways: the neurons have a broad distribution of midpoints covering the reward distribution; neurons with higher thresholds have higher gains, more convex tuning functions and lower slopes; and their slope is higher when the reward distribution is narrower. Furthermore, we derive learning rules that converge to the efficient code. The learning rule for the position of the neuron on the reward axis closely resembles distributional reinforcement learning. Thus, reward prediction error neuron responses may be optimized to broadcast an efficient reward signal, forming a connection between efficient coding and reinforcement learning, two of the most successful theories in computational neuroscience.

Repulsion bias is insensitive to spatial attention, yet expands during active working memory maintenance

Our brain sometimes represents visual information in a biased manner. Multiple visual features presented simultaneously or sequentially may interact with each other when we perceive them or maintain them in visual working memory (WM), giving rise to report bias. How goal-directed attention influences target representation is not fully understood, especially concerning whether attention towards distractors modulates report bias for the target. Our study investigated the WM biases of the target when it is concurrent with (1) one attended distractor only, (2) one unattended distractor only, and (3) both kinds of distractors during perception. It was found that the target WM is reported as being repelled away from concurrent distractors, attended or unattended, suggesting attention is not necessary for the occurrence of repulsion bias during perception. Furthermore, goal-directed attention towards the distractors modulates the strength of interitem interaction, and the repulsion bias was found to be stronger when attention was directed toward the distractor than when it was not. However, the exaggerated repulsion associated with the attended distractor is likely due to increased relevance to the memory task and (or) WM load instead of spatial attention. In contrast, spatial attention towards the distractor increases the chances of misreporting the distractor for the target.

Adaptation in the visual system: Networked fatigue or suppressed prediction error signalling?

Our brains are constantly adapting to changes in our visual environments. Neural adaptation exerts a persistent influence on the activity of sensory neurons and our perceptual experience, however there is a lack of consensus regarding how adaptation is implemented in the visual system. One account describes fatigue-based mechanisms embedded within local networks of stimulus-selective neurons (networked fatigue models). Another depicts adaptation as a product of stimulus expectations (predictive coding models). In this review, I evaluate neuroimaging and psychophysical evidence that poses fundamental problems for predictive coding models of neural adaptation. Specifically, I discuss observations of distinct repetition and expectation effects, as well as incorrect predictions of repulsive adaptation aftereffects made by predictive coding accounts. Based on this evidence, I argue that networked fatigue models provide a more parsimonious account of adaptation effects in the visual system. Although stimulus expectations can be formed based on recent stimulation history, any consequences of these expectations are likely to co-occur (or interact) with effects of fatigue-based adaptation. I conclude by proposing novel, testable hypotheses relating to interactions between fatigue-based adaptation and other predictive processes, focusing on stimulus feature extrapolation phenomena.

Working memory flips the direction of serial bias through memory-based decision

Reported perception of a new stimulus is either attracted toward or repelled away from task-irrelevant prior stimuli. While prevailing theories propose that the opposing serial biases may stem from distinct stages of information processing, the exact role of working memory (WM) in the serial bias remains unclear despite its consistent involvement in nearly all pertinent studies. Additionally, it is not well understood whether this bias is primarily driven by the biased representation itself or by the decision-making process for the new stimulus. In the present study, we used an orientation delayed estimation paradigm with an attention-demanding intervening task, designed to disrupt the maintenance of stimulus information to investigate the role of WM in serial bias. In the analysis, we scrutinized the trajectory of mouse reports and response time to investigate how the response unfolds over time. Our findings indicate that the serial bias went from repulsive to attractive when WM maintenance was interrupted by the intervening task, and that the associated response trajectories and response time exhibited patterns that cannot be explained by the biased representation alone. These results demonstrate that the task-irrelevant prior stimulus influences the decision for the new stimulus, with the direction of the bias being determined by attentional demand during WM maintenance, thereby placing significant constraints on existing theories on the serial bias effect.

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Oblique warping: A general distortion of spatial perception

There are many putatively distinct phenomena related to perception in the oblique regions of space. For instance, the classic oblique effect describes a deficit in visual acuity for oriented lines in the obliques, and classic "prototype effects" reflect a bias to misplace objects towards the oblique regions of space. Yet these effects are explained in very different terms: The oblique effect itself is often understood as arising from orientation-selective neurons, whereas prototype effects are described as arising from categorical biases. Here, we explore the possibility that these effects (and others) may stem from a single underlying spatial distortion. We show that there is a general distortion of (angular) space in the oblique regions that influences not only orientation judgments, but also location, extent, and size. We argue that these findings reflect oblique warping, a general distortion of spatial representations in the oblique regions which may be the root cause of many oblique effects.

The color of fruits in photographs and still life paintings

Still life paintings comprise a wealth of data on visual perception. Prior work has shown that the color statistics of objects show a marked bias for warm colors. Here, we ask about the relative chromatic contrast of these object-associated colors compared with background colors in still life paintings. We reasoned that, owing to the memory color effect, where the color of familiar objects is perceived more saturated, warm colors will be relatively more saturated than cool colors in still life paintings as compared with photographs. We analyzed color in 108 slides of still life paintings of fruit from the teaching slide collection of the Fogg University Art Museum and 41 color-calibrated photographs of fruit from the McGill data set. The results show that the relatively higher chromatic contrast of warm colors was greater for paintings compared with photographs, consistent with the hypothesis.

A paradigm for characterizing motion misperception in people with typical vision and low vision

PURPOSE

We aimed to develop a paradigm that can efficiently characterize motion percepts in people with low vision and compare their responses with well-known misperceptions made by people with typical vision when targets are hard to see.

METHODS

We recruited a small cohort of individuals with reduced acuity and contrast sensitivity (n = 5) as well as a comparison cohort with typical vision (n = 5) to complete a psychophysical study. Study participants were asked to judge the motion direction of a tilted rhombus that was either high or low contrast. In a series of trials, the rhombus oscillated vertically, horizontally, or diagonally. Participants indicated the perceived motion direction using a number wheel with 12 possible directions, and statistical tests were used to examine response biases.

RESULTS

All participants with typical vision showed systematic misperceptions well predicted by a Bayesian inference model. Specifically, their perception of vertical or horizontal motion was biased toward directions orthogonal to the long axis of the rhombus. They had larger biases for hard-to-see (low contrast) stimuli. Two participants with low vision had a similar bias, but with no difference between high- and low-contrast stimuli. The other participants with low vision were unbiased in their percepts or biased in the opposite direction.

CONCLUSIONS

Our results suggest that some people with low vision may misperceive motion in a systematic way similar to people with typical vision. However, we observed large individual differences. Future work will aim to uncover reasons for such differences and identify aspects of vision that predict susceptibility.

Conflicts between short- and long-term experiences affect visual perception through modulating sensory or motor response systems: Evidence from Bayesian inference models

The independent effects of short- and long-term experiences on visual perception have been discussed for decades. However, no study has investigated whether and how these experiences simultaneously affect our visual perception. To address this question, we asked participants to estimate their self-motion directions (i.e., headings) simulated from optic flow, in which a long-term experience learned in everyday life (i.e., straight-forward motion being more common than lateral motion) plays an important role. The headings were selected from three distributions that resembled a peak, a hill, and a flat line, creating different short-term experiences. Importantly, the proportions of headings deviating from the straight-forward motion gradually increased in the peak, hill, and flat distributions, leading to a greater conflict between long- and short-term experiences. The results showed that participants biased their heading estimates towards the straight-ahead direction and previously seen headings, which increased with the growing experience conflict. This suggests that both long- and short-term experiences simultaneously affect visual perception. Finally, we developed two Bayesian models (Model 1 vs. Model 2) based on two assumptions that the experience conflict altered the likelihood distribution of sensory representation or the motor response system. The results showed that both models accurately predicted participants' estimation biases. However, Model 1 predicted a higher variance of serial dependence compared to Model 2, while Model 2 predicted a higher variance of the bias towards the straight-ahead direction compared to Model 1. This suggests that the experience conflict can influence visual perception by affecting both sensory and motor response systems. Taken together, the current study systematically revealed the effects of long- and short-term experiences on visual perception and the underlying Bayesian processing mechanisms.

Perceptual similarity judgments do not predict the distribution of errors in working memory

Population coding models provide a quantitative account of visual working memory (VWM) retrieval errors with a plausible link to the response characteristics of sensory neurons. Recent work has provided an important new perspective linking population coding to variables of signal detection, including d-prime, and put forward a new hypothesis: that the distribution of recall errors on, for example, a color wheel, is a consequence of the psychological similarity between points in that stimulus space, such that the exponential-like psychophysical distance scaling function can fulfil the role of population tuning and obviate the need to fit a tuning width parameter to recall data. Using four different visual feature spaces, we measured psychophysical similarity and memory errors in the same participants. Our results revealed strong evidence for a common source of variability affecting similarity judgments and recall estimates but did not support any consistent relationship between psychophysical similarity functions and VWM errors. At the group level, the responsiveness functions obtained from the psychophysical similarity task diverged strongly from those that provided the best fit to working memory errors. At the individual level, we found convincing evidence against an association between observed and best-fitting similarity functions. Finally, our results show that the newly proposed exponential-like responsiveness function has in general no advantage over the canonical von Mises (circular normal) function assumed by previous population coding models. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

An efficient Bayesian observer model of attractive and repulsive temporal context effects when perceiving multistable dot lattices

In multistable dot lattices, the orientation we perceive is attracted toward the orientation we perceived in the immediately preceding stimulus and repelled from the orientation for which most evidence was present previously (Van Geert, Moors, Haaf, & Wagemans, 2022). Theoretically-inspired models have been proposed to explain the co-occurrence of attractive and repulsive context effects in multistable dot lattice tasks, but these models artificially induced an influence of the previous trial on the current one without detailing the process underlying such an influence (Gepshtein & Kubovy, 2005; Schwiedrzik et al., 2014). We conducted a simulation study to test whether the observed attractive and repulsive context effects could be explained with an efficient Bayesian observer model (Wei & Stocker, 2015). This model assumes variable encoding precision of orientations in line with their frequency of occurrence (i.e., efficient encoding) and takes the dissimilarity between stimulus space and sensory space into account. An efficient Bayesian observer model including both a stimulus and a perceptual level was needed to explain the co-occurrence of both attractive and repulsive temporal context effects. Furthermore, this model could reproduce the empirically observed strong positive correlation between individuals' attractive and repulsive effects (Van Geert et al., 2022), by assuming a positive correlation between temporal integration constants at the stimulus and the perceptual level. To conclude, the study brings evidence that efficient encoding and likelihood repulsion on the stimulus level can explain the repulsive context effect, whereas perceptual prior attraction can explain the attractive temporal context effect when perceiving multistable dot lattices.

Stereoscopic slant contrast revisited

The perceived slant of a stereoscopic surface is altered by the presence of a surrounding surface, a phenomenon termed stereo slant contrast. Previous studies have shown that a slanted surround causes a fronto-parallel surface to appear slanted in the opposite direction, an instance of "bidirectional" contrast. A few studies have examined slant contrast using slanted as opposed to fronto-parallel test surfaces, and these also have shown slant contrast. Here, we use a matching method to examine slant contrast over a wide range of combinations of surround and test slants, one aim being to determine whether stereo slant contrast transfers across opposite directions of test and surround slant. We also examine the effect of the test on the perceived slant of the surround. Test slant contrast was found to be bidirectional in virtually all test-surround combinations and transferred across opposite test and surround slants, with little or no decline in magnitude as the test-surround slant difference approached the limit. There was a weak bidirectional effect of the test slant on the perceived slant of the surround. We consider how our results might be explained by four mechanisms: (a) normalization of stereo slant to vertical; (b) divisive normalization of stereo slant channels in a manner analogous to the tilt illusion; (c) interactions between center and surround disparity-gradient detectors; and (d) uncertainty in slant estimation. We conclude that the third of these (interactions between center and surround disparity-gradient detectors) is the most likely cause of stereo slant contrast.

A unifying theory explains seemingly contradictory biases in perceptual estimation

Perceptual biases are widely regarded as offering a window into the neural computations underlying perception. To understand these biases, previous work has proposed a number of conceptually different, and even seemingly contradictory, explanations, including attraction to a Bayesian prior, repulsion from the prior due to efficient coding and central tendency effects on a bounded range. We present a unifying Bayesian theory of biases in perceptual estimation derived from first principles. We demonstrate theoretically an additive decomposition of perceptual biases into attraction to a prior, repulsion away from regions with high encoding precision and regression away from the boundary. The results reveal a simple and universal rule for predicting the direction of perceptual biases. Our theory accounts for, and yields, new insights regarding biases in the perception of a variety of stimulus attributes, including orientation, color and magnitude. These results provide important constraints on the neural implementations of Bayesian computations.

Cardinal bias interacts with the stimulus history bias in orientation working memory

Reports in a visual working memory(WM) task exhibit biases related to the categorical structure of the stimulus space (e.g., cardinal bias) as well as biases related to previously seen stumuli (e.g., serial bias). While these biases are common and can occur simultaneously, the extent to which they interact in WM remains unknown. In the present study, I used orientation delayed estimation tasks known to produce both cardinal and serial biases and found that the serial bias systematically varied based on the relative positions of the cardinal axis and the preceding stimulus in orientation space. When they were positioned in a way that generated cardinal and serial biases in the same direction (i.e., on the same side of the target orientation), reports for the target orientation exhibited a regular repulsive serial bias. However, when their positions resulted in the biases in the opposite directions (i.e., on the opposite side of the target orientation), no serial bias occurred. This absence of serial bias was replicated in a follow-up experiment where the locations of the stimulus orientation and the response probe were completely randomized, suggesting that the interaction occurs independently from location-based response preparation processes. Together, these results demonstrate that the prior stimulus and the cardinal axis impose interactive impact on the processing of new stimulus, producing differential patterns of serial bias depending on the specific stimulus being processed. These findings place significant implications on computational models addressing the nature of the stimulus history effect and its underlying mechanisms.

Enhanced Working Memory Representations for Rare Events

Rare events (oddballs) produce a variety of enhanced physiological responses relative to frequent events (standards), including the P3b component of the event-related potential (ERP) waveform. Previous research has suggested that the P3b component is related to working memory, which implies that working memory representations will be enhanced for rare stimuli. To test this hypothesis, we devised a modified oddball paradigm in which the target was a disk presented at one of 16 different locations, which were divided into a rare set and a frequent set. Participants made a binary response on each trial to report whether the target appeared in the rare set or the frequent set. As expected, the P3b was much larger for stimuli appearing at a location within the rare set. We also included occasional probe trials in which the subject reported the exact location of the target. We found that these reports were more accurate for locations within the rare set than for locations within the frequent set. Moreover, the mean accuracy of these reports was correlated with the mean amplitude of the P3b. We also applied multivariate pattern analysis to the ERP data to "decode" the remembered location of the target. Decoding accuracy was greater for locations within the rare set than for locations within the frequent set. These behavioral and electrophysiological results demonstrate that although both frequent and rare events are stored in working memory, the representations are enhanced for rare events.

Rationality, preferences, and emotions with biological constraints: it all starts from our senses

Is the role of our sensory systems to represent the physical world as accurately as possible? If so, are our preferences and emotions, often deemed irrational, decoupled from these 'ground-truth' sensory experiences? We show why the answer to both questions is 'no'. Brain function is metabolically costly, and the brain loses some fraction of the information that it encodes and transmits. Therefore, if brains maximize objective functions that increase the fitness of their species, they should adapt to the objective-maximizing rules of the environment at the earliest stages of sensory processing. Consequently, observed 'irrationalities', preferences, and emotions stem from the necessity for our early sensory systems to adapt and process information while considering the metabolic costs and internal states of the organism.

What's in a sample? Epistemic uncertainty and metacognitive awareness in risk taking

In a fundamentally uncertain world, sound information processing is a prerequisite for effective behavior. Given that information processing is subject to inevitable cognitive imprecision, decision makers should adapt to this imprecision and to the resulting epistemic uncertainty when taking risks. We tested this metacognitive ability in two experiments in which participants estimated the expected value of different number distributions from sequential samples and then bet on their own estimation accuracy. Results show that estimates were imprecise, and this imprecision increased with higher distributional standard deviations. Importantly, participants adapted their risk-taking behavior to this imprecision and hence deviated from the predictions of Bayesian models of uncertainty that assume perfect integration of information. To explain these results, we developed a computational model that combines Bayesian updating with a metacognitive awareness of cognitive imprecision in the integration of information. Modeling results were robust to the inclusion of an empirical measure of participants' perceived variability. In sum, we show that cognitive imprecision is crucial to understanding risk taking in decisions from experience. The results further demonstrate the importance of metacognitive awareness as a cognitive building block for adaptive behavior under (partial) uncertainty.

Emergent human-like covert attention in feedforward convolutional neural networks

Covert attention allows the selection of locations or features of the visual scene without moving the eyes. Cues and contexts predictive of a target's location orient covert attention and improve perceptual performance. The performance benefits are widely attributed to theories of covert attention as a limited resource, zoom, spotlight, or weighting of visual information. However, such concepts are difficult to map to neuronal populations. We show that a feedforward convolutional neural network (CNN) trained on images to optimize target detection accuracy and with no explicit incorporation of an attention mechanism, a limited resource, or feedback connections learns to utilize cues and contexts in the three most prominent covert attention tasks (Posner cueing, set size effects in search, and contextual cueing) and predicts the cue/context influences on human accuracy. The CNN's cueing/context effects generalize across network training schemes, to peripheral and central pre-cues, discrimination tasks, and reaction time measures, and critically do not vary with reductions in network resources (size). The CNN shows comparable cueing/context effects to a model that optimally uses image information to make decisions (Bayesian ideal observer) but generalizes these effects to cue instances unseen during training. Together, the findings suggest that human-like behavioral signatures of covert attention in the three landmark paradigms might be an emergent property of task accuracy optimization in neuronal populations without positing limited attentional resources. The findings might explain recent behavioral results showing cueing and context effects across a variety of simple organisms with no neocortex, from archerfish to fruit flies.

The push-pull of serial dependence effects: Attraction to the prior response and repulsion from the prior stimulus

In the "serial dependence" effect, responses to visual stimuli appear biased toward the last trial's stimulus. However, several kinds of serial dependence exist, with some reflecting prior stimuli and others reflecting prior responses. One-factor analyses consider the prior stimulus alone or the prior response alone and can consider both variables only via separate analyses. We demonstrate that one-factor analyses are potentially misleading and can reach conclusions that are opposite from the truth if both dependencies exist. To address this limitation, we developed two-factor analyses (model comparison with hierarchical Bayesian modeling and an empirical "quadrant analysis"), which consider trial-by-trial combinations of prior response and prior stimulus. Two-factor analyses can tease apart the two dependencies if applied to a sufficiently large dataset. We applied these analyses to a new study and to four previously published studies. When applying a model that included the possibility of both dependencies, there was no evidence of attraction to the prior stimulus in any dataset, but there was evidence of attraction to the prior response in all datasets. Two of the datasets contained sufficient constraint to determine that both dependencies were needed to explain the results. For these datasets, the dependency on the prior stimulus was repulsive rather than attractive. Our results are consistent with the claim that both dependencies exist in most serial dependence studies (the two-dependence model was not ruled out for any dataset) and, furthermore, that the two dependencies work against each other.

Attention influences the effects of the previous form orientation on the current motion direction estimation

Recent studies have found that the estimates of motion directions are biased toward the previous form orientations, showing serial dependence, and the serial dependence does not involve cognitive abilities. In the current study, we conducted two experiments to investigate whether and how attention-a cognitive ability-affected the serial dependence. The results showed that serial dependence was present in the current study, reproducing the previous findings. Importantly, when the attentional load reduced the reliability (i.e., estimation accuracy and precision) of previous form orientations (Experiment 1), the serial dependence decreased, meaning that the biases of motion direction estimates toward previous form orientations were reduced; in contrast, when the attentional load reduced the reliability of current motion directions (Experiment 2), the serial dependence increased, meaning that the biases of motion direction estimates toward previous form orientations were increased. These trends were well consistent with the prediction of the Bayesian inference theory. Therefore, the current study revealed the involvement of attention in the serial dependence of current motion direction estimation on the previous form orientation, demonstrating that the serial dependence was cognitive and the attentional effect can be a Bayesian inference process, initially revealing its computational mechanism.

Resource-rational account of sequential effects in human prediction

An abundant literature reports on 'sequential effects' observed when humans make predictions on the basis of stochastic sequences of stimuli. Such sequential effects represent departures from an optimal, Bayesian process. A prominent explanation posits that humans are adapted to changing environments, and erroneously assume non-stationarity of the environment, even if the latter is static. As a result, their predictions fluctuate over time. We propose a different explanation in which sub-optimal and fluctuating predictions result from cognitive constraints (or costs), under which humans however behave rationally. We devise a framework of costly inference, in which we develop two classes of models that differ by the nature of the constraints at play: in one case the precision of beliefs comes at a cost, resulting in an exponential forgetting of past observations, while in the other beliefs with high predictive power are favored. To compare model predictions to human behavior, we carry out a prediction task that uses binary random stimuli, with probabilities ranging from 0.05 to 0.95. Although in this task the environment is static and the Bayesian belief converges, subjects' predictions fluctuate and are biased toward the recent stimulus history. Both classes of models capture this 'attractive effect', but they depart in their characterization of higher-order effects. Only the precision-cost model reproduces a 'repulsive effect', observed in the data, in which predictions are biased away from stimuli presented in more distant trials. Our experimental results reveal systematic modulations in sequential effects, which our theoretical approach accounts for in terms of rationality under cognitive constraints.

The influence of natural image statistics on upright orientation judgements

Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.

Effects of spatial frequency cross-adaptation on the visual number sense

When observing a simple visual scene such as an array of dots, observers can easily and automatically extract their number. How does our visual system accomplish this? We investigate the role of specific spatial frequencies to the encoding of number through cross-adaptation. In two experiments, observers were peripherally adapted to six randomly generated sinusoidal gratings varying from relatively low-spatial frequency (M = 0.44 c/deg) to relatively high-spatial frequency (M = 5.88 c/deg). Subsequently, observers judged which side of the screen had a higher number of dots. We found a strong number-adaptation effect to low-spatial frequency gratings (i.e., participants significantly underestimated the number of dots on the adapted side) but a significantly reduced adaptation effect for high-spatial frequency gratings. Various control conditions demonstrate that these effects are not due to a generic response bias for the adapted side, nor moderated by dot size or spacing effects. In a third experiment, we observed no cross-adaptation for centrally presented gratings. Our results show that observers' peripheral number perception can be adapted even with stimuli lacking any numeric or segmented object information and that low spatial frequencies adapt peripheral number perception more than high ones. Together, our results are consistent with recent number perception models that suggest a key role for spatial frequency in the extraction of number from the visual signal (e.g., Paul, Ackooij, Ten Cate, & Harvey, 2022), but additionally suggest that some spatial frequencies - especially in the low range and in the periphery - may be weighted more by the visual system when estimating number. We argue that the cross-adaptation paradigm is also a useful methodology for discovering the primitives of visual number encoding.

Rapid calibration to dynamic temporal contexts

The prediction of future events and the preparation of appropriate behavioural reactions rely on an accurate perception of temporal regularities. In dynamic environments, temporal regularities are subject to slow and sudden changes, and adaptation to these changes is an important requirement for efficient behaviour. Bayesian models have proven a useful tool to understand the processing of temporal regularities in humans; yet an open question pertains to the degree of flexibility of the prior that is required for optimal modelling of behaviour. Here we directly compare dynamic models (with continuously changing prior expectations) and static models (a stable prior for each experimental session) with their ability to describe regression effects in interval timing. Our results show that dynamic Bayesian models are superior when describing the responses to slow, continuous environmental changes, whereas static models are more suitable to describe responses to sudden changes. In time perception research, these results will be informative for the choice of adequate computational models and enhance our understanding of the neuronal computations underlying human timing behaviour.

Motion direction is represented as a bimodal probability distribution in the human visual cortex

Humans infer motion direction from noisy sensory signals. We hypothesize that to make these inferences more precise, the visual system computes motion direction not only from velocity but also spatial orientation signals - a 'streak' created by moving objects. We implement this hypothesis in a Bayesian model, which quantifies knowledge with probability distributions, and test its predictions using psychophysics and fMRI. Using a probabilistic pattern-based analysis, we decode probability distributions of motion direction from trial-by-trial activity in the human visual cortex. Corroborating the predictions, the decoded distributions have a bimodal shape, with peaks that predict the direction and magnitude of behavioral errors. Interestingly, we observe similar bimodality in the distribution of the observers' behavioral responses across trials. Together, these results suggest that observers use spatial orientation signals when estimating motion direction. More broadly, our findings indicate that the cortical representation of low-level visual features, such as motion direction, can reflect a combination of several qualitatively distinct signals.

Learning efficient representations of environmental priors in working memory

Experience shapes our expectations and helps us learn the structure of the environment. Inference models render such learning as a gradual refinement of the observer's estimate of the environmental prior. For instance, when retaining an estimate of an object's features in working memory, learned priors may bias the estimate in the direction of common feature values. Humans display such biases when retaining color estimates on short time intervals. We propose that these systematic biases emerge from modulation of synaptic connectivity in a neural circuit based on the experienced stimulus history, shaping the persistent and collective neural activity that encodes the stimulus estimate. Resulting neural activity attractors are aligned to common stimulus values. Using recently published human response data from a delayed-estimation task in which stimuli (colors) were drawn from a heterogeneous distribution that did not necessarily correspond with reported population biases, we confirm that most subjects' response distributions are better described by experience-dependent learning models than by models with fixed biases. This work suggests systematic limitations in working memory reflect efficient representations of inferred environmental structure, providing new insights into how humans integrate environmental knowledge into their cognitive strategies.

Reference induces biases in late visual processing

How we perceive a visual stimulus can be influenced by its surrounding context. For example, the presence of a reference skews the perception of a similar feature in a stimulus, a phenomenon called reference repulsion. Ongoing research so far remains inconclusive regarding the stage of visual information processing where such repulsion occurs. We examined the influence of a reference on late visual processing. We measured the repulsion effect caused by an orientation reference presented after an orientation ensemble stimulus. The participants' reported orientations were significantly biased away from the post-stimulus reference, displaying typical characteristics of reference repulsion. Moreover, explicit discrimination choices between the reference and the stimulus influenced the magnitudes of repulsion effects, which can be explained by an encoding-decoding model that differentiates the re-weighting of sensory representations in implicit and explicit processes. These results support the notion that reference repulsion may arise at a late decision-related stage of visual processing, where different sensory decoding strategies are employed depending on the specific task.

Neural tuning instantiates prior expectations in the human visual system

Perception is often modelled as a process of active inference, whereby prior expectations are combined with noisy sensory measurements to estimate the structure of the world. This mathematical framework has proven critical to understanding perception, cognition, motor control, and social interaction. While theoretical work has shown how priors can be computed from environmental statistics, their neural instantiation could be realised through multiple competing encoding schemes. Using a data-driven approach, here we extract the brain's representation of visual orientation and compare this with simulations from different sensory coding schemes. We found that the tuning of the human visual system is highly conditional on stimulus-specific variations in a way that is not predicted by previous proposals. We further show that the adopted encoding scheme effectively embeds an environmental prior for natural image statistics within the sensory measurement, providing the functional architecture necessary for optimal inference in the earliest stages of cortical processing.

Individual risk attitudes arise from noise in neurocognitive magnitude representations

Humans are generally risk averse, preferring smaller certain over larger uncertain outcomes. Economic theories usually explain this by assuming concave utility functions. Here, we provide evidence that risk aversion can also arise from relative underestimation of larger monetary payoffs, a perceptual bias rooted in the noisy logarithmic coding of numerical magnitudes. We confirmed this with psychophysics and functional magnetic resonance imaging, by measuring behavioural and neural acuity of magnitude representations during a magnitude perception task and relating these measures to risk attitudes during separate risky financial decisions. Computational modelling indicated that participants use similar mental magnitude representations in both tasks, with correlated precision across perceptual and risky choices. Participants with more precise magnitude representations in parietal cortex showed less variable behaviour and less risk aversion. Our results highlight that at least some individual characteristics of economic behaviour can reflect capacity limitations in perceptual processing rather than processes that assign subjective values to monetary outcomes.

Dynamics of cortical contrast adaptation predict perception of signals in noise

Neurons throughout the sensory pathway adapt their responses depending on the statistical structure of the sensory environment. Contrast gain control is a form of adaptation in the auditory cortex, but it is unclear whether the dynamics of gain control reflect efficient adaptation, and whether they shape behavioral perception. Here, we trained mice to detect a target presented in background noise shortly after a change in the contrast of the background. The observed changes in cortical gain and behavioral detection followed the dynamics of a normative model of efficient contrast gain control; specifically, target detection and sensitivity improved slowly in low contrast, but degraded rapidly in high contrast. Auditory cortex was required for this task, and cortical responses were not only similarly affected by contrast but predicted variability in behavioral performance. Combined, our results demonstrate that dynamic gain adaptation supports efficient coding in auditory cortex and predicts the perception of sounds in noise.

Tasks and their role in visual neuroscience

Vision is widely used as a model system to gain insights into how sensory inputs are processed and interpreted by the brain. Historically, careful quantification and control of visual stimuli have served as the backbone of visual neuroscience. There has been less emphasis, however, on how an observer's task influences the processing of sensory inputs. Motivated by diverse observations of task-dependent activity in the visual system, we propose a framework for thinking about tasks, their role in sensory processing, and how we might formally incorporate tasks into our models of vision.

Interhemispheric multisensory perception and Bayesian causal inference

In daily life, our brain needs to eliminate irrelevant signals and integrate relevant signals to facilitate natural interactions with the surrounding. Previous study focused on paradigms without effect of dominant laterality and found that human observers process multisensory signals consistent with Bayesian causal inference (BCI). However, most human activities are of bilateral interaction involved in processing of interhemispheric sensory signals. It remains unclear whether the BCI framework also fits to such activities. Here, we presented a bilateral hand-matching task to understand the causal structure of interhemispheric sensory signals. In this task, participants were asked to match ipsilateral visual or proprioceptive cues with the contralateral hand. Our results suggest that interhemispheric causal inference is most derived from the BCI framework. The interhemispheric perceptual bias may vary strategy models to estimate the contralateral multisensory signals. The findings help to understand how the brain processes the uncertainty information coming from interhemispheric sensory signals.

A gradual transition from veridical to categorical representations along the visual hierarchy during working memory, but not perception

The ability to stably maintain visual information over brief delays is central to cognitive functioning. One possible way to achieve robust working memory maintenance is by having multiple concurrent mnemonic representations across multiple cortical loci. For example, early visual cortex might contribute to storage by representing information in a "sensory-like" format, while intraparietal sulcus uses a format transformed away from sensory driven responses. As an explicit test of mnemonic code transformations along the visual hierarchy, we quantitatively modeled the progression of veridical-to-categorical orientation representations in human participants. Participants directly viewed, or held in mind, an oriented grating pattern, and the similarity between fMRI activation patterns for different orientations was calculated throughout retinotopic cortex. During direct perception, similarity was clustered around cardinal orientations, while during working memory the obliques were represented more similarly. We modeled these similarity patterns based on the known distribution of orientation information in the natural world: The "veridical" model uses an efficient coding framework to capture hypothesized representations during visual perception. The "categorical" model assumes that different "psychological distances" between orientations result in orientation categorization relative to cardinal axes. During direct perception, the veridical model explained the data well in early visual areas, while the categorical model did worse. During working memory, the veridical model only explained some of the data, while the categorical model gradually gained explanatory power for increasingly anterior retinotopic regions. These findings suggest that directly viewed images are represented veridically, but once visual information is no longer tethered to the sensory world, there is a gradual progression to more categorical mnemonic formats along the visual hierarchy.

Sensory perception relies on fitness-maximizing codes

Sensory information encoded by humans and other organisms is generally presumed to be as accurate as their biological limitations allow. However, perhaps counterintuitively, accurate sensory representations may not necessarily maximize the organism's chances of survival. To test this hypothesis, we developed a unified normative framework for fitness-maximizing encoding by combining theoretical insights from neuroscience, computer science, and economics. Behavioural experiments in humans revealed that sensory encoding strategies are flexibly adapted to promote fitness maximization, a result confirmed by deep neural networks with information capacity constraints trained to solve the same task as humans. Moreover, human functional MRI data revealed that novel behavioural goals that rely on object perception induce efficient stimulus representations in early sensory structures. These results suggest that fitness-maximizing rules imposed by the environment are applied at early stages of sensory processing in humans and machines.

Natural statistics of human head orientation constrain models of vestibular processing

Head orientation relative to gravity determines how gravity-dependent environmental structure is sampled by the visual system, as well as how gravity itself is sampled by the vestibular system. Therefore, both visual and vestibular sensory processing should be shaped by the statistics of head orientation relative to gravity. Here we report the statistics of human head orientation during unconstrained natural activities in humans for the first time, and we explore implications for models of vestibular processing. We find that the distribution of head pitch is more variable than head roll and that the head pitch distribution is asymmetrical with an over-representation of downward head pitch, consistent with ground-looking behavior. We further suggest that pitch and roll distributions can be used as empirical priors in a Bayesian framework to explain previously measured biases in perception of both roll and pitch. Gravitational and inertial acceleration stimulate the otoliths in an equivalent manner, so we also analyze the dynamics of human head orientation to better understand how knowledge of these dynamics can constrain solutions to the problem of gravitoinertial ambiguity. Gravitational acceleration dominates at low frequencies and inertial acceleration dominates at higher frequencies. The change in relative power of gravitational and inertial components as a function of frequency places empirical constraints on dynamic models of vestibular processing, including both frequency segregation and probabilistic internal model accounts. We conclude with a discussion of methodological considerations and scientific and applied domains that will benefit from continued measurement and analysis of natural head movements moving forward.

Adaptive tuning of human learning and choice variability to unexpected uncertainty

Human value-based decisions are notably variable under uncertainty. This variability is known to arise from two distinct sources: variable choices aimed at exploring available options and imprecise learning of option values due to limited cognitive resources. However, whether these two sources of decision variability are tuned to their specific costs and benefits remains unclear. To address this question, we compared the effects of expected and unexpected uncertainty on decision-making in the same reinforcement learning task. Across two large behavioral datasets, we found that humans choose more variably between options but simultaneously learn less imprecisely their values in response to unexpected uncertainty. Using simulations of learning agents, we demonstrate that these opposite adjustments reflect adaptive tuning of exploration and learning precision to the structure of uncertainty. Together, these findings indicate that humans regulate not only how much they explore uncertain options but also how precisely they learn the values of these options.

Efficient sensory encoding predicts robust averaging

Not every item in a stimulus ensemble equally contributes to the perceived ensemble average. Rather, items with feature values close to the ensemble mean (inlying items) contribute stronger compared to those items whose feature values are further away from the mean (outlying items). This nonuniform weighting process, named robust averaging, has been interpreted as evidence against an optimal integration of sensory information. Here, however, we show that robust averaging naturally emerges from an optimal integration process when sensory encoding is efficiently adapted to the ensemble statistics in the experiment. We demonstrate that such a model can accurately fit several existing datasets showing robust perceptual averaging in discriminating low-level stimulus features such as orientation. Across various feature domains, our model accurately predicts subjects' decision accuracy and nonuniform weighting profile, and both their dependency on the specific stimulus distribution in the experiments. Our results suggest that the human visual system forms efficient sensory representations on short time-scales to improve overall decision performance.

10 years of Bayesian theories of autism: A comprehensive review

Ten years ago, Pellicano and Burr published one of the most influential articles in the study of autism spectrum disorders, linking them to aberrant Bayesian inference processes in the brain. In particular, they proposed that autistic individuals are less influenced by their brains' prior beliefs about the environment. In this systematic review, we investigate if this theory is supported by the experimental evidence. To that end, we collect all studies which included comparisons across diagnostic groups or autistic traits and categorise them based on the investigated priors. Our results are highly mixed, with a slight majority of studies finding no difference in the integration of Bayesian priors. We find that priors developed during the experiments exhibited reduced influences more frequently than priors acquired previously, with various studies providing evidence for learning differences between participant groups. Finally, we focus on the methodological and computational aspects of the included studies, showing low statistical power and often inconsistent approaches. Based on our findings, we propose guidelines for future research.

Perceptual comparisons modulate memory biases induced by new visual inputs

It is well-established that stimulus-specific information in visual working memory (VWM) can be systematically biased by new perceptual inputs. These memory biases are commonly attributed to interference that arises when perceptual inputs are physically similar to VWM contents. However, recent work has suggested that explicitly comparing the similarity between VWM contents and new perceptual inputs modulates the size of memory biases above and beyond stimulus-driven effects. Here, we sought to directly investigate this modulation hypothesis by comparing the size of memory biases following explicit comparisons to those induced when new perceptual inputs are ignored (Experiment 1) or maintained in VWM alongside target information (Experiment 2). We found that VWM reports showed larger attraction biases following explicit perceptual comparisons than when new perceptual inputs were ignored or maintained in VWM. An analysis of participants' perceptual comparisons revealed that memory biases were amplified after perceptual inputs were endorsed as similar-but not dissimilar-to one's VWM representation. These patterns were found to persist even after accounting for variability in the physical similarity between the target and perceptual stimuli across trials, as well as the baseline memory precision between the distinct task demands. Together, these findings illustrate a causal role of perceptual comparisons in modulating naturally-occurring memory biases.

Levels of orientation bias differ across digital content categories: Implications for visual perception

With the continued growth of digital device use, a greater portion of the visual world experienced daily by many people has shifted towards digital environments. The "oblique effect" denotes a bias for horizontal and vertical (canonical) contours over oblique contours, which is derived from a disproportionate exposure to canonical content. Carpentered environments have been shown to possess proportionally more canonical than oblique contours, leading to perceptual bias in those who live in "built" environments. Likewise, there is potential for orientation sensitivity to be shaped by frequent exposure to digital content. The potential influence of digital content on the oblique effect was investigated by measuring the degree of orientation anisotropy from a range of digital scenes using Fourier analysis. Content from popular cartoons, video games, and social communication websites was compared to real-life nature, suburban, and urban scenes. Findings suggest that digital content varies widely in orientation anisotropy, but pixelated video games and social communication websites were found to exhibit a degree of orientation anisotropy substantially exceeding that observed in all measured categories of real-world environments. Therefore, the potential may exist for digital content to induce an even greater shift in orientation bias than has been observed in previous research. This potential, and implications of such a shift, is discussed.

A General Framework for Inferring Bayesian Ideal Observer Models from Psychophysical Data

A central question in neuroscience is how sensory inputs are transformed into percepts. At this point, it is clear that this process is strongly influenced by prior knowledge of the sensory environment. Bayesian ideal observer models provide a useful link between data and theory that can help researchers evaluate how prior knowledge is represented and integrated with incoming sensory information. However, the statistical prior employed by a Bayesian observer cannot be measured directly, and must instead be inferred from behavioral measurements. Here, we review the general problem of inferring priors from psychophysical data, and the simple solution that follows from assuming a prior that is a Gaussian probability distribution. As our understanding of sensory processing advances, however, there is an increasing need for methods to flexibly recover the shape of Bayesian priors that are not well approximated by elementary functions. To address this issue, we describe a novel approach that applies to arbitrary prior shapes, which we parameterize using mixtures of Gaussian distributions. After incorporating a simple approximation, this method produces an analytical solution for psychophysical quantities that can be numerically optimized to recover the shapes of Bayesian priors. This approach offers advantages in flexibility, while still providing an analytical framework for many scenarios. We provide a MATLAB toolbox implementing key computations described herein.

Efficient neural codes naturally emerge through gradient descent learning

Human sensory systems are more sensitive to common features in the environment than uncommon features. For example, small deviations from the more frequently encountered horizontal orientations can be more easily detected than small deviations from the less frequent diagonal ones. Here we find that artificial neural networks trained to recognize objects also have patterns of sensitivity that match the statistics of features in images. To interpret these findings, we show mathematically that learning with gradient descent in neural networks preferentially creates representations that are more sensitive to common features, a hallmark of efficient coding. This effect occurs in systems with otherwise unconstrained coding resources, and additionally when learning towards both supervised and unsupervised objectives. This result demonstrates that efficient codes can naturally emerge from gradient-like learning.

Efficient coding theory of dynamic attentional modulation

Activity of sensory neurons is driven not only by external stimuli but also by feedback signals from higher brain areas. Attention is one particularly important internal signal whose presumed role is to modulate sensory representations such that they only encode information currently relevant to the organism at minimal cost. This hypothesis has, however, not yet been expressed in a normative computational framework. Here, by building on normative principles of probabilistic inference and efficient coding, we developed a model of dynamic population coding in the visual cortex. By continuously adapting the sensory code to changing demands of the perceptual observer, an attention-like modulation emerges. This modulation can dramatically reduce the amount of neural activity without deteriorating the accuracy of task-specific inferences. Our results suggest that a range of seemingly disparate cortical phenomena such as intrinsic gain modulation, attention-related tuning modulation, and response variability could be manifestations of the same underlying principles, which combine efficient sensory coding with optimal probabilistic inference in dynamic environments.

Perceived speed at low luminance: Lights out for the Bayesian observer?

To account for perceptual bias, Bayesian models use the precision of early sensory measurements to weight the influence of prior expectations. As precision decreases, prior expectations start to dominate. Important examples come from motion perception, where the slow-motion prior has been used to explain a variety of motion illusions in vision, hearing, and touch, many of which correlate appropriately with threshold measures of underlying precision. However, the Bayesian account seems defeated by the finding that moving objects appear faster in the dark, because most motion thresholds are worse at low luminance. Here we show this is not the case for speed discrimination. Our results show that performance improves at low light levels by virtue of a perceived contrast cue that is more salient in the dark. With this cue removed, discrimination becomes independent of luminance. However, we found perceived speed still increased in the dark for the same observers, and by the same amount. A possible interpretation is that motion processing is therefore not Bayesian, because our findings challenge a key assumption these models make, namely that the accuracy of early sensory measurements is independent of basic stimulus properties like luminance. However, a final experiment restored Bayesian behaviour by adding external noise, making discrimination worse and slowing perceived speed down. Our findings therefore suggest that motion is processed in a Bayesian fashion but based on noisy sensory measurements that also vary in accuracy.

Calibrating Vision: Concepts and Questions

The idea that visual coding and perception are shaped by experience and adjust to changes in the environment or the observer is universally recognized as a cornerstone of visual processing, yet the functions and processes mediating these calibrations remain in many ways poorly understood. In this article we review a number of facets and issues surrounding the general notion of calibration, with a focus on plasticity within the encoding and representational stages of visual processing. These include how many types of calibrations there are - and how we decide; how plasticity for encoding is intertwined with other principles of sensory coding; how it is instantiated at the level of the dynamic networks mediating vision; how it varies with development or between individuals; and the factors that may limit the form or degree of the adjustments. Our goal is to give a small glimpse of an enormous and fundamental dimension of vision, and to point to some of the unresolved questions in our understanding of how and why ongoing calibrations are a pervasive and essential element of vision.

Attractive and repulsive effects of sensory history concurrently shape visual perception

Background

Sequential effects of environmental stimuli are ubiquitous in most behavioral tasks involving magnitude estimation, memory, decision making, and emotion. The human visual system exploits continuity in the visual environment, which induces two contrasting perceptual phenomena shaping visual perception. Previous work reported that perceptual estimation of a stimulus may be influenced either by attractive serial dependencies or repulsive aftereffects, with a number of experimental variables suggested as factors determining the direction and magnitude of sequential effects. Recent studies have theorized that these two effects concurrently arise in perceptual processing, but empirical evidence that directly supports this hypothesis is lacking, and it remains unclear whether and how attractive and repulsive sequential effects interact in a trial. Here we show that the two effects concurrently modulate estimation behavior in a typical sequence of perceptual tasks.

Results

We first demonstrate that observers’ estimation error as a function of both the previous stimulus and response cannot be fully described by either attractive or repulsive bias but is instead well captured by a summation of repulsion from the previous stimulus and attraction toward the previous response. We then reveal that the repulsive bias is centered on the observer’s sensory encoding of the previous stimulus, which is again repelled away from its own preceding trial, whereas the attractive bias is centered precisely on the previous response, which is the observer’s best prediction about the incoming stimuli.

Conclusions

Our findings provide strong evidence that sensory encoding is shaped by dynamic tuning of the system to the past stimuli, inducing repulsive aftereffects, and followed by inference incorporating the prediction from the past estimation, leading to attractive serial dependence.