A Bayesian observer model constrained by efficient coding can explain 'anti-Bayesian' percepts

Attractive and repulsive effects of sensory history concurrently shape visual perception

BACKGROUND

Sequential effects of environmental stimuli are ubiquitous in most behavioral tasks involving magnitude estimation, memory, decision making, and emotion. The human visual system exploits continuity in the visual environment, which induces two contrasting perceptual phenomena shaping visual perception. Previous work reported that perceptual estimation of a stimulus may be influenced either by attractive serial dependencies or repulsive aftereffects, with a number of experimental variables suggested as factors determining the direction and magnitude of sequential effects. Recent studies have theorized that these two effects concurrently arise in perceptual processing, but empirical evidence that directly supports this hypothesis is lacking, and it remains unclear whether and how attractive and repulsive sequential effects interact in a trial. Here we show that the two effects concurrently modulate estimation behavior in a typical sequence of perceptual tasks.

RESULTS

We first demonstrate that observers' estimation error as a function of both the previous stimulus and response cannot be fully described by either attractive or repulsive bias but is instead well captured by a summation of repulsion from the previous stimulus and attraction toward the previous response. We then reveal that the repulsive bias is centered on the observer's sensory encoding of the previous stimulus, which is again repelled away from its own preceding trial, whereas the attractive bias is centered precisely on the previous response, which is the observer's best prediction about the incoming stimuli.

CONCLUSIONS

Our findings provide strong evidence that sensory encoding is shaped by dynamic tuning of the system to the past stimuli, inducing repulsive aftereffects, and followed by inference incorporating the prediction from the past estimation, leading to attractive serial dependence.

How much I moved: Robust biases in self-rotation perception

Vestibular cues are crucial to sense the linear and angular acceleration of our head in three-dimensional space. Previous literature showed that vestibular information precociously combines with other sensory modalities, such as proprioceptive and visual, to facilitate spatial navigation. Recent studies suggest that auditory cues may improve self-motion perception as well. The present study investigated the ability to estimate passive rotational displacements with and without virtual acoustic landmarks to determine how vestibular and auditory information interact in processing self-motion information. We performed two experiments. In both, healthy participants sat on a Rotational-Translational Chair. They experienced yaw rotations along the earth-vertical axis and performed a self-motion discrimination task. Their goal was to estimate both clockwise and counterclockwise rotations’ amplitude, with no visual information available, reporting whether they felt to be rotated more or less than 45°. According to the condition, vestibular-only or audio-vestibular information was present. Between the two experiments, we manipulated the procedure of presentation of the auditory cues (passive vs. active production of sounds). We computed the point of subjective equality (PSE) as a measure of accuracy and the just noticeable difference (JND) as the precision of the estimations for each condition and direction of rotations. Results in both experiments show a strong overestimation bias of the rotations, regardless of the condition, the direction, and the sound generation conditions. Similar to previously found heading biases, this bias in rotation estimation may facilitate the perception of substantial deviations from the most relevant directions in daily navigation activities.

Rule-based and stimulus-based cues bias auditory decisions via different computational and physiological mechanisms

Expectations, such as those arising from either learned rules or recent stimulus regularities, can bias subsequent auditory perception in diverse ways. However, it is not well understood if and how these diverse effects depend on the source of the expectations. Further, it is unknown whether different sources of bias use the same or different computational and physiological mechanisms. We examined how rule-based and stimulus-based expectations influenced behavior and pupil-linked arousal, a marker of certain forms of expectation-based processing, of human subjects performing an auditory frequency-discrimination task. Rule-based cues consistently biased choices and response times (RTs) toward the more-probable stimulus. In contrast, stimulus-based cues had a complex combination of effects, including choice and RT biases toward and away from the frequency of recently presented stimuli. These different behavioral patterns also had: 1) distinct computational signatures, including different modulations of key components of a novel form of a drift-diffusion decision model and 2) distinct physiological signatures, including substantial bias-dependent modulations of pupil size in response to rule-based but not stimulus-based cues. These results imply that different sources of expectations can modulate auditory processing via distinct mechanisms: one that uses arousal-linked, rule-based information and another that uses arousal-independent, stimulus-based information to bias the speed and accuracy of auditory perceptual decisions.

Prior information about upcoming stimuli can bias our perception of those stimuli. Whether different sources of prior information bias perception in similar or distinct ways is not well understood. We compared the influence of two kinds of prior information on tone-frequency discrimination: rule-based cues, in the form of explicit information about the most-likely identity of the upcoming tone; and stimulus-based cues, in the form of sequences of tones presented before the to-be-discriminated tone. Although both types of prior information biased auditory decision-making, they demonstrated distinct behavioral, computational, and physiological signatures. Our results suggest that the brain processes prior information in a form-specific manner rather than utilizing a general-purpose prior. Such form-specific processing has implications for understanding decision biases real-world contexts, in which prior information comes from many different sources and modalities.

Divisive normalization is an efficient code for multivariate Pareto-distributed environments

Divisive normalization is a canonical computation in the brain, observed across neural systems, that is often considered to be an implementation of the efficient coding principle. We provide a theoretical result that makes the conditions under which divisive normalization is an efficient code analytically precise: We show that, in a low-noise regime, encoding an n-dimensional stimulus via divisive normalization is efficient if and only if its prevalence in the environment is described by a multivariate Pareto distribution. We generalize this multivariate analog of histogram equalization to allow for arbitrary metabolic costs of the representation, and show how different assumptions on costs are associated with different shapes of the distributions that divisive normalization efficiently encodes. Our result suggests that divisive normalization may have evolved to efficiently represent stimuli with Pareto distributions. We demonstrate that this efficiently encoded distribution is consistent with stylized features of naturalistic stimulus distributions such as their characteristic conditional variance dependence, and we provide empirical evidence suggesting that it may capture the statistics of filter responses to naturalistic images. Our theoretical finding also yields empirically testable predictions across sensory domains on how the divisive normalization parameters should be tuned to features of the input distribution.

Attractive serial dependence overcomes repulsive neuronal adaptation

Sensory responses and behavior are strongly shaped by stimulus history. For example, perceptual reports are sometimes biased toward previously viewed stimuli (serial dependence). While behavioral studies have pointed to both perceptual and postperceptual origins of this phenomenon, neural data that could elucidate where these biases emerge is limited. We recorded functional magnetic resonance imaging (fMRI) responses while human participants (male and female) performed a delayed orientation discrimination task. While behavioral reports were attracted to the previous stimulus, response patterns in visual cortex were repelled. We reconciled these opposing neural and behavioral biases using a model where both sensory encoding and readout are shaped by stimulus history. First, neural adaptation reduces redundancy at encoding and leads to the repulsive biases that we observed in visual cortex. Second, our modeling work suggest that serial dependence is induced by readout mechanisms that account for adaptation in visual cortex. According to this account, the visual system can simultaneously improve efficiency via adaptation while still optimizing behavior based on the temporal structure of natural stimuli.

Expansion and contraction of resource allocation in sensory bottlenecks

Topographic sensory representations often do not scale proportionally to the size of their input regions, with some expanded and others contracted. In vision, the foveal representation is magnified cortically, as are the fingertips in touch. What principles drive this allocation, and how should receptor density, for example, the high innervation of the fovea or the fingertips, and stimulus statistics, for example, the higher contact frequencies on the fingertips, contribute? Building on work in efficient coding, we address this problem using linear models that optimally decorrelate the sensory signals. We introduce a sensory bottleneck to impose constraints on resource allocation and derive the optimal neural allocation. We find that bottleneck width is a crucial factor in resource allocation, inducing either expansion or contraction. Both receptor density and stimulus statistics affect allocation and jointly determine convergence for wider bottlenecks. Furthermore, we show a close match between the predicted and empirical cortical allocations in a well-studied model system, the star-nosed mole. Overall, our results suggest that the strength of cortical magnification depends on resource limits.

A reference-based theory of motivation and effort allocation

Motivation is key for performance in domains such as work, sport, and learning. Research has established that motivation and the willingness to invest effort generally increase as a function of reward. However, this view struggles to explain some empirical observations-for example, in the domain of sport, athletes sometimes appear to lose motivation when playing against weak opponents-this despite objective rewards being high. This and similar evidence highlight the role of subjective value in motivation and effort allocation. To capture this, here, we advance a novel theory and computational model where motivation and effort allocation arise from reference-based evaluation processes. Our proposal argues that motivation (and the ensuing willingness to exert effort) stems from subjective value, which in turns depends on one's standards about performance and on the confidence about these standards. In a series of simulations, we show that the model explains puzzling motivational dynamics and associated feelings. Crucially, the model identifies realistic standards (i.e., those matching one's own actual ability) as those more beneficial for motivation and performance. On this basis, the model establishes a normative solution to the problem of optimal allocation of effort, analogous to the optimal allocation of neural and computational resources as in efficient coding.

Prior Expectations in Visual Speed Perception Predict Encoding Characteristics of Neurons in Area MT

Bayesian inference provides an elegant theoretical framework for understanding the characteristic biases and discrimination thresholds in visual speed perception. However, the framework is difficult to validate because of its flexibility and the fact that suitable constraints on the structure of the sensory uncertainty have been missing. Here, we demonstrate that a Bayesian observer model constrained by efficient coding not only well explains human visual speed perception but also provides an accurate quantitative account of the tuning characteristics of neurons known for representing visual speed. Specifically, we found that the population coding accuracy for visual speed in area MT ("neural prior") is precisely predicted by the power-law, slow-speed prior extracted from fitting the Bayesian observer model to psychophysical data ("behavioral prior") to the point that the two priors are indistinguishable in a cross-validation model comparison. Our results demonstrate a quantitative validation of the Bayesian observer model constrained by efficient coding at both the behavioral and neural levels.SIGNIFICANCE STATEMENT Statistical regularities of the environment play an important role in shaping both neural representations and perceptual behavior. Most previous work addressed these two aspects independently. Here we present a quantitative validation of a theoretical framework that makes joint predictions for neural coding and behavior, based on the assumption that neural representations of sensory information are efficient but also optimally used in generating a percept. Specifically, we demonstrate that the neural tuning characteristics for visual speed in brain area MT are precisely predicted by the statistical prior expectations extracted from psychophysical data. As such, our results provide a normative link between perceptual behavior and the neural representation of sensory information in the brain.

Emergence of probabilistic representation in the neural network of primary visual cortex

During the early development of the mammalian visual system, the distribution of neuronal preferred orientations in the primary visual cortex (V1) gradually shifts to match major orientation features of the environment, achieving its optimal representation. By combining computational modeling and electrophysiological recording, we provide a circuit plasticity mechanism that underlies the developmental emergence of such matched representation in the visual cortical network. Specifically, in a canonical circuit of densely-interconnected pyramidal cells and inhibitory parvalbumin-expressing (PV+) fast-spiking interneurons in V1 layer 2/3, our model successfully simulates the experimental observations and further reveals that the nonuniform inhibition plays a key role in shaping the network representation through spike timing-dependent plasticity. The experimental results suggest that PV + interneurons in V1 are capable of providing nonuniform inhibition shortly after vision onset. Our study elucidates a circuit mechanism for acquisition of prior knowledge of environment for optimal inference in sensory neural systems

•

Computational and experimental methods are combined to representation in mice V1

•

Nonuniform inhibition plays a key role in shaping the network representation

•

PV + interneurons provide nonuniform inhibition shortly after vision onset

Luminance and Contrast of Images in the THINGS Database

The THINGS database is a freely available stimulus set that has the potential to facilitate the generation of theory that bridges multiple areas within cognitive neuroscience. The database consists of 26,107 high quality digital photos that are sorted into 1,854 concepts. While a valuable resource, relatively few technical details relevant to the design of studies in cognitive neuroscience have been described. We present an analysis of two key low-level properties of THINGS images, luminance and luminance contrast. These image statistics are known to influence common physiological and neural correlates of perceptual and cognitive processes. In general, we found that the distributions of luminance and contrast are in close agreement with the statistics of natural images reported previously. However, we found that image concepts are separable in their luminance and contrast: we show that luminance and contrast alone are sufficient to classify images into their concepts with above chance accuracy. We describe how these factors may confound studies using the THINGS images, and suggest simple controls that can be implemented a priori or post-hoc. We discuss the importance of using such natural images as stimuli in psychological research.

Automatic compensation enhances the orientation perception in chronic astigmatism

Astigmatism is a prevalent optical problem in which two or more focal points blur the retinal image at a particular meridian. Although many features of astigmatic vision, including orientation perception, are impaired at the retinal image level, the visual system appears to partly restore perceptual impairment after an extended period of astigmatism. However, the mechanism of orientation perception restoration in chronic astigmatism has not yet been clarified. We investigated the notable reduction of perceptual error in chronic astigmatism by comparing the orientation perception of a chronic astigmatism group with the perception of a normal-vision group, in which astigmatism was transiently induced. We found that orientation perception in the chronic group was more accurate than in the normal vision group. Interestingly, the reduction of perceptual errors was automatic; it remained even after the optical refractive errors were fully corrected, and the orientation perception was much more stable across different orientations, despite the uneven noise levels of the retinal images across meridians. We provide here a mechanistic explanation for how the compensation of astigmatic orientation perception occurred, using neural adaptation to the biased distribution of orientations.

Task-induced neural covariability as a signature of approximate Bayesian learning and inference

Perception is often characterized computationally as an inference process in which uncertain or ambiguous sensory inputs are combined with prior expectations. Although behavioral studies have shown that observers can change their prior expectations in the context of a task, robust neural signatures of task-specific priors have been elusive. Here, we analytically derive such signatures under the general assumption that the responses of sensory neurons encode posterior beliefs that combine sensory inputs with task-specific expectations. Specifically, we derive predictions for the task-dependence of correlated neural variability and decision-related signals in sensory neurons. The qualitative aspects of our results are parameter-free and specific to the statistics of each task. The predictions for correlated variability also differ from predictions of classic feedforward models of sensory processing and are therefore a strong test of theories of hierarchical Bayesian inference in the brain. Importantly, we find that Bayesian learning predicts an increase in so-called “differential correlations” as the observer’s internal model learns the stimulus distribution, and the observer’s behavioral performance improves. This stands in contrast to classic feedforward encoding/decoding models of sensory processing, since such correlations are fundamentally information-limiting. We find support for our predictions in data from existing neurophysiological studies across a variety of tasks and brain areas. Finally, we show in simulation how measurements of sensory neural responses can reveal information about a subject’s internal beliefs about the task. Taken together, our results reinterpret task-dependent sources of neural covariability as signatures of Bayesian inference and provide new insights into their cause and their function.

Perceptual decision-making has classically been studied in the context of feedforward encoding/ decoding models. Here, we derive predictions for the responses of sensory neurons under the assumption that the brain performs hierarchical Bayesian inference, including feedback signals that communicate task-specific prior expectations. Interestingly, those predictions stand in contrast to some of the conclusions drawn in the classic framework. In particular, we find that Bayesian learning predicts the increase of a type of correlated variability called “differential correlations” over the course of learning. Differential correlations limit information, and hence are seen as harmful in feedforward models. Since our results are also specific to the statistics of a given task, and since they hold under a wide class of theories about how Bayesian probabilities may be represented by neural responses, they constitute a strong test of the Bayesian Brain hypothesis. Our results can explain the task-dependence of correlated variability in prior studies and suggest a reason why these kinds of correlations are surprisingly common in empirical data. Interpreted in a probabilistic framework, correlated variability provides a window into an observer’s task-related beliefs.

Rational inattention in mice

Behavior exhibited by humans and other organisms is generally inconsistent and biased and, thus, is often labeled irrational. However, the origins of this seemingly suboptimal behavior remain elusive. We developed a behavioral task and normative framework to reveal how organisms should allocate their limited processing resources such that sensory precision and its related metabolic investment are balanced to guarantee maximal utility. We found that mice act as rational inattentive agents by adaptively allocating their sensory resources in a way that maximizes reward consumption in previously unexperienced stimulus-reward association environments. Unexpectedly, perception of commonly occurring stimuli was relatively imprecise; however, this apparent statistical fallacy implies "awareness" and efficient adaptation to their neurocognitive limitations. Arousal systems carry reward distribution information of sensory signals, and distributional reinforcement learning mechanisms regulate sensory precision via top-down normalization. These findings reveal how organisms efficiently perceive and adapt to previously unexperienced environmental contexts within the constraints imposed by neurobiology.

More Prominent Nonlinear Mixed Selectivity in the Dorsolateral Prefrontal than Posterior Parietal Cortex

Neurons in the dorsolateral prefrontal cortex (dlPFC) and posterior parietal cortex (PPC) are activated by different cognitive tasks and respond differently to the same stimuli depending on task. The conjunctive representations of multiple tasks in nonlinear fashion in single neuron activity, is known as nonlinear mixed selectivity (NMS). Here, we compared NMS in a working memory task in areas 8a and 46 of the dlPFC and 7a and lateral intraparietal cortex (LIP) of the PPC in macaque monkeys. NMS neurons were more frequent in dlPFC than in PPC and this was attributed to more cells gaining selectivity in the course of a trial. Additionally, in our task, the subjects' behavioral performance improved within a behavioral session as they learned the session-specific statistics of the task. The magnitude of NMS in the dlPFC also increased as a function of time within a single session. On the other hand, we observed minimal rotation of population responses and no appreciable differences in NMS between correct and error trials in either area. Our results provide direct evidence demonstrating a specialization in NMS between dlPFC and PPC and reveal mechanisms of neural selectivity in areas recruited in working memory tasks.

Efficient coding in the economics of human brain connectomics

In systems neuroscience, most models posit that brain regions communicate information under constraints of efficiency. Yet, evidence for efficient communication in structural brain networks characterized by hierarchical organization and highly connected hubs remains sparse. The principle of efficient coding proposes that the brain transmits maximal information in a metabolically economical or compressed form to improve future behavior. To determine how structural connectivity supports efficient coding, we develop a theory specifying minimum rates of message transmission between brain regions to achieve an expected fidelity, and we test five predictions from the theory based on random walk communication dynamics. In doing so, we introduce the metric of compression efficiency, which quantifies the trade-off between lossy compression and transmission fidelity in structural networks. In a large sample of youth (n = 1,042; age 8-23 years), we analyze structural networks derived from diffusion-weighted imaging and metabolic expenditure operationalized using cerebral blood flow. We show that structural networks strike compression efficiency trade-offs consistent with theoretical predictions. We find that compression efficiency prioritizes fidelity with development, heightens when metabolic resources and myelination guide communication, explains advantages of hierarchical organization, links higher input fidelity to disproportionate areal expansion, and shows that hubs integrate information by lossy compression. Lastly, compression efficiency is predictive of behavior-beyond the conventional network efficiency metric-for cognitive domains including executive function, memory, complex reasoning, and social cognition. Our findings elucidate how macroscale connectivity supports efficient coding and serve to foreground communication processes that utilize random walk dynamics constrained by network connectivity.

Subjective confidence reflects representation of Bayesian probability in cortex

What gives rise to the human sense of confidence? Here, we tested the Bayesian hypothesis that confidence is based on a probability distribution represented in neural population activity. We implemented several computational models of confidence, and tested their predictions using psychophysics and fMRI. Using a generative model-based fMRI decoding approach, we extracted probability distributions from neural population activity in human visual cortex. We found that subjective confidence tracks the shape of the decoded distribution. That is, when sensory evidence was more precise, as indicated by the decoded distribution, observers reported higher levels of confidence. We furthermore found that neural activity in the insula, anterior cingulate, and prefrontal cortex was linked to both the shape of the decoded distribution and reported confidence, in ways consistent with the Bayesian model. Altogether, our findings support recent statistical theories of confidence and suggest that probabilistic information guides the computation of one’s sense of confidence.

An image reconstruction framework for characterizing initial visual encoding

We developed an image-computable observer model of the initial visual encoding that operates on natural image input, based on the framework of Bayesian image reconstruction from the excitations of the retinal cone mosaic. Our model extends previous work on ideal observer analysis and evaluation of performance beyond psychophysical discrimination, takes into account the statistical regularities of the visual environment, and provides a unifying framework for answering a wide range of questions regarding the visual front end. Using the error in the reconstructions as a metric, we analyzed variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method is widely applicable to experiments and applications in which the initial visual encoding plays an important role.

Cognitive, Systems, and Computational Neurosciences of the Self in Motion

Navigating by path integration requires continuously estimating one's self-motion. This estimate may be derived from visual velocity and/or vestibular acceleration signals. Importantly, these senses in isolation are ill-equipped to provide accurate estimates, and thus visuo-vestibular integration is an imperative. After a summary of the visual and vestibular pathways involved, the crux of this review focuses on the human and theoretical approaches that have outlined a normative account of cue combination in behavior and neurons, as well as on the systems neuroscience efforts that are searching for its neural implementation. We then highlight a contemporary frontier in our state of knowledge: understanding how velocity cues with time-varying reliabilities are integrated into an evolving position estimate over prolonged time periods. Further, we discuss how the brain builds internal models inferring when cues ought to be integrated versus segregated-a process of causal inference. Lastly, we suggest that the study of spatial navigation has not yet addressed its initial condition: self-location.

Building and Understanding the Minimal Self

Within the methodologically diverse interdisciplinary research on the minimal self, we identify two movements with seemingly disparate research agendas - cognitive science and cognitive (developmental) robotics. Cognitive science, on the one hand, devises rather abstract models which can predict and explain human experimental data related to the minimal self. Incorporating the established models of cognitive science and ideas from artificial intelligence, cognitive robotics, on the other hand, aims to build embodied learning machines capable of developing a self "from scratch" similar to human infants. The epistemic promise of the latter approach is that, at some point, robotic models can serve as a testbed for directly investigating the mechanisms that lead to the emergence of the minimal self. While both approaches can be productive for creating causal mechanistic models of the minimal self, we argue that building a minimal self is different from understanding the human minimal self. Thus, one should be cautious when drawing conclusions about the human minimal self based on robotic model implementations and vice versa. We further point out that incorporating constraints arising from different levels of analysis will be crucial for creating models that can predict, generate, and causally explain behavior in the real world.

Joint representation of working memory and uncertainty in human cortex

Neural representations of visual working memory (VWM) are noisy, and thus, decisions based on VWM are inevitably subject to uncertainty. However, the mechanisms by which the brain simultaneously represents the content and uncertainty of memory remain largely unknown. Here, inspired by the theory of probabilistic population codes, we test the hypothesis that the human brain represents an item maintained in VWM as a probability distribution over stimulus feature space, thereby capturing both its content and uncertainty. We used a neural generative model to decode probability distributions over memorized locations from fMRI activation patterns. We found that the mean of the probability distribution decoded from retinotopic cortical areas predicted memory reports on a trial-by-trial basis. Moreover, in several of the same mid-dorsal stream areas, the spread of the distribution predicted subjective trial-by-trial uncertainty judgments. These results provide evidence that VWM content and uncertainty are jointly represented by probabilistic neural codes.

Neural tuning and representational geometry

A central goal of neuroscience is to understand the representations formed by brain activity patterns and their connection to behaviour. The classic approach is to investigate how individual neurons encode stimuli and how their tuning determines the fidelity of the neural representation. Tuning analyses often use the Fisher information to characterize the sensitivity of neural responses to small changes of the stimulus. In recent decades, measurements of large populations of neurons have motivated a complementary approach, which focuses on the information available to linear decoders. The decodable information is captured by the geometry of the representational patterns in the multivariate response space. Here we review neural tuning and representational geometry with the goal of clarifying the relationship between them. The tuning induces the geometry, but different sets of tuned neurons can induce the same geometry. The geometry determines the Fisher information, the mutual information and the behavioural performance of an ideal observer in a range of psychophysical tasks. We argue that future studies can benefit from considering both tuning and geometry to understand neural codes and reveal the connections between stimuli, brain activity and behaviour.

Breaking the cardinal rule: The impact of interitem interaction and attentional priority on the cardinal biases in orientation working memory

Although it is not typically assumed in influential models of visual working memory (WM), representations in WM are systematically biased by multiple factors. Orientation representations are biased away from the cardinal axis (i.e., cardinal bias) and they are biased away from or toward the other orientation simultaneously held in WM (i.e., interitem interaction). The present study investigated the extent to which these two bias mechanisms interact in WM. In Experiment 1, participants remembered two sequentially presented orientations and reproduced both orientations after a short delay. Cardinal biases were assessed separately for the trials where the two mechanisms produce biases in the same direction (i.e., congruent trials) and the trials where they produce biases in the opposite direction (i.e., incongruent trials). Whereas congruent trials exhibited a typical cardinal bias, incongruent trials exhibited no cardinal bias, demonstrating that the cardinal bias was canceled out by the interitem interaction. Follow-up experiments extended these results by manipulating attentional priority for the two orientations by means of precue (Experiment 2) and postcue (Experiment 3). In both experiments, attentionally prioritized items exhibited a typical cardinal bias irrespective of the congruency whereas attentionally unprioritized items exhibited a reversal of the cardinal bias in the incongruent trials, demonstrating that selective attention modulates the influence of the interitem interaction. Together, these results suggest that WM leverages information about specific stimuli and their relationship to support a given behavioral goal.

Modelling the neural code in large populations of correlated neurons

Neurons respond selectively to stimuli, and thereby define a code that associates stimuli with population response patterns. Certain correlations within population responses (noise correlations) significantly impact the information content of the code, especially in large populations. Understanding the neural code thus necessitates response models that quantify the coding properties of modelled populations, while fitting large-scale neural recordings and capturing noise correlations. In this paper, we propose a class of response model based on mixture models and exponential families. We show how to fit our models with expectation-maximization, and that they capture diverse variability and covariability in recordings of macaque primary visual cortex. We also show how they facilitate accurate Bayesian decoding, provide a closed-form expression for the Fisher information, and are compatible with theories of probabilistic population coding. Our framework could allow researchers to quantitatively validate the predictions of neural coding theories against both large-scale neural recordings and cognitive performance.

Normative Principles for Decision-Making in Natural Environments

The decisions we make are shaped by a lifetime of learning. Past experience guides the way that we encode information in neural systems for perception and valuation, and determines the information we retrieve when making decisions. Distinct literatures have discussed how lifelong learning and local context shape decisions made about sensory signals, propositional information, or economic prospects. Here, we build bridges between these literatures, arguing for common principles of adaptive rationality in perception, cognition, and economic choice. We discuss how a single common framework, based on normative principles of efficient coding and Bayesian inference, can help us understand a myriad of human decision biases, including sensory illusions, adaptive aftereffects, choice history biases, central tendency effects, anchoring effects, contrast effects, framing effects, congruency effects, reference-dependent valuation, nonlinear utility functions, and discretization heuristics. We describe a simple computational framework for explaining these phenomena. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Gravity and Known Size Calibrate Visual Information to Time Parabolic Trajectories

Catching a ball in a parabolic flight is a complex task in which the time and area of interception are strongly coupled, making interception possible for a short period. Although this makes the estimation of time-to-contact (TTC) from visual information in parabolic trajectories very useful, previous attempts to explain our precision in interceptive tasks circumvent the need to estimate TTC to guide our action. Obtaining TTC from optical variables alone in parabolic trajectories would imply very complex transformations from 2D retinal images to a 3D layout. We propose based on previous work and show by using simulations that exploiting prior distributions of gravity and known physical size makes these transformations much simpler, enabling predictive capacities from minimal early visual information. Optical information is inherently ambiguous, and therefore, it is necessary to explain how these prior distributions generate predictions. Here is where the role of prior information comes into play: it could help to interpret and calibrate visual information to yield meaningful predictions of the remaining TTC. The objective of this work is: (1) to describe the primary sources of information available to the observer in parabolic trajectories; (2) unveil how prior information can be used to disambiguate the sources of visual information within a Bayesian encoding-decoding framework; (3) show that such predictions might be robust against complex dynamic environments; and (4) indicate future lines of research to scrutinize the role of prior knowledge calibrating visual information and prediction for action control.

Memory Fidelity Reveals Qualitative Changes in Interactions Between Items in Visual Working Memory

Memory for objects in a display sometimes reveals attraction-the objects are remembered as more similar to one another than they actually were-and sometimes reveals repulsion-the objects are remembered as more different from one another. The conditions that lead to these opposing memory biases are poorly understood; there is no theoretical framework that explains these contrasting dynamics. In three experiments (each N = 30 adults), we demonstrate that memory fidelity provides a unifying dimension that accommodates the existence of both types of visual working memory interactions. We show that either attraction or repulsion can arise simply as a function of manipulations of memory fidelity. We also demonstrate that subjective ratings of fidelity predict the presence of attraction or repulsion on a trial-by-trial basis. We discuss how these results bear on computational models of visual working memory and contextualize these results within the literature of attraction and repulsion effects in long-term memory and perception.

Biased orientation representations can be explained by experience with nonuniform training set statistics

Visual acuity is better for vertical and horizontal compared to other orientations. This cross-species phenomenon is often explained by “efficient coding,” whereby more neurons show sharper tuning for the orientations most common in natural vision. However, it is unclear if experience alone can account for such biases. Here, we measured orientation representations in a convolutional neural network, VGG-16, trained on modified versions of ImageNet (rotated by 0°, 22.5°, or 45° counterclockwise of upright). Discriminability for each model was highest near the orientations that were most common in the network's training set. Furthermore, there was an overrepresentation of narrowly tuned units selective for the most common orientations. These effects emerged in middle layers and increased with depth in the network, though this layer-wise pattern may depend on properties of the evaluation stimuli used. Biases emerged early in training, consistent with the possibility that nonuniform representations may play a functional role in the network's task performance. Together, our results suggest that biased orientation representations can emerge through experience with a nonuniform distribution of orientations, supporting the efficient coding hypothesis.

Neural evidence for categorical biases in location and orientation representations in a working memory task

Previous research demonstrated that visual representations in working memory exhibit biases with respect to the categorical structure of the stimulus space. However, a majority of those studies used behavioral measures of working memory, and it is not clear whether the working memory representations per se are influenced by the categorical structure or whether the biases arise in decision or response processes during the report. Here, I applied a multivariate decoding technique to EEG data collected during working memory tasks to determine whether neural activity associated with the representations in working memory is categorically biased prior to the report. I found that the decoding of spatial working memory was biased away from the nearest cardinal location, consistent with the biases observed in the behavioral responses. In a follow-up experiment which was designed to prevent the use of a response preparation strategy, I found that the decoding still exhibited categorical biases. Together, these results provide neural evidence that working memory representations themselves are categorically biased, imposing important constraints on the models of working memory representations.

"Impossible" Somatosensation and the (Ir)rationality of Perception

Impossible figures represent the world in ways it cannot be. From the work of M. C. Escher to any popular perception textbook, such experiences show how some principles of mental processing can be so entrenched and inflexible as to produce absurd and even incoherent outcomes that could not occur in reality. However, impossible experiences of this sort are mostly limited to visual perception; are there “impossible figures” for other sensory modalities? Here, we import a known magic trick into the laboratory to report and investigate an impossible experience for somatosensation—one that can be physically felt. We show that, even under full-cue conditions with objects that can be freely inspected, subjects can be made to experience a single object alone as feeling heavier than a group of objects that includes the single object as a member—an impossible and phenomenologically striking experience of weight. Moreover, we suggest that this phenomenon—a special case of the size-weight illusion—reflects a kind of “anti-Bayesian” perceptual updating that amplifies a challenge to rational models of perception and cognition.

Categorical judgments do not modify sensory representations in working memory

Categorical judgments can systematically bias the perceptual interpretation of stimulus features. However, it remained unclear whether categorical judgments directly modify working memory representations or, alternatively, generate these biases via an inference process down-stream from working memory. To address this question we ran two novel psychophysical experiments in which human subjects had to reverse their categorical judgments about a stimulus feature, if incorrect, before providing an estimate of the feature. If categorical judgments indeed directly altered sensory representations in working memory, subjects' estimates should reflect some aspects of their initial (incorrect) categorical judgment in those trials. We found no traces of the initial categorical judgment. Rather, subjects seemed to be able to flexibly switch their categorical judgment if needed and use the correct corresponding categorical prior to properly perform feature inference. A cross-validated model comparison also revealed that feedback may lead to selective memory recall such that only memory samples that are consistent with the categorical judgment are accepted for the inference process. Our results suggest that categorical judgments do not modify sensory information in working memory but rather act as top-down expectations in the subsequent sensory recall and inference process.

Atypical and inflexible visual encoding in autism spectrum disorder

Encoding, which involves translating sensory information into neural representations, is a critical first step in the sensory-perceptual pathway. Using a visual orientation task, a new study found both lower encoding capacity and less flexible adaptation in people with autism spectrum disorder.

Individuals with autism spectrum disorder have altered visual encoding capacity

Perceptual anomalies in individuals with autism spectrum disorder (ASD) have been attributed to an imbalance in weighting incoming sensory evidence with prior knowledge when interpreting sensory information. Here, we show that sensory encoding and how it adapts to changing stimulus statistics during feedback also characteristically differs between neurotypical and ASD groups. In a visual orientation estimation task, we extracted the accuracy of sensory encoding from psychophysical data by using an information theoretic measure. Initially, sensory representations in both groups reflected the statistics of visual orientations in natural scenes, but encoding capacity was overall lower in the ASD group. Exposure to an artificial (i.e., uniform) distribution of visual orientations coupled with performance feedback altered the sensory representations of the neurotypical group toward the novel experimental statistics, while also increasing their total encoding capacity. In contrast, neither total encoding capacity nor its allocation significantly changed in the ASD group. Across both groups, the degree of adaptation was correlated with participants’ initial encoding capacity. These findings highlight substantial deficits in sensory encoding—independent from and potentially in addition to deficits in decoding—in individuals with ASD.

It is increasingly recognized that individuals with Autism Spectrum Disorder (ASD) show anomalies in perception, and these have been recently attributed to altered decoding (i.e. interpretation of sensory signals). This study reveals that independent of these changes, individuals with ASD show upstream deficits in sensory encoding (i.e., how samples are drawn from the environment).

The virtual loss function in the summary perception of motion and its limited adjustability

Humans can grasp the "average" feature of a visual ensemble quickly and effortlessly. However, it is largely unknown what is the exact form of the summary statistic humans perceive and it is even less known whether this form can be changed by feedback. Here we borrow the concept of loss function to characterize how the summary perception is related to the distribution of feature values in the ensemble, assuming that the summary statistic minimizes a virtual expected loss associated with its deviation from individual feature values. In two experiments, we investigated a random-dot motion estimation task to infer the virtual loss function implicit in ensemble perception and see whether it can be changed by feedback. On each trial, participants reported the average moving direction of an ensemble of moving dots whose distribution of moving directions was skewed. In Experiment 1, where no feedback was available, participants' estimates fell between the mean and the mode of the distribution and were closer to the mean. In particular, the deviation from the mean and toward the mode increased almost linearly with the mode-to-mean distance. The pattern was best modeled by an inverse Gaussian loss function, which punishes large errors less heavily than the quadratic loss function does. In Experiment 2, we tested whether this virtual loss function can be altered by feedback. Two groups of participants either received the mode or the mean as the correct answer. After extensive training up to five days, both groups' estimates moved slightly towards the mode. That is, feedback had no specific influence on participants' virtual loss function. To conclude, the virtual loss function in the summary perception of motion is close to inverse Gaussian, and it can hardly be changed by feedback.

Trading mental effort for confidence in the metacognitive control of value-based decision-making

Why do we sometimes opt for actions or items that we do not value the most? Under current neurocomputational theories, such preference reversals are typically interpreted in terms of errors that arise from the unreliable signaling of value to brain decision systems. But, an alternative explanation is that people may change their mind because they are reassessing the value of alternative options while pondering the decision. So, why do we carefully ponder some decisions, but not others? In this work, we derive a computational model of the metacognitive control of decisions or MCD. In brief, we assume that fast and automatic processes first provide initial (and largely uncertain) representations of options' values, yielding prior estimates of decision difficulty. These uncertain value representations are then refined by deploying cognitive (e.g., attentional, mnesic) resources, the allocation of which is controlled by an effort-confidence tradeoff. Importantly, the anticipated benefit of allocating resources varies in a decision-by-decision manner according to the prior estimate of decision difficulty. The ensuing MCD model predicts response time, subjective feeling of effort, choice confidence, changes of mind, as well as choice-induced preference change and certainty gain. We test these predictions in a systematic manner, using a dedicated behavioral paradigm. Our results provide a quantitative link between mental effort, choice confidence, and preference reversals, which could inform interpretations of related neuroimaging findings.

Color Tuning of Face-Selective Neurons in Macaque Inferior Temporal Cortex

What role does color play in the neural representation of complex shapes? We approached the question by measuring color responses of face-selective neurons, using fMRI-guided microelectrode recording of the middle and anterior face patches of inferior temporal cortex (IT) in rhesus macaques. Face-selective cells responded weakly to pure color (equiluminant) photographs of faces. But many of the cells nonetheless showed a bias for warm colors when assessed using images that preserved the luminance contrast relationships of the original photographs. This bias was also found for non-face-selective neurons. Fourier analysis uncovered two components: the first harmonic, accounting for most of the tuning, was biased toward reddish colors, corresponding to the L>M pole of the L-M cardinal axis. The second harmonic showed a bias for modulation between blue and yellow colors axis, corresponding to the S-cone axis. To test what role face-selective cells play in behavior, we related the information content of the neural population with the distribution of face colors. The analyses show that face-selective cells are not optimally tuned to discriminate face colors, but are consistent with the idea that face-selective cells contribute selectively to processing the green-red contrast of faces. The research supports the hypothesis that color-specific information related to the discrimination of objects, including faces, is handled by neural circuits that are independent of shape-selective cortex, as captured by the multistage parallel processing framework of IT (Lafer-Sousa and Conway, 2013).

Serial reproduction reveals the geometry of visuospatial representations

A primary function of human vision is to encode and recall spatial information about visual scenes. We developed an experimental paradigm that reveals the structure of human spatial memory priors in unprecedented detail. We ran a series of 85 large-scale online experiments with 9,202 participants that paint an intricate picture of these priors. Our results suggest a way to understand visuospatial representations as reflecting the efficient allocation of coding resources. In a radical departure from traditional theory, we introduce a model that reinterprets spatial memory priors as reflecting an optimal allocation of perceptual resources. We validate the predictions of the model experimentally by showing that perceptual biases are correlated with variations in discrimination accuracy.

An essential function of the human visual system is to locate objects in space and navigate the environment. Due to limited resources, the visual system achieves this by combining imperfect sensory information with a belief state about locations in a scene, resulting in systematic distortions and biases. These biases can be captured by a Bayesian model in which internal beliefs are expressed in a prior probability distribution over locations in a scene. We introduce a paradigm that enables us to measure these priors by iterating a simple memory task where the response of one participant becomes the stimulus for the next. This approach reveals an unprecedented richness and level of detail in these priors, suggesting a different way to think about biases in spatial memory. A prior distribution on locations in a visual scene can reflect the selective allocation of coding resources to different visual regions during encoding (“efficient encoding”). This selective allocation predicts that locations in the scene will be encoded with variable precision, in contrast to previous work that has assumed fixed encoding precision regardless of location. We demonstrate that perceptual biases covary with variations in discrimination accuracy, a finding that is aligned with simulations of our efficient encoding model but not the traditional fixed encoding view. This work demonstrates the promise of using nonparametric data-driven approaches that combine crowdsourcing with the careful curation of information transmission within social networks to reveal the hidden structure of shared visual representations.

Emulating Cued Recall of Abstract Concepts via Regulated Activation Networks

Abstract concepts play a vital role in decision-making or recall operations because the associations among them are essential for contextual processing. Abstract concepts are complex and difficult to represent (conceptually, formally, or computationally), leading to difficulties in their comprehension and recall. This contribution reports the computational simulation of the cued recall of abstract concepts by exploiting their learned associations. The cued recall operation is realized via a novel geometric back-propagation algorithm that emulates the recall of abstract concepts learned through regulated activation network (RAN) modeling. During recall operation, another algorithm uniquely regulates the activation of concepts (nodes) by injecting excitatory, neutral, and inhibitory signals to other concepts of the same level. A Toy-data problem is considered to illustrate the RAN modeling and recall procedure. The results display how regulation enables contextual awareness among abstract nodes during the recall process. The MNIST dataset is used to show how recall operations retrieve intuitive and non-intuitive blends of abstract nodes. We show that every recall process converges to an optimal image. With more cues, better images are recalled, and every intermediate image obtained during the recall iterations corresponds to the varying cognitive states of the recognition procedure.

Expectations about motion direction affect perception and anticipatory smooth pursuit differently

Smooth pursuit eye movements and visual motion perception rely on the integration of current sensory signals with past experience. Experience shapes our expectation of current visual events and can drive eye movement responses made in anticipation of a target, such as anticipatory pursuit. Previous research revealed consistent effects of expectation on anticipatory pursuit-eye movements follow the expected target direction or speed-and contrasting effects on motion perception, but most studies considered either eye movement or perceptual responses. The current study directly compared effects of direction expectation on perception and anticipatory pursuit within the same direction discrimination task to investigate whether both types of responses are affected similarly or differently. Observers (n = 10) viewed high-coherence random-dot kinematograms (RDKs) moving rightward and leftward with a probability of 50%, 70%, or 90% in a given block of trials to build up an expectation of motion direction. They were asked to judge motion direction of interleaved low-coherence RDKs (0%-15%). Perceptual judgements were compared with changes in anticipatory pursuit eye movements as a function of probability. Results show that anticipatory pursuit velocity scaled with probability and followed direction expectation (attraction bias), whereas perceptual judgments were biased opposite to direction expectation (repulsion bias). Control experiments suggest that the repulsion bias in perception was not caused by retinal slip induced by anticipatory pursuit, or by motion adaptation. We conclude that direction expectation can be processed differently for perception and anticipatory pursuit.NEW & NOTEWORTHY We show that expectations about motion direction that are based on long-term trial history affect perception and anticipatory pursuit differently. Whereas anticipatory pursuit direction was coherent with the expected motion direction (attraction bias), perception was biased opposite to the expected direction (repulsion bias). These opposite biases potentially reveal different ways in which perception and action utilize prior information and support the idea of different information processing for perception and pursuit.

Population codes of prior knowledge learned through environmental regularities

How the brain makes correct inferences about its environment based on noisy and ambiguous observations is one of the fundamental questions in Neuroscience. Prior knowledge about the probability with which certain events occur in the environment plays an important role in this process. Humans are able to incorporate such prior knowledge in an efficient, Bayes optimal, way in many situations, but it remains an open question how the brain acquires and represents this prior knowledge. The long time spans over which prior knowledge is acquired make it a challenging question to investigate experimentally. In order to guide future experiments with clear empirical predictions, we used a neural network model to learn two commonly used tasks in the experimental literature (i.e. orientation classification and orientation estimation) where the prior probability of observing a certain stimulus is manipulated. We show that a population of neurons learns to correctly represent and incorporate prior knowledge, by only receiving feedback about the accuracy of their inference from trial-to-trial and without any probabilistic feedback. We identify different factors that can influence the neural responses to unexpected or expected stimuli, and find a novel mechanism that changes the activation threshold of neurons, depending on the prior probability of the encoded stimulus. In a task where estimating the exact stimulus value is important, more likely stimuli also led to denser tuning curve distributions and narrower tuning curves, allocating computational resources such that information processing is enhanced for more likely stimuli. These results can explain several different experimental findings, clarify why some contradicting observations concerning the neural responses to expected versus unexpected stimuli have been reported and pose some clear and testable predictions about the neural representation of prior knowledge that can guide future experiments.

Uncertainty-based overestimation in the perception of group actions

Individuals are adept at estimating average properties of group visual stimuli, even following brief presentations. In estimating the directional heading of walking human figures, judgments are biased in a peculiar manner: groups facing intermediate directions are perceived to be more leftward- or rightward-facing than actual averages. This effect was previously explained as a repulsive bias away from a central category boundary; groups along this boundary (directly facing the observer) are estimated with lower variability and with relatively greater accuracy. Here we show that: (i) the original effect replicates and is constant over time in a novel estimation task with persistent directional states; and, (ii) novel patterns of response variability and durations align with the entire range of overestimation. A simple model of additive errors proportional to viewer uncertainty matches the observed bias magnitudes. We furthermore show that the bias generalizes beyond approaching walkers with the use of rearward-facing walkers presented at a nonparallel angle. Overall, the recurring relation between bias and uncertainty is also consistent with top-down and post-perceptual causes of misestimation.

How the known reference weakens the visual oblique effect: a Bayesian account of cognitive improvement by cue influence

This paper investigates the influence of a known cue on the oblique effect in orientation identification and explains how subjects integrate cue information to identify target orientations. We design the psychophysical task in which subjects estimate target orientations in the presence of a known oriented reference line. For comparison the control experiments without the reference are conducted. Under Bayesian inference framework, a cue integration model is proposed to explain the perceptual improvement in the presence of the reference. The maximum likelihood estimates of the parameters of our model are obtained. In the presence of the reference, the variability and biases of identification are significantly reduced and the oblique effect of orientation identification is obviously weakened. Moreover, the identification of orientation in the vicinity of the reference line is consistently biased away from the reference line (i.e., reference repulsion). Comparing the predictions of the model with the experimental results, the Bayesian Least Squares estimator under the Variable-Precision encoding (BLS_VP) provides a better description of the experimental outcomes and captures the trade-off relationship of bias and precision of identification. Our results provide a useful step toward a better understanding of human visual perception in context of the known cues.

Temperature-Based Crossmodal Correspondences: Causes and Consequences

The last few years have seen an explosive growth of research interest in the crossmodal correspondences, the sometimes surprising associations that people experience between stimuli, attributes, or perceptual dimensions, such as between auditory pitch and visual size, or elevation. To date, the majority of this research has tended to focus on audiovisual correspondences. However, a variety of crossmodal correspondences have also been demonstrated with tactile stimuli, involving everything from felt shape to texture, and from weight through to temperature. In this review, I take a closer look at temperature-based correspondences. The empirical research not only supports the existence of robust crossmodal correspondences between temperature and colour (as captured by everyday phrases such as 'red hot') but also between temperature and auditory pitch. Importantly, such correspondences have (on occasion) been shown to influence everything from our thermal comfort in coloured environments through to our response to the thermal and chemical warmth associated with stimulation of the chemical senses, as when eating, drinking, and sniffing olfactory stimuli. Temperature-based correspondences are considered in terms of the four main classes of correspondence that have been identified to date, namely statistical, structural, semantic, and affective. The hope is that gaining a better understanding of temperature-based crossmodal correspondences may one day also potentially help in the design of more intuitive sensory-substitution devices, and support the delivery of immersive virtual and augmented reality experiences.