Nonstimulated early visual areas carry information about surrounding context

The cognitive penetrability of perception: A blocked debate and a tentative solution

Despite the extensive body of psychological findings suggesting that cognition influences perception, the debate between defenders and detractors of the cognitive penetrability of perception persists. While detractors demand more strictness in psychological experiments, proponents consider that empirical studies show that cognitive penetrability occurs. These considerations have led some theorists to propose that the debate has reached a dead end. The issue about where perception ends and cognition begins is, I argue, one of the reasons why the debate is cornered. Another reason is the inability of psychological studies to present uncontroversial interpretations of the results obtained. To dive into other kinds of empirical sources is, therefore, required to clarify the debate. In this paper, I explain where the debate is blocked, and suggest that neuroscientific evidence together with the predictive coding account, might decant the discussion on the side of the penetrability thesis.

Scene Representations Conveyed by Cortical Feedback to Early Visual Cortex Can Be Described by Line Drawings

Human behavior is dependent on the ability of neuronal circuits to predict the outside world. Neuronal circuits in early visual areas make these predictions based on internal models that are delivered via non-feedforward connections. Despite our extensive knowledge of the feedforward sensory features that drive cortical neurons, we have a limited grasp on the structure of the brain's internal models. Progress in neuroscience therefore depends on our ability to replicate the models that the brain creates internally. Here we record human fMRI data while presenting partially occluded visual scenes. Visual occlusion allows us to experimentally control sensory input to subregions of visual cortex while internal models continue to influence activity in these regions. Because the observed activity is dependent on internal models, but not on sensory input, we have the opportunity to map visual features conveyed by the brain's internal models. Our results show that activity related to internal models in early visual cortex are more related to scene-specific features than to categorical or depth features. We further demonstrate that behavioral line drawings provide a good description of internal model structure representing scene-specific features. These findings extend our understanding of internal models, showing that line drawings provide a window into our brains' internal models of vision.

SIGNIFICANCE STATEMENT We find that fMRI activity patterns corresponding to occluded visual information in early visual cortex fill in scene-specific features. Line drawings of the missing scene information correlate with our recorded activity patterns, and thus to internal models. Despite our extensive knowledge of the sensory features that drive cortical neurons, we have a limited grasp on the structure of our brains' internal models. These results therefore constitute an advance to the field of neuroscience by extending our knowledge about the models that our brains construct to efficiently represent and predict the world. Moreover, they link a behavioral measure to these internal models, which play an active role in many components of human behavior, including visual predictions, action planning, and decision making.

Towards a Unified View on Pathways and Functions of Neural Recurrent Processing

There are three neural feedback pathways to the primary visual cortex (V1): corticocortical, pulvinocortical, and cholinergic. What are the respective functions of these three projections? Possible functions range from contextual modulation of stimulus processing and feedback of high-level information to predictive processing (PP). How are these functions subserved by different pathways and can they be integrated into an overarching theoretical framework? We propose that corticocortical and pulvinocortical connections are involved in all three functions, whereas the role of cholinergic projections is limited by their slow response to stimuli. PP provides a broad explanatory framework under which stimulus-context modulation and high-level processing are subsumed, involving multiple feedback pathways that provide mechanisms for inferring and interpreting what sensory inputs are about.

Automatic and feature-specific prediction-related neural activity in the human auditory system

Prior experience enables the formation of expectations of upcoming sensory events. However, in the auditory modality, it is not known whether prediction-related neural signals carry feature-specific information. Here, using magnetoencephalography (MEG), we examined whether predictions of future auditory stimuli carry tonotopic specific information. Participants passively listened to sound sequences of four carrier frequencies (tones) with a fixed presentation rate, ensuring strong temporal expectations of when the next stimulus would occur. Expectation of which frequency would occur was parametrically modulated across the sequences, and sounds were occasionally omitted. We show that increasing the regularity of the sequence boosts carrier-frequency-specific neural activity patterns during both the anticipatory and omission periods, indicating that prediction-related neural activity is indeed feature-specific. Our results illustrate that even without bottom-up input, auditory predictions can activate tonotopically specific templates.

After listening to a predictable sequence of sounds, we can anticipate and predict the next sound in the sequence. Here, the authors show that during expectation of a sound, the brain generates neural activity matching that which is produced by actually hearing the same sound.

Modelling face memory reveals task-generalizable representations

Current cognitive theories are cast in terms of information-processing mechanisms that use mental representations^1-4. For example, people use their mental representations to identify familiar faces under various conditions of pose, illumination and ageing, or to draw resemblance between family members. Yet, the actual information contents of these representations are rarely characterized, which hinders knowledge of the mechanisms that use them. Here, we modelled the three-dimensional representational contents of 4 faces that were familiar to 14 participants as work colleagues. The representational contents were created by reverse-correlating identity information generated on each trial with judgements of the face's similarity to the individual participant's memory of this face. In a second study, testing new participants, we demonstrated the validity of the modelled contents using everyday face tasks that generalize identity judgements to new viewpoints, age and sex. Our work highlights that such models of mental representations are critical to understanding generalization behaviour and its underlying information-processing mechanisms.

Feature-specific prediction errors for visual mismatch

Predictive coding (PC) theory posits that our brain employs a predictive model of the environment to infer the causes of its sensory inputs. A fundamental but untested prediction of this theory is that the same stimulus should elicit distinct precision weighted prediction errors (pwPEs) when different (feature-specific) predictions are violated, even in the absence of attention. Here, we tested this hypothesis using functional magnetic resonance imaging (fMRI) and a multi-feature roving visual mismatch paradigm where rare changes in either color (red, green), or emotional expression (happy, fearful) of faces elicited pwPE responses in human participants. Using a computational model of learning and inference, we simulated pwPE and prediction trajectories of a Bayes-optimal observer and used these to analyze changes in blood oxygen level dependent (BOLD) responses to changes in color and emotional expression of faces while participants engaged in a distractor task. Controlling for visual attention by eye-tracking, we found pwPE responses to unexpected color changes in the fusiform gyrus. Conversely, unexpected changes of facial emotions elicited pwPE responses in cortico-thalamo-cerebellar structures associated with emotion and theory of mind processing. Predictions pertaining to emotions activated fusiform, occipital and temporal areas. Our results are consistent with a general role of PC across perception, from low-level to complex and socially relevant object features, and suggest that monitoring of the social environment occurs continuously and automatically, even in the absence of attention.

Decoding the dynamic representation of facial expressions of emotion in explicit and incidental tasks

Faces transmit a wealth of important social signals. While previous studies have elucidated the network of cortical regions important for perception of facial expression, and the associated temporal components such as the P100, N170 and EPN, it is still unclear how task constraints may shape the representation of facial expression (or other face categories) in these networks. In the present experiment, we used Multivariate Pattern Analysis (MVPA) with EEG to investigate the neural information available across time about two important face categories (expression and identity) when those categories are either perceived under explicit (e.g. decoding facial expression category from the EEG when task is on expression) or incidental task contexts (e.g. decoding facial expression category from the EEG when task is on identity). Decoding of both face categories, across both task contexts, peaked in time-windows spanning 91-170 ms (across posterior electrodes). Peak decoding of expression, however, was not affected by task context whereas peak decoding of identity was significantly reduced under incidental processing conditions. In addition, errors in EEG decoding correlated with errors in behavioral categorization under explicit processing for both expression and identity, however under incidental conditions only errors in EEG decoding of expression correlated with behavior. Furthermore, decoding time-courses and the spatial pattern of informative electrodes showed consistently better decoding of identity under explicit conditions at later-time periods, with weak evidence for similar effects for decoding of expression at isolated time-windows. Taken together, these results reveal differences and commonalities in the processing of face categories under explicit Vs incidental task contexts and suggest that facial expressions are processed to a richer degree under incidental processing conditions, consistent with prior work indicating the relative automaticity by which emotion is processed. Our work further demonstrates the utility in applying multivariate decoding analyses to EEG for revealing the dynamics of face perception.

Neuroimaging Findings on Amodal Completion: A Review

Amodal completion is the phenomenon of perceiving completed objects even though physically they are partially occluded. In this review, we provide an extensive overview of the results obtained from a variety of neuroimaging studies on the neural correlates of amodal completion. We discuss whether low-level and high-level cortical areas are implicated in amodal completion; provide an overview of how amodal completion unfolds over time while dissociating feedforward, recurrent, and feedback processes; and discuss how amodal completion is represented at the neuronal level. The involvement of low-level visual areas such as V1 and V2 is not yet clear, while several high-level structures such as the lateral occipital complex and fusiform face area seem invariant to occlusion of objects and faces, respectively, and several motor areas seem to code for object permanence. The variety of results on the timing of amodal completion hints to a mixture of feedforward, recurrent, and feedback processes. We discuss whether the invisible parts of the occluded object are represented as if they were visible, contrary to a high-level representation. While plenty of questions on amodal completion remain, this review presents an overview of the neuroimaging findings reported to date, summarizes several insights from computational models, and connects research of other perceptual completion processes such as modal completion. In all, it is suggested that amodal completion is the solution to deal with various types of incomplete retinal information, and highly depends on stimulus complexity and saliency, and therefore also give rise to a variety of observed neural patterns.

Sleep Strengthens Predictive Sequence Coding

Predictive-coding theories assume that perception and action are based on internal models derived from previous experience. Such internal models require selection and consolidation to be stored over time. Sleep is known to support memory consolidation. We hypothesized that sleep supports both consolidation and abstraction of an internal task model that is subsequently used to predict upcoming stimuli. Human subjects (of either sex) were trained on deterministic visual sequences and tested with interleaved deviant stimuli after retention intervals of sleep or wakefulness. Adopting a predictive-coding approach, we found increased prediction strength after sleep, as expressed by increased error rates to deviant stimuli, but fewer errors for the immediately following standard stimuli. Sleep likewise enhanced the formation of an abstract sequence model, independent of the temporal context during training. Moreover, sleep increased confidence for sequence knowledge, reflecting enhanced metacognitive access to the model. Our results suggest that sleep supports the formation of internal models which can be used to predict upcoming events in different contexts.SIGNIFICANCE STATEMENT To efficiently interact with the ever-changing world, we predict upcoming events based on similar previous experiences. Sleep is known to benefit memory consolidation. However, it is not clear whether sleep specifically supports the transformation of past experience into predictions of future events. Here, we find that, when human subjects sleep after learning a sequence of predictable visual events, they make better predictions about upcoming events compared with subjects who stayed awake for an equivalent period of time. In addition, sleep supports the transfer of such knowledge between different temporal contexts (i.e., when sequences unfold at different speeds). Thus, sleep supports perception and action by enhancing the predictive utility of previous experiences.

How Do Expectations Shape Perception?

Perception and perceptual decision-making are strongly facilitated by prior knowledge about the probabilistic structure of the world. While the computational benefits of using prior expectation in perception are clear, there are myriad ways in which this computation can be realized. We review here recent advances in our understanding of the neural sources and targets of expectations in perception. Furthermore, we discuss Bayesian theories of perception that prescribe how an agent should integrate prior knowledge and sensory information, and investigate how current and future empirical data can inform and constrain computational frameworks that implement such probabilistic integration in perception.

Blindness and Human Brain Plasticity

Early blindness causes fundamental alterations of neural function across more than 25% of cortex-changes that span the gamut from metabolism to behavior and collectively represent one of the most dramatic examples of plasticity in the human brain. The goal of this review is to describe how the remarkable behavioral and neuroanatomical compensations demonstrated by blind individuals provide insights into the extent, mechanisms, and limits of human brain plasticity.

The Importance of Amodal Completion in Everyday Perception

Amodal completion is the representation of those parts of the perceived object that we get no sensory stimulation from. In the case of vision, it is the representation of occluded parts of objects we see: When we see a cat behind a picket fence, our perceptual system represents those parts of the cat that are occluded by the picket fence. The aim of this piece is to argue that amodal completion plays a constitutive role in our everyday perception and trace the theoretical consequences of this claim.

Identifying and detecting facial expressions of emotion in peripheral vision

Facial expressions of emotion are signals of high biological value. Whilst recognition of facial expressions has been much studied in central vision, the ability to perceive these signals in peripheral vision has only seen limited research to date, despite the potential adaptive advantages of such perception. In the present experiment, we investigate facial expression recognition and detection performance for each of the basic emotions (plus neutral) at up to 30 degrees of eccentricity. We demonstrate, as expected, a decrease in recognition and detection performance with increasing eccentricity, with happiness and surprised being the best recognized expressions in peripheral vision. In detection however, while happiness and surprised are still well detected, fear is also a well detected expression. We show that fear is a better detected than recognized expression. Our results demonstrate that task constraints shape the perception of expression in peripheral vision and provide novel evidence that detection and recognition rely on partially separate underlying mechanisms, with the latter more dependent on the higher spatial frequency content of the face stimulus.

Mapping Frequency-Specific Tone Predictions in the Human Auditory Cortex at High Spatial Resolution

Auditory inputs reaching our ears are often incomplete, but our brains nevertheless transform them into rich and complete perceptual phenomena such as meaningful conversations or pleasurable music. It has been hypothesized that our brains extract regularities in inputs, which enables us to predict the upcoming stimuli, leading to efficient sensory processing. However, it is unclear whether tone predictions are encoded with similar specificity as perceived signals. Here, we used high-field fMRI to investigate whether human auditory regions encode one of the most defining characteristics of auditory perception: the frequency of predicted tones. Two pairs of tone sequences were presented in ascending or descending directions, with the last tone omitted in half of the trials. Every pair of incomplete sequences contained identical sounds, but was associated with different expectations about the last tone (a high- or low-frequency target). This allowed us to disambiguate predictive signaling from sensory-driven processing. We recorded fMRI responses from eight female participants during passive listening to complete and incomplete sequences. Inspection of specificity and spatial patterns of responses revealed that target frequencies were encoded similarly during their presentations, as well as during omissions, suggesting frequency-specific encoding of predicted tones in the auditory cortex (AC). Importantly, frequency specificity of predictive signaling was observed already at the earliest levels of auditory cortical hierarchy: in the primary AC. Our findings provide evidence for content-specific predictive processing starting at the earliest cortical levels.SIGNIFICANCE STATEMENT Given the abundance of sensory information around us in any given moment, it has been proposed that our brain uses contextual information to prioritize and form predictions about incoming signals. However, there remains a surprising lack of understanding of the specificity and content of such prediction signaling; for example, whether a predicted tone is encoded with similar specificity as a perceived tone. Here, we show that early auditory regions encode the frequency of a tone that is predicted yet omitted. Our findings contribute to the understanding of how expectations shape sound processing in the human auditory cortex and provide further insights into how contextual information influences computations in neuronal circuits.

Detailed spatiotemporal brain mapping of chromatic vision combining high-resolution VEP with fMRI and retinotopy

Neuroimaging studies have identified so far, several color-sensitive visual areas in the human brain, and the temporal dynamics of these activities have been separately investigated using the visual-evoked potentials (VEPs). In the present study, we combined electrophysiological and neuroimaging methods to determine a detailed spatiotemporal profile of chromatic VEP and to localize its neural generators. The accuracy of the present co-registration study was obtained by combining standard fMRI data with retinotopic and motion mapping data at the individual level. We found a sequence of occipito activities more complex than that typically reported for chromatic VEPs, including feed-forward and reentrant feedback. Results showed that chromatic human perception arises by the combined activity of at the least five parieto-occipital areas including V1, LOC, V8/VO, and the motion-sensitive dorsal region MT+. However, the contribution of V1 and V8/VO seems dominant because the re-entrant activity in these areas was present more than once (twice in V8/VO and thrice in V1). This feedforward and feedback chromatic processing appears delayed compared with the luminance processing. Associating VEPs and neuroimaging measures, we showed for the first time a complex spatiotemporal pattern of activity, confirming that chromatic stimuli produce intricate interactions of many different brain dorsal and ventral areas.

Connectivity Reveals Sources of Predictive Coding Signals in Early Visual Cortex During Processing of Visual Optic Flow

Superimposed on the visual feed-forward pathway, feedback connections convey higher level information to cortical areas lower in the hierarchy. A prominent framework for these connections is the theory of predictive coding where high-level areas send stimulus interpretations to lower level areas that compare them with sensory input. Along these lines, a growing body of neuroimaging studies shows that predictable stimuli lead to reduced blood oxygen level-dependent (BOLD) responses compared with matched nonpredictable counterparts, especially in early visual cortex (EVC) including areas V1-V3. The sources of these modulatory feedback signals are largely unknown. Here, we re-examined the robust finding of relative BOLD suppression in EVC evident during processing of coherent compared with random motion. Using functional connectivity analysis, we show an optic flow-dependent increase of functional connectivity between BOLD suppressed EVC and a network of visual motion areas including MST, V3A, V6, the cingulate sulcus visual area (CSv), and precuneus (Pc). Connectivity decreased between EVC and 2 areas known to encode heading direction: entorhinal cortex (EC) and retrosplenial cortex (RSC). Our results provide first evidence that BOLD suppression in EVC for predictable stimuli is indeed mediated by specific high-level areas, in accord with the theory of predictive coding.

Cortical feedback signals generalise across different spatial frequencies of feedforward inputs

Visual processing in cortex relies on feedback projections contextualising feedforward information flow. Primary visual cortex (V1) has small receptive fields and processes feedforward information at a fine-grained spatial scale, whereas higher visual areas have larger, spatially invariant receptive fields. Therefore, feedback could provide coarse information about the global scene structure or alternatively recover fine-grained structure by targeting small receptive fields in V1. We tested if feedback signals generalise across different spatial frequencies of feedforward inputs, or if they are tuned to the spatial scale of the visual scene. Using a partial occlusion paradigm, functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (MVPA) we investigated whether feedback to V1 contains coarse or fine-grained information by manipulating the spatial frequency of the scene surround outside an occluded image portion. We show that feedback transmits both coarse and fine-grained information as it carries information about both low (LSF) and high spatial frequencies (HSF). Further, feedback signals containing LSF information are similar to feedback signals containing HSF information, even without a large overlap in spatial frequency bands of the HSF and LSF scenes. Lastly, we found that feedback carries similar information about the spatial frequency band across different scenes. We conclude that cortical feedback signals contain information which generalises across different spatial frequencies of feedforward inputs.

Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition

Feedforward neural networks provide the dominant model of how the brain performs visual object recognition. However, these networks lack the lateral and feedback connections, and the resulting recurrent neuronal dynamics, of the ventral visual pathway in the human and non-human primate brain. Here we investigate recurrent convolutional neural networks with bottom-up (B), lateral (L), and top-down (T) connections. Combining these types of connections yields four architectures (B, BT, BL, and BLT), which we systematically test and compare. We hypothesized that recurrent dynamics might improve recognition performance in the challenging scenario of partial occlusion. We introduce two novel occluded object recognition tasks to test the efficacy of the models, digit clutter (where multiple target digits occlude one another) and digit debris (where target digits are occluded by digit fragments). We find that recurrent neural networks outperform feedforward control models (approximately matched in parametric complexity) at recognizing objects, both in the absence of occlusion and in all occlusion conditions. Recurrent networks were also found to be more robust to the inclusion of additive Gaussian noise. Recurrent neural networks are better in two respects: (1) they are more neurobiologically realistic than their feedforward counterparts; (2) they are better in terms of their ability to recognize objects, especially under challenging conditions. This work shows that computer vision can benefit from using recurrent convolutional architectures and suggests that the ubiquitous recurrent connections in biological brains are essential for task performance.

Predictions Shape Confidence in Right Inferior Frontal Gyrus

It is clear that prior expectations shape perceptual decision-making, yet their contribution to the construction of subjective decision confidence remains largely unexplored. We recorded fMRI data while participants made perceptual decisions and confidence judgments, manipulating perceptual prior expectations while controlling for potential confounds of attention. Results show that subjective confidence increases as expectations increasingly support the decision, and that this relationship is associated with BOLD activity in right inferior frontal gyrus (rIFG). Specifically, rIFG is sensitive to the discrepancy between expectation and decision (mismatch), and higher mismatch responses are associated with lower decision confidence. Connectivity analyses revealed expectancy information to be represented in bilateral orbitofrontal cortex and sensory signals to be represented in intracalcarine sulcus. Together, our results indicate that predictive information is integrated into subjective confidence in rIFG, and reveal an occipital-frontal network that constructs confidence from top-down and bottom-up signals. This interpretation was further supported by exploratory findings that the white matter density of right orbitofrontal cortex negatively predicted its respective contribution to the construction of confidence. Our findings advance our understanding of the neural basis of subjective perceptual processes by revealing an occipitofrontal functional network that integrates prior beliefs into the construction of confidence.

SIGNIFICANCE STATEMENT

Perceptual decision-making is typically conceived as an integration of bottom-up and top-down influences. However, perceptual decisions are accompanied by a sense of confidence. Confidence is an important facet of perceptual consciousness yet remains poorly understood. Here we implicate right inferior frontal gyrus in constructing confidence from the discrepancy between perceptual judgment and its prior probability. Furthermore, we place right inferior frontal gyrus within an occipitofrontal network, consisting of orbitofrontal cortex and intracalcarine sulcus, which represents and communicates relevant top-down and bottom-up signals. Together, our data reveal a role of frontal regions in the top-down processes enabling perceptual decisions to become available for conscious report.

Invariance of surface color representations across illuminant changes in the human cortex

A central problem in color vision is that the light reaching the eye from a given surface can vary dramatically depending on the illumination. Despite this, our color percept, the brain's estimate of surface reflectance, remains remarkably stable. This phenomenon is called color constancy. Here we investigated which human brain regions represent surface color in a way that is invariant with respect to illuminant changes. We used physically realistic rendering methods to display natural yet abstract 3D scenes that were displayed under three distinct illuminants. The scenes embedded, in different conditions, surfaces that differed in their surface color (i.e. in their reflectance property). We used multivariate fMRI pattern analysis to probe neural coding of surface reflectance and illuminant, respectively. While all visual regions encoded surface color when viewed under the same illuminant, we found that only in V1 and V4α surface color representations were invariant to illumination changes. Along the visual hierarchy there was a gradient from V1 to V4α to increasingly encode surface color rather than illumination. Finally, effects of a stimulus manipulation on individual behavioral color constancy indices correlated with neural encoding of the illuminant in hV4. This provides neural evidence for the Equivalent Illuminant Model. Our results provide a principled characterization of color constancy mechanisms across the visual hierarchy, and demonstrate complementary contributions in early and late processing stages.

Early retinotopic responses to violations of emotion-location associations may depend on conscious awareness

Reports of modulations of early visual processing suggest that retinotopic visual cortex may actively predict upcoming stimuli. We tested this idea by showing healthy human participants images of human faces at fixation, with different emotional expressions predicting stimuli in either the upper or the lower visual field. On infrequent test trials, emotional faces were followed by combined stimulation of upper and lower visual fields, thus violating previously established associations. Results showed no effects of such violations at the level of the retinotopic C1 of the visual evoked potential over the full sample. However, when separating participants who became aware of these associations from those who did not, we observed significant group differences during extrastriate processing of emotional faces, with inverse solution results indicating stronger activity in unaware subjects throughout the ventral visual stream. Moreover, within-group comparisons showed that the same peripheral stimuli elicited differential activity patterns during the C1 interval, depending on which stimulus elements were predictable. This effect was selectively observed in manipulation-aware subjects. Our results provide preliminary evidence for the notion that early visual processing stages implement predictions of upcoming events. They also point to conscious awareness as a moderator of predictive coding.

Human visual perceptual organization beats thinking on speed

What is the degree to which knowledge influences visual perceptual processes? This question, which is central to the seeing-versus-thinking debate in cognitive science, is often discussed using examples claimed to be proof of one stance or another. It has, however, also been muddled by the usage of different and unclear definitions of perception. Here, for the well-defined process of perceptual organization, I argue that including speed (or efficiency) into the equation opens a new perspective on the limits of top-down influences of thinking on seeing. While the input of the perceptual organization process may be modifiable and its output enrichable, the process itself seems so fast (or efficient) that thinking hardly has time to intrude and is effective mostly after the fact.

Contextual modulation of primary visual cortex by auditory signals

Early visual cortex receives non-feedforward input from lateral and top-down connections (Muckli & Petro 2013 Curr. Opin. Neurobiol.23, 195–201. (doi:10.1016/j.conb.2013.01.020)), including long-range projections from auditory areas. Early visual cortex can code for high-level auditory information, with neural patterns representing natural sound stimulation (Vetter et al. 2014 Curr. Biol.24, 1256–1262. (doi:10.1016/j.cub.2014.04.020)). We discuss a number of questions arising from these findings. What is the adaptive function of bimodal representations in visual cortex? What type of information projects from auditory to visual cortex? What are the anatomical constraints of auditory information in V1, for example, periphery versus fovea, superficial versus deep cortical layers? Is there a putative neural mechanism we can infer from human neuroimaging data and recent theoretical accounts of cortex? We also present data showing we can read out high-level auditory information from the activation patterns of early visual cortex even when visual cortex receives simple visual stimulation, suggesting independent channels for visual and auditory signals in V1. We speculate which cellular mechanisms allow V1 to be contextually modulated by auditory input to facilitate perception, cognition and behaviour. Beyond cortical feedback that facilitates perception, we argue that there is also feedback serving counterfactual processing during imagery, dreaming and mind wandering, which is not relevant for immediate perception but for behaviour and cognition over a longer time frame.

This article is part of the themed issue ‘Auditory and visual scene analysis’.

Local expectation violations result in global activity gain in primary visual cortex

During natural perception, we often form expectations about upcoming input. These expectations are usually multifaceted - we expect a particular object at a particular location. However, expectations about spatial location and stimulus features have mostly been studied in isolation, and it is unclear whether feature-based expectation can be spatially specific. Interestingly, feature-based attention automatically spreads to unattended locations. It is still an open question whether the neural mechanisms underlying feature-based expectation differ from those underlying feature-based attention. Therefore, establishing whether the effects of feature-based expectation are spatially specific may inform this debate. Here, we investigated this by inducing expectations of a specific stimulus feature at a specific location, and probing the effects on sensory processing across the visual field using fMRI. We found an enhanced sensory response for unexpected stimuli, which was elicited only when there was a violation of expectation at the specific location where participants formed a stimulus expectation. The neural consequences of this expectation violation, however, spread to cortical locations processing the stimulus in the opposite hemifield. This suggests that an expectation violation at one location in the visual world can lead to a spatially non-specific gain increase across the visual field.

Perceptual suppression of predicted natural images

Perception is shaped not only by current sensory inputs but also by expectations generated from past sensory experience. Humans viewing ambiguous stimuli in a stable visual environment are generally more likely to see the perceptual interpretation that matches their expectations, but it is less clear how expectations affect perception when the environment is changing predictably. We used statistical learning to teach observers arbitrary sequences of natural images and employed binocular rivalry to measure perceptual selection as a function of predictive context. In contrast to previous demonstrations of preferential selection of predicted images for conscious awareness, we found that recently acquired sequence predictions biased perceptual selection toward unexpected natural images and image categories. These perceptual biases were not associated with explicit recall of the learned image sequences. Our results show that exposure to arbitrary sequential structure in the environment impacts subsequent visual perceptual selection and awareness. Specifically, for natural image sequences, the visual system prioritizes what is surprising, or statistically informative, over what is expected, or statistically likely.

Single-trial prediction of reaction time variability from MEG brain activity

Neural activity prior to movement onset contains essential information for predictive assistance for humans using brain-machine-interfaces (BMIs). Even though previous studies successfully predicted different goals for upcoming movements, it is unclear whether non-invasive recording signals contain the information to predict trial-by-trial behavioral variability under the same movement. In this paper, we examined the predictability of subsequent short or long reaction times (RTs) from magnetoencephalography (MEG) signals in a delayed-reach task. The difference in RTs was classified significantly above chance from 550 ms before the go-signal onset using the cortical currents in the premotor cortex. Significantly above-chance classification was performed in the lateral prefrontal and the right inferior parietal cortices at the late stage of the delay period. Thus, inter-trial variability in RTs is predictable information. Our study provides a proof-of-concept of the future development of non-invasive BMIs to prevent delayed movements.

Linking pattern completion in the hippocampus to predictive coding in visual cortex

Models of predictive coding frame perception as a generative process in which expectations constrain sensory representations. These models account for expectations about how a stimulus will move or change from moment to moment, but do not address expectations about what other, distinct stimuli are likely to appear based on prior experience. We show that such memory-based expectations in human visual cortex are related to the hippocampal mechanism of pattern completion.

Reconstructing representations of dynamic visual objects in early visual cortex

As raw sensory data are partial, our visual system extensively fills in missing details, creating enriched percepts based on incomplete bottom-up information. Despite evidence for internally generated representations at early stages of cortical processing, it is not known whether these representations include missing information of dynamically transforming objects. Long-range apparent motion (AM) provides a unique test case because objects in AM can undergo changes both in position and in features. Using fMRI and encoding methods, we found that the "intermediate" orientation of an apparently rotating grating, never presented in the retinal input but interpolated during AM, is reconstructed in population-level, feature-selective tuning responses in the region of early visual cortex (V1) that corresponds to the retinotopic location of the AM path. This neural representation is absent when AM inducers are presented simultaneously and when AM is visually imagined. Our results demonstrate dynamic filling-in in V1 for object features that are interpolated during kinetic transformations.

Covariation between Spike and LFP Modulations Revealed with Focal and Asynchronous Stimulation of Receptive Field Surround in Monkey Primary Visual Cortex

A focal visual stimulus outside the classical receptive field (RF) of a V1 neuron does not evoke a spike response by itself, and yet evokes robust changes in the local field potential (LFP). This subthreshold LFP provides a unique opportunity to investigate how changes induced by surround stimulation leads to modulation of spike activity. In the current study, two identical Gabor stimuli were sequentially presented with a variable stimulus onset asynchrony (SOA) ranging from 0 to 100 ms: the first (S1) outside the RF and the second (S2) over the RF of primary visual cortex neurons, while trained monkeys performed a fixation task. This focal and asynchronous stimulation of the RF surround enabled us to analyze the modulation of S2-evoked spike activity and covariation between spike and LFP modulation across SOA. In this condition, the modulation of S2-evoked spike response was dominantly facilitative and was correlated with the change in LFP amplitude, which was pronounced for the cells recorded in the upper cortical layers. The time course of covariation between the SOA-dependent spike modulation and LFP amplitude suggested that the subthreshold LFP evoked by the S1 can predict the magnitude of upcoming spike modulation.

Contextual Feedback to Superficial Layers of V1

Neuronal cortical circuitry comprises feedforward, lateral, and feedback projections, each of which terminates in distinct cortical layers [1–3]. In sensory systems, feedforward processing transmits signals from the external world into the cortex, whereas feedback pathways signal the brain’s inference of the world [4–11]. However, the integration of feedforward, lateral, and feedback inputs within each cortical area impedes the investigation of feedback, and to date, no technique has isolated the feedback of visual scene information in distinct layers of healthy human cortex. We masked feedforward input to a region of V1 cortex and studied the remaining internal processing. Using high-resolution functional brain imaging (0.8 mm³) and multivoxel pattern information techniques, we demonstrate that during normal visual stimulation scene information peaks in mid-layers. Conversely, we found that contextual feedback information peaks in outer, superficial layers. Further, we found that shifting the position of the visual scene surrounding the mask parametrically modulates feedback in superficial layers of V1. Our results reveal the layered cortical organization of external versus internal visual processing streams during perception in healthy human subjects. We provide empirical support for theoretical feedback models such as predictive coding [10, 12] and coherent infomax [13] and reveal the potential of high-resolution fMRI to access internal processing in sub-millimeter human cortex.

•

High-resolution MRI shows functional information patterns in non-stimulated V1

•

Non-stimulated V1 receives cortical feedback information to superficial layers

•

Feedback to non-stimulated V1 superficial layers is predictive of visual context

The Role of Dendritic Signaling in the Anesthetic Suppression of Consciousness

Despite considerable progress in the identification of the molecular targets of general anesthetics, it remains unclear how these drugs affect the brain at the systems level to suppress consciousness. According to recent proposals, anesthetics may achieve this feat by interfering with corticocortical top–down processes, that is, by interrupting information flow from association to early sensory cortices. Such a view entails two immediate questions. First, at which anatomical site, and by virtue of which physiological mechanism, do anesthetics interfere with top–down signals? Second, why does a breakdown of top–down signaling cause unconsciousness? While an answer to the first question can be gleaned from emerging neurophysiological evidence on dendritic signaling in cortical pyramidal neurons, a response to the second is offered by increasingly popular theoretical frameworks that place the element of prediction at the heart of conscious perception.

Stable Task Representations under Attentional Load Revealed with Multivariate Pattern Analysis of Human Brain Activity

Performing multiple tasks concurrently places a load on limited attentional resources and results in disrupted task performance. Although human neuroimaging studies have investigated the neural correlates of attentional load, how attentional load affects task processing is poorly understood. Here, task-related neural activity was investigated using fMRI with conventional univariate analysis and multivariate pattern analysis (MVPA) while participants performed blocks of prosaccades and antisaccades, either with or without a rapid serial visual presentation (RSVP) task. Performing prosaccades and antisaccades with RSVP increased error rates and RTs, decreased mean activation in frontoparietal brain areas associated with oculomotor control, and eliminated differences in activation between prosaccades and antisaccades. However, task identity could be decoded from spatial patterns of activation both in the absence and presence of an attentional load. Furthermore, in the FEFs and intraparietal sulcus, these spatial representations were found to be similar using cross-trial-type MVPA, which suggests stability under attentional load. These results demonstrate that attentional load may disrupt the strength of task-related neural activity, rather than the identity of task representations.

Decoding the future from past experience: learning shapes predictions in early visual cortex

Learning the structure of the environment is critical for interpreting the current scene and predicting upcoming events. However, the brain mechanisms that support our ability to translate knowledge about scene statistics to sensory predictions remain largely unknown. Here we provide evidence that learning of temporal regularities shapes representations in early visual cortex that relate to our ability to predict sensory events. We tested the participants' ability to predict the orientation of a test stimulus after exposure to sequences of leftward- or rightward-oriented gratings. Using fMRI decoding, we identified brain patterns related to the observers' visual predictions rather than stimulus-driven activity. Decoding of predicted orientations following structured sequences was enhanced after training, while decoding of cued orientations following exposure to random sequences did not change. These predictive representations appear to be driven by the same large-scale neural populations that encode actual stimulus orientation and to be specific to the learned sequence structure. Thus our findings provide evidence that learning temporal structures supports our ability to predict future events by reactivating selective sensory representations as early as in primary visual cortex.

Contributions of cortical feedback to sensory processing in primary visual cortex

Closing the structure-function divide is more challenging in the brain than in any other organ (Lichtman and Denk, 2011). For example, in early visual cortex, feedback projections to V1 can be quantified (e.g., Budd, 1998) but the understanding of feedback function is comparatively rudimentary (Muckli and Petro, 2013). Focusing on the function of feedback, we discuss how textbook descriptions mask the complexity of V1 responses, and how feedback and local activity reflects not only sensory processing but internal brain states.

Face processing regions are sensitive to distinct aspects of temporal sequence in facial dynamics

Facial movement conveys important information for social interactions, yet its neural processing is poorly understood. Computational models propose that shape- and temporal sequence sensitive mechanisms interact in processing dynamic faces. While face processing regions are known to respond to facial movement, their sensitivity to particular temporal sequences has barely been studied. Here we used fMRI to examine the sensitivity of human face-processing regions to two aspects of directionality in facial movement trajectories. We presented genuine movie recordings of increasing and decreasing fear expressions, each of which were played in natural or reversed frame order. This two-by-two factorial design matched low-level visual properties, static content and motion energy within each factor, emotion-direction (increasing or decreasing emotion) and timeline (natural versus artificial). The results showed sensitivity for emotion-direction in FFA, which was timeline-dependent as it only occurred within the natural frame order, and sensitivity to timeline in the STS, which was emotion-direction-dependent as it only occurred for decreased fear. The occipital face area (OFA) was sensitive to the factor timeline. These findings reveal interacting temporal sequence sensitive mechanisms that are responsive to both ecological meaning and to prototypical unfolding of facial dynamics. These mechanisms are temporally directional, provide socially relevant information regarding emotional state or naturalness of behavior, and agree with predictions from modeling and predictive coding theory.

Decoding sound and imagery content in early visual cortex

Human early visual cortex was traditionally thought to process simple visual features such as orientation, contrast, and spatial frequency via feedforward input from the lateral geniculate nucleus (e.g., [1]). However, the role of nonretinal influence on early visual cortex is so far insufficiently investigated despite much evidence that feedback connections greatly outnumber feedforward connections [2–5]. Here, we explored in five fMRI experiments how information originating from audition and imagery affects the brain activity patterns in early visual cortex in the absence of any feedforward visual stimulation. We show that category-specific information from both complex natural sounds and imagery can be read out from early visual cortex activity in blindfolded participants. The coding of nonretinal information in the activity patterns of early visual cortex is common across actual auditory perception and imagery and may be mediated by higher-level multisensory areas. Furthermore, this coding is robust to mild manipulations of attention and working memory but affected by orthogonal, cognitively demanding visuospatial processing. Crucially, the information fed down to early visual cortex is category specific and generalizes to sound exemplars of the same category, providing evidence for abstract information feedback rather than precise pictorial feedback. Our results suggest that early visual cortex receives nonretinal input from other brain areas when it is generated by auditory perception and/or imagery, and this input carries common abstract information. Our findings are compatible with feedback of predictive information to the earliest visual input level (e.g., [6]), in line with predictive coding models [7–10].

•

Early visual cortex receives nonretinal input carrying abstract information

•

Both auditory perception and imagery generate consistent top-down input

•

Information feedback may be mediated by multisensory areas

•

Feedback is robust to attentional, but not visuospatial, manipulation

Visual space and object space in the cerebral cortex of retinal disease patients

The lower areas of the hierarchically organized visual cortex are strongly retinotopically organized, with strong responses to specific retinotopic stimuli, and no response to other stimuli outside these preferred regions. Higher areas in the ventral occipitotemporal cortex show a weak eccentricity bias, and are mainly sensitive for object category (e.g., faces versus buildings). This study investigated how the mapping of eccentricity and category sensitivity using functional magnetic resonance imaging is affected by a retinal lesion in two very different low vision patients: a patient with a large central scotoma, affecting central input to the retina (juvenile macular degeneration), and a patient where input to the peripheral retina is lost (retinitis pigmentosa). From the retinal degeneration, we can predict specific losses of retinotopic activation. These predictions were confirmed when comparing stimulus activations with a no-stimulus fixation baseline. At the same time, however, seemingly contradictory patterns of activation, unexpected given the retinal degeneration, were observed when different stimulus conditions were directly compared. These unexpected activations were due to position-specific deactivations, indicating the importance of investigating absolute activation (relative to a no-stimulus baseline) rather than relative activation (comparing different stimulus conditions). Data from two controls, with simulated scotomas that matched the lesions in the two patients also showed that retinotopic mapping results could be explained by a combination of activations at the stimulated locations and deactivations at unstimulated locations. Category sensitivity was preserved in the two patients. In sum, when we take into account the full pattern of activations and deactivations elicited in retinotopic cortex and throughout the ventral object vision pathway in low vision patients, the pattern of (de)activation is consistent with the retinal loss.

Robust single trial identification of conscious percepts triggered by sensory events of variable saliency

The neural correlates of visual awareness are elusive because of its fleeting nature. Here we have addressed this issue by using single trial statistical "brain reading" of neurophysiological event related (ERP) signatures of conscious perception of visual attributes with different levels of saliency. Behavioral reports were taken at every trial in 4 experiments addressing conscious access to color, luminance, and local phase offset cues. We found that single trial neurophysiological signatures of target presence can be observed around 300 ms at central parietal sites. Such signatures are significantly related with conscious perception, and their probability is related to sensory saliency levels. These findings identify a general neural correlate of conscious perception at the single trial level, since conscious perception can be decoded as such independently of stimulus salience and fluctuations of threshold levels. This approach can be generalized to successfully detect target presence in other individuals.