Dynamic representation of partially occluded objects in primate prefrontal and visual cortex

Amodal completion across the brain: The impact of structure and knowledge

This study investigates the phenomenon of amodal completion within the context of naturalistic objects, employing a repetition suppression paradigm to disentangle the influence of structure and knowledge cues on how objects are completed. The research focuses on early visual cortex (EVC) and lateral occipital complex (LOC), shedding light on how these brain regions respond to different completion scenarios. In LOC, we observed suppressed responses to structure and knowledge-compatible stimuli, providing evidence that both cues influence neural processing in higher-level visual areas. However, in EVC, we did not find evidence for differential responses to completions compatible or incompatible with either structural or knowledge-based expectations. Together, our findings suggest that the interplay between structure and knowledge cues in amodal completion predominantly impacts higher-level visual processing, with less pronounced effects on the early visual cortex. This study contributes to our understanding of the complex mechanisms underlying visual perception and highlights the distinct roles played by different brain regions in amodal completion.

Understanding the role of pathways in a deep neural network

Deep neural networks have demonstrated superior performance in artificial intelligence applications, but the opaqueness of their inner working mechanism is one major drawback in their application. The prevailing unit-based interpretation is a statistical observation of stimulus-response data, which fails to show a detailed internal process of inherent mechanisms of neural networks. In this work, we analyze a convolutional neural network (CNN) trained in the classification task and present an algorithm to extract the diffusion pathways of individual pixels to identify the locations of pixels in an input image associated with object classes. The pathways allow us to test the causal components which are important for classification and the pathway-based representations are clearly distinguishable between categories. We find that the few largest pathways of an individual pixel from an image tend to cross the feature maps in each layer that is important for classification. And the large pathways of images of the same category are more consistent in their trends than those of different categories. We also apply the pathways to understanding adversarial attacks, object completion, and movement perception. Further, the total number of pathways on feature maps in all layers can clearly discriminate the original, deformed, and target samples.

Brain Functional Representation of Highly Occluded Object Recognition

Recognizing highly occluded objects is believed to arise from the interaction between the brain's vision and cognition-controlling areas, although supporting neuroimaging data are currently limited. To explore the neural mechanism during this activity, we conducted an occlusion object recognition experiment using functional magnetic resonance imaging (fMRI). During magnet resonance examinations, 66 subjects engaged in object recognition tasks with three different occlusion degrees. Generalized linear model (GLM) analysis showed that the activation degree of the occipital lobe (inferior occipital gyrus, middle occipital gyrus, and occipital fusiform gyrus) and dorsal anterior cingulate cortex (dACC) was related to the occlusion degree of the objects. Multivariate pattern analysis (MVPA) further unearthed a considerable surge in classification precision when dACC activation was incorporated as a feature. This suggested the combined role of dACC and the occipital lobe in occluded object recognition tasks. Moreover, psychophysiological interaction (PPI) analysis disclosed that functional connectivity (FC) between the dACC and the occipital lobe was enhanced with increased occlusion, highlighting the necessity of FC between these two brain regions in effectively identifying exceedingly occluded objects. In conclusion, these findings contribute to understanding the neural mechanisms of highly occluded object recognition, augmenting our appreciation of how the brain manages incomplete visual data.

Multisensory interactions of face and vocal information during perception and memory in ventrolateral prefrontal cortex

The ventral frontal lobe is a critical node in the circuit that underlies communication, a multisensory process where sensory features of faces and vocalizations come together. The neural basis of face and vocal integration is a topic of great importance since the integration of multiple sensory signals is essential for the decisions that govern our social interactions. Investigations have shown that the macaque ventrolateral prefrontal cortex (VLPFC), a proposed homologue of the human inferior frontal gyrus, is involved in the processing, integration and remembering of audiovisual signals. Single neurons in VLPFC encode and integrate species-specific faces and corresponding vocalizations. During working memory, VLPFC neurons maintain face and vocal information online and exhibit selective activity for face and vocal stimuli. Population analyses indicate that identity, a critical feature of social stimuli, is encoded by VLPFC neurons and dictates the structure of dynamic population activity in the VLPFC during the perception of vocalizations and their corresponding facial expressions. These studies suggest that VLPFC may play a primary role in integrating face and vocal stimuli with contextual information, in order to support decision making during social communication. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Spatiotemporal neural dynamics of object recognition under uncertainty in humans

While there is a wealth of knowledge about core object recognition-our ability to recognize clear, high-contrast object images-how the brain accomplishes object recognition tasks under increased uncertainty remains poorly understood. We investigated the spatiotemporal neural dynamics underlying object recognition under increased uncertainty by combining MEG and 7 Tesla (7T) fMRI in humans during a threshold-level object recognition task. We observed an early, parallel rise of recognition-related signals across ventral visual and frontoparietal regions that preceded the emergence of category-related information. Recognition-related signals in ventral visual regions were best explained by a two-state representational format whereby brain activity bifurcated for recognized and unrecognized images. By contrast, recognition-related signals in frontoparietal regions exhibited a reduced representational space for recognized images, yet with sharper category information. These results provide a spatiotemporally resolved view of neural activity supporting object recognition under uncertainty, revealing a pattern distinct from that underlying core object recognition.

Towards a pluralistic neurobiological understanding of consciousness

Theories of consciousness are often based on the assumption that a single, unified neurobiological account will explain different types of conscious awareness. However, recent findings show that, even within a single modality such as conscious visual perception, the anatomical location, timing, and information flow of neural activity related to conscious awareness vary depending on both external and internal factors. This suggests that the search for generic neural correlates of consciousness may not be fruitful. I argue that consciousness science requires a more pluralistic approach and propose a new framework: joint determinant theory (JDT). This theory may be capable of accommodating different brain circuit mechanisms for conscious contents as varied as percepts, wills, memories, emotions, and thoughts, as well as their integrated experience.

Frontal-to-visual information flow explains predictive motion tracking

Predictive tracking demonstrates our ability to maintain a line of vision on moving objects even when they temporarily disappear. Models of smooth pursuit eye movements posit that our brain achieves this ability by directly streamlining motor programming from continuously updated sensory motion information. To test this hypothesis, we obtained sensory motion representation from multivariate electroencephalogram activity while human participants covertly tracked a temporarily occluded moving stimulus with their eyes remaining stationary at the fixation point. The sensory motion representation of the occluded target evolves to its maximum strength at the expected timing of reappearance, suggesting a timely modulation of the internal model of the visual target. We further characterize the spatiotemporal dynamics of the task-relevant motion information by computing the phase gradients of slow oscillations. We discovered a predominant posterior-to-anterior phase gradient immediately after stimulus occlusion; however, at the expected timing of reappearance, the axis reverses the gradient, becoming anterior-to-posterior. The behavioral bias of smooth pursuit eye movements, which is a signature of the predictive process of the pursuit, was correlated with the posterior division of the gradient. These results suggest that the sensory motion area modulated by the prediction signal is involved in updating motor programming.

Dynamic attention signalling in V4: Relation to fast-spiking/non-fast-spiking cell class and population coupling

The computational role of a neuron during attention depends on its firing properties, neurotransmitter expression and functional connectivity. Neurons in the visual cortical area V4 are reliably engaged by selective attention but exhibit diversity in the effect of attention on firing rates and correlated variability. It remains unclear what specific neuronal properties shape these attention effects. In this study, we quantitatively characterised the distribution of attention modulation of firing rates across populations of V4 neurons. Neurons exhibited a continuum of time-varying attention effects. At one end of the continuum, neurons' spontaneous firing rates were slightly depressed with attention (compared to when unattended), whereas their stimulus responses were enhanced with attention. The other end of the continuum showed the converse pattern: attention depressed stimulus responses but increased spontaneous activity. We tested whether the particular pattern of time-varying attention effects that a neuron exhibited was related to the shape of their actions potentials (so-called 'fast-spiking' [FS] neurons have been linked to inhibition) and the strength of their coupling to the overall population. We found an interdependence among neural attention effects, neuron type and population coupling. In particular, we found neurons for which attention enhanced spontaneous activity but suppressed stimulus responses were less likely to be fast-spiking (more likely to be non-fast-spiking) and tended to have stronger population coupling, compared to neurons with other types of attention effects. These results add important information to our understanding of visual attention circuits at the cellular level.

Task Context Modulates Feature-Selective Responses in Area V4

Feature selectivity of visual cortical responses measured during passive fixation provides only a partial view of selectivity because it does not account for the influence of cognitive factors. Here we focus on primate area V4 and ask how neuronal tuning is modulated by task engagement. We investigated whether responses to colored shapes during active shape discrimination are simple, stimulus-agnostic, scaled versions of responses during passive fixation, akin to results from attentional studies. Alternatively, responses could be subject to stimulus-specific scaling, that is, responses to different stimuli are modulated differently, resulting in changes in underlying shape/color selectivity. Among 83 well-isolated V4 neurons in two male macaques, only a minority (16 of 83), which were weakly tuned to both shape and color, displayed responses during fixation and discrimination tasks that could be related by stimulus-agnostic scaling. The majority (67 of 83), which were strongly tuned to shape, color, or both, displayed stimulus-dependent response changes during discrimination. For some of these neurons (39 of 83), the shape or color of the stimulus dictated the magnitude of the change, and for others (28 of 83) it was the combination of stimulus shape and color. Importantly, for neurons with one strong and one weak tuning dimension, stimulus-dependent response changes during discrimination were associated with a relative increase in selectivity along the stronger tuning dimension, without changes in tuning peak. These results reveal that more strongly tuned V4 neurons may also be more flexible in their selectivity, and imbalances in selectivity are amplified during active task contexts.SIGNIFICANCE STATEMENT Tuning for stimulus features is typically characterized by recording responses during passive fixation, but cognitive factors, including attention, influence responses in visual cortex. To determine how behavioral engagement influences neuronal responses in area V4, we compared responses to colored shapes during passive fixation and active behavior. For a large fraction of neurons, differences in responses between passive fixation and active behavior depended on the identity of the visual stimulus. For a subgroup of strongly feature-selective neurons, this response modulation was associated with enhanced selectivity for one feature at the expense of selectivity for the other. Such flexibility in tuning strength could improve performance in tasks requiring active judgment of stimuli.

Neurons in inferior temporal cortex are sensitive to motion trajectory during degraded object recognition

Our brains continuously acquire sensory information and make judgments even when visual information is limited. In some circumstances, an ambiguous object can be recognized from how it moves, such as an animal hopping or a plane flying overhead. Yet it remains unclear how movement is processed by brain areas involved in visual object recognition. Here we investigate whether inferior temporal (IT) cortex, an area known for its relevance in visual form processing, has access to motion information during recognition. We developed a matching task that required monkeys to recognize moving shapes with variable levels of shape degradation. Neural recordings in area IT showed that, surprisingly, some IT neurons responded stronger to degraded shapes than clear ones. Furthermore, neurons exhibited motion sensitivity at different times during the presentation of the blurry target. Population decoding analyses showed that motion patterns could be decoded from IT neuron pseudo-populations. Contrary to previous findings, these results suggest that neurons in IT can integrate visual motion and shape information, particularly when shape information is degraded, in a way that has been previously overlooked. Our results highlight the importance of using challenging multifeature recognition tasks to understand the role of area IT in naturalistic visual object recognition.

Everyday Activities

The ease with which humans usually perform everyday activities masks their inherit complexity. Tasks such as setting a table prior to a meal or preparing a hot beverage require the coordination of several cognitive abilities. At the same time, many everyday activities are simple enough to afford investigation in controlled lab settings. One main goal of this issue is to raise awareness of everyday activities as a topic and a field of study in its own right, which allows investigating (a) selected cognitive abilities with high ecological validity and (b) the interplay and integration of key cognitive abilities. To this end, this topic consists of eight papers that span different aspects of everyday activities, ranging from neuroscience through philosophical considerations and implications to lessons from robotics.

Recurrent processing improves occluded object recognition and gives rise to perceptual hysteresis

Over the past decades, object recognition has been predominantly studied and modelled as a feedforward process. This notion was supported by the fast response times in psychophysical and neurophysiological experiments and the recent success of deep feedforward neural networks for object recognition. Recently, however, this prevalent view has shifted and recurrent connectivity in the brain is now believed to contribute significantly to object recognition — especially under challenging conditions, including the recognition of partially occluded objects. Moreover, recurrent dynamics might be the key to understanding perceptual phenomena such as perceptual hysteresis. In this work we investigate if and how artificial neural networks can benefit from recurrent connections. We systematically compare architectures comprised of bottom-up, lateral, and top-down connections. To evaluate the impact of recurrent connections for occluded object recognition, we introduce three stereoscopic occluded object datasets, which span the range from classifying partially occluded hand-written digits to recognizing three-dimensional objects. We find that recurrent architectures perform significantly better than parameter-matched feedforward models. An analysis of the hidden representation of the models suggests that occluders are progressively discounted in later time steps of processing. We demonstrate that feedback can correct the initial misclassifications over time and that the recurrent dynamics lead to perceptual hysteresis. Overall, our results emphasize the importance of recurrent feedback for object recognition in difficult situations.

A Stable Population Code for Attention in Prefrontal Cortex Leads a Dynamic Attention Code in Visual Cortex

Attention often requires maintaining a stable mental state over time while simultaneously improving perceptual sensitivity. These requirements place conflicting demands on neural populations, as sensitivity implies a robust response to perturbation by incoming stimuli, which is antithetical to stability. Functional specialization of cortical areas provides one potential mechanism to resolve this conflict. We reasoned that attention signals in executive control areas might be highly stable over time, reflecting maintenance of the cognitive state, thereby freeing up sensory areas to be more sensitive to sensory input (i.e., unstable), which would be reflected by more dynamic attention signals in those areas. To test these predictions, we simultaneously recorded neural populations in prefrontal cortex (PFC) and visual cortical area V4 in rhesus macaque monkeys performing an endogenous spatial selective attention task. Using a decoding approach, we found that the neural code for attention states in PFC was substantially more stable over time compared with the attention code in V4 on a moment-by-moment basis, in line with our guiding thesis. Moreover, attention signals in PFC predicted the future attention state of V4 better than vice versa, consistent with a top-down role for PFC in attention. These results suggest a functional specialization of attention mechanisms across cortical areas with a division of labor. PFC signals the cognitive state and maintains this state stably over time, whereas V4 responds to sensory input in a manner dynamically modulated by that cognitive state.SIGNIFICANCE STATEMENT Attention requires maintaining a stable mental state while simultaneously improving perceptual sensitivity. We hypothesized that these two demands (stability and sensitivity) are distributed between prefrontal and visual cortical areas, respectively. Specifically, we predicted attention signals in visual cortex would be less stable than in prefrontal cortex, and furthermore prefrontal cortical signals would predict attention signals in visual cortex in line with the hypothesized role of prefrontal cortex in top-down executive control. Our results are consistent with suggestions deriving from previous work using separate recordings in the two brain areas in different animals performing different tasks and represent the first direct evidence in support of this hypothesis with simultaneous multiarea recordings within individual animals.

The visual prefrontal cortex of anthropoids: interaction with temporal cortex in decision making and its role in the making of "visual animals"

The ventral prefrontal cortex (PFC) of primates-a region strongly implicated in decision making-receives highly processed visual sensory inputs from the inferior temporal cortex (ITC) and perirhinal cortex (PRC) and can therefore be considered visual PFC. Usually, the functions of temporal cortex and visual PFC have been discussed in separate literatures. By considering them together, we aim to clarify the ways in which fronto-temporal networks guide decision making. After discussing the ways in which visual PFC interacts with temporal cortex to promote decision making, we offer specific predictions about the selective roles of the ITC- and PRC-based fronto-temporal networks. Finally, we suggest that an increased reliance on visual PFC in anthropoid primates led to our emergence as 'visual' animals.

Encoding of Partially Occluded and Occluding Objects in Primate Inferior Temporal Cortex

Object segmentation-the process of parsing visual scenes-is essential for object recognition and scene understanding. We investigated how responses of neurons in macaque inferior temporal (IT) cortex contribute to object segmentation under partial occlusion. Specifically, we asked whether IT responses to occluding and occluded objects are bound together as in the visual image or linearly separable reflecting their segmentation. We recorded the activity of 121 IT neurons while two male animals performed a shape discrimination task under partial occlusion. We found that for a majority (60%) of neurons, responses were enhanced by partial occlusion, but they were only weakly shape selective for the discriminanda at all levels of occlusion. Enhancement of IT responses in these neurons depended largely on the area of occlusion but only minimally on the color and shape of the occluding dots. In contrast to the above group of neurons, a sizable minority responded best to the unoccluded stimulus and showed strong selectivity for the shape of the discriminanda. In these neurons, response magnitude and shape selectivity declined with increasing levels of occlusion. Simulations revealed that the response characteristics of both classes of neurons were consistent with a model in which the responses to the occluded shape and the occluders are weighted separately and linearly combined. Overall, our results support the hypothesis that information about occluded and occluding stimuli are linearly separable and easily decodable from IT responses and that IT neurons encode a segmented representation of the visual scene.SIGNIFICANCE STATEMENT Recognizing partially occluded objects can be challenging, yet the primate visual system achieves it rapidly and effortlessly. For successful recognition in the face of occlusion, segmentation of the occluded and occluding objects is a critical first step. Using a combination of experimental data and simulations, here we demonstrate that responses of neurons in macaque IT cortex, the highest stage of the form processing pathway, reflect occluded and occluding stimuli as segmented components and are not bound together as they appear in the visual image. These results support the idea that segmentation and perception of occluded and occluding stimuli are directly mirrored in the responses of neurons in the highest form processing stages.

A View from the Top: Prefrontal Control of Object Recognition

In this issue of Neuron, Kar and DiCarlo (2021) demonstrate that feedback from ventrolateral prefrontal cortex (VLPFC) to inferotemporal cortex (IT) is required for object recognition. They show that inactivating VLPFC selectively degrades object recognition performance and population encoding of object identity in IT.

Going in circles is the way forward: the role of recurrence in visual inference

Biological visual systems exhibit abundant recurrent connectivity. State-of-the-art neural network models for visual recognition, by contrast, rely heavily or exclusively on feedforward computation. Any finite-time recurrent neural network (RNN) can be unrolled along time to yield an equivalent feedforward neural network (FNN). This important insight suggests that computational neuroscientists may not need to engage recurrent computation, and that computer-vision engineers may be limiting themselves to a special case of FNN if they build recurrent models. Here we argue, to the contrary, that FNNs are a special case of RNNs and that computational neuroscientists and engineers should engage recurrence to understand how brains and machines can (1) achieve greater and more flexible computational depth (2) compress complex computations into limited hardware (3) integrate priors and priorities into visual inference through expectation and attention (4) exploit sequential dependencies in their data for better inference and prediction and (5) leverage the power of iterative computation.

Visual Functions of Primate Area V4

Area V4-the focus of this review-is a mid-level processing stage along the ventral visual pathway of the macaque monkey. V4 is extensively interconnected with other visual cortical areas along the ventral and dorsal visual streams, with frontal cortical areas, and with several subcortical structures. Thus, it is well poised to play a broad and integrative role in visual perception and recognition-the functional domain of the ventral pathway. Neurophysiological studies in monkeys engaged in passive fixation and behavioral tasks suggest that V4 responses are dictated by tuning in a high-dimensional stimulus space defined by form, texture, color, depth, and other attributes of visual stimuli. This high-dimensional tuning may underlie the development of object-based representations in the visual cortex that are critical for tracking, recognizing, and interacting with objects. Neurophysiological and lesion studies also suggest that V4 responses are important for guiding perceptual decisions and higher-order behavior.

Mapping Cortical Integration of Sensory and Affective Pain Pathways

Pain is an integrated sensory and affective experience. Cortical mechanisms of sensory and affective integration, however, remain poorly defined. Here, we investigate the projection from the primary somatosensory cortex (S1), which encodes the sensory pain information, to the anterior cingulate cortex (ACC), a key area for processing pain affect, in freely behaving rats. By using a combination of optogenetics, in vivo electrophysiology, and machine learning analysis, we find that a subset of neurons in the ACC receives S1 inputs, and activation of the S1 axon terminals increases the response to noxious stimuli in ACC neurons. Chronic pain enhances this cortico-cortical connection, as manifested by an increased number of ACC neurons that respond to S1 inputs and the magnified contribution of these neurons to the nociceptive response in the ACC. Furthermore, modulation of this S1→ACC projection regulates aversive responses to pain. Our results thus define a cortical circuit that plays a potentially important role in integrating sensory and affective pain signals.

Beyond the feedforward sweep: feedback computations in the visual cortex

Visual perception involves the rapid formation of a coarse image representation at the onset of visual processing, which is iteratively refined by late computational processes. These early versus late time windows approximately map onto feedforward and feedback processes, respectively. State-of-the-art convolutional neural networks, the main engine behind recent machine vision successes, are feedforward architectures. Their successes and limitations provide critical information regarding which visual tasks can be solved by purely feedforward processes and which require feedback mechanisms. We provide an overview of recent work in cognitive neuroscience and machine vision that highlights the possible role of feedback processes for both visual recognition and beyond. We conclude by discussing important open questions for future research.

Specializations for reward-guided decision-making in the primate ventral prefrontal cortex

The estimated values of choices, and therefore decision-making based on those values, are influenced by both the chance that the chosen items or goods can be obtained (availability) and their current worth (desirability) as well as by the ability to link the estimated values to choices (a process sometimes called credit assignment). In primates, the prefrontal cortex (PFC) has been thought to contribute to each of these processes; however, causal relationships between particular subdivisions of the PFC and specific functions have been difficult to establish. Recent lesion-based research studies have defined the roles of two different parts of the primate PFC - the orbitofrontal cortex (OFC) and the ventral lateral frontal cortex (VLFC) - and their subdivisions in evaluating each of these factors and in mediating credit assignment during reward-based decision-making.

Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior

Non-recurrent deep convolutional neural networks (CNNs) are currently the best at modeling core object recognition, a behavior that is supported by the densely recurrent primate ventral stream, culminating in the inferior temporal (IT) cortex. If recurrence is critical to this behavior, then primates should outperform feedforward-only deep CNNs for images that require additional recurrent processing beyond the feedforward IT response. Here we first used behavioral methods to discover hundreds of these 'challenge' images. Second, using large-scale electrophysiology, we observed that behaviorally sufficient object identity solutions emerged ~30 ms later in the IT cortex for challenge images compared with primate performance-matched 'control' images. Third, these behaviorally critical late-phase IT response patterns were poorly predicted by feedforward deep CNN activations. Notably, very-deep CNNs and shallower recurrent CNNs better predicted these late IT responses, suggesting that there is a functional equivalence between additional nonlinear transformations and recurrence. Beyond arguing that recurrent circuits are critical for rapid object identification, our results provide strong constraints for future recurrent model development.

Neuroimaging Findings on Amodal Completion: A Review

Amodal completion is the phenomenon of perceiving completed objects even though physically they are partially occluded. In this review, we provide an extensive overview of the results obtained from a variety of neuroimaging studies on the neural correlates of amodal completion. We discuss whether low-level and high-level cortical areas are implicated in amodal completion; provide an overview of how amodal completion unfolds over time while dissociating feedforward, recurrent, and feedback processes; and discuss how amodal completion is represented at the neuronal level. The involvement of low-level visual areas such as V1 and V2 is not yet clear, while several high-level structures such as the lateral occipital complex and fusiform face area seem invariant to occlusion of objects and faces, respectively, and several motor areas seem to code for object permanence. The variety of results on the timing of amodal completion hints to a mixture of feedforward, recurrent, and feedback processes. We discuss whether the invisible parts of the occluded object are represented as if they were visible, contrary to a high-level representation. While plenty of questions on amodal completion remain, this review presents an overview of the neuroimaging findings reported to date, summarizes several insights from computational models, and connects research of other perceptual completion processes such as modal completion. In all, it is suggested that amodal completion is the solution to deal with various types of incomplete retinal information, and highly depends on stimulus complexity and saliency, and therefore also give rise to a variety of observed neural patterns.

Recurrent computations for visual pattern completion

The ability to complete patterns and interpret partial information is a central property of intelligence. Deep convolutional network architectures have proved successful in labeling whole objects in images and capturing the initial 150 ms of processing along the ventral visual cortex. This study shows that human object recognition abilities remain robust when only small amounts of information are available due to heavy occlusion, but the performance of bottom-up computational models is impaired under limited visibility. The results provide combined behavioral, neurophysiological, and modeling insights showing how recurrent computations may help the brain solve the fundamental challenge of pattern completion.

Making inferences from partial information constitutes a critical aspect of cognition. During visual perception, pattern completion enables recognition of poorly visible or occluded objects. We combined psychophysics, physiology, and computational models to test the hypothesis that pattern completion is implemented by recurrent computations and present three pieces of evidence that are consistent with this hypothesis. First, subjects robustly recognized objects even when they were rendered <15% visible, but recognition was largely impaired when processing was interrupted by backward masking. Second, invasive physiological responses along the human ventral cortex exhibited visually selective responses to partially visible objects that were delayed compared with whole objects, suggesting the need for additional computations. These physiological delays were correlated with the effects of backward masking. Third, state-of-the-art feed-forward computational architectures were not robust to partial visibility. However, recognition performance was recovered when the model was augmented with attractor-based recurrent connectivity. The recurrent model was able to predict which images of heavily occluded objects were easier or harder for humans to recognize, could capture the effect of introducing a backward mask on recognition behavior, and was consistent with the physiological delays along the human ventral visual stream. These results provide a strong argument of plausibility for the role of recurrent computations in making visual inferences from partial information.

Predictive Coding in Area V4: Dynamic Shape Discrimination under Partial Occlusion

The primate visual system has an exquisite ability to discriminate partially occluded shapes. Recent electrophysiological recordings suggest that response dynamics in intermediate visual cortical area V4, shaped by feedback from prefrontal cortex (PFC), may play a key role. To probe the algorithms that may underlie these findings, we build and test a model of V4 and PFC interactions based on a hierarchical predictive coding framework. We propose that probabilistic inference occurs in two steps. Initially, V4 responses are driven solely by bottom-up sensory input and are thus strongly influenced by the level of occlusion. After a delay, V4 responses combine both feedforward input and feedback signals from the PFC; the latter reflect predictions made by PFC about the visual stimulus underlying V4 activity. We find that this model captures key features of V4 and PFC dynamics observed in experiments. Specifically, PFC responses are strongest for occluded stimuli and delayed responses in V4 are less sensitive to occlusion, supporting our hypothesis that the feedback signals from PFC underlie robust discrimination of occluded shapes. Thus, our study proposes that area V4 and PFC participate in hierarchical inference, with feedback signals encoding top-down predictions about occluded shapes.