How does the brain solve visual object recognition?

The neural code for face orientation in the human fusiform face area

Humans recognize faces and objects with high speed and accuracy regardless of their orientation. Recent studies have proposed that orientation invariance in face recognition involves an intermediate representation where neural responses are similar for mirror-symmetric views. Here, we used fMRI, multivariate pattern analysis, and computational modeling to investigate the neural encoding of faces and vehicles at different rotational angles. Corroborating previous studies, we demonstrate a representation of face orientation in the fusiform face-selective area (FFA). We go beyond these studies by showing that this representation is category-selective and tolerant to retinal translation. Critically, by controlling for low-level confounds, we found the representation of orientation in FFA to be compatible with a linear angle code. Aspects of mirror-symmetric coding cannot be ruled out when FFA mean activity levels are considered as a dimension of coding. Finally, we used a parametric family of computational models, involving a biased sampling of view-tuned neuronal clusters, to compare different face angle encoding models. The best fitting model exhibited a predominance of neuronal clusters tuned to frontal views of faces. In sum, our findings suggest a category-selective and monotonic code of face orientation in the human FFA, in line with primate electrophysiology studies that observed mirror-symmetric tuning of neural responses at higher stages of the visual system, beyond the putative homolog of human FFA.

Distributed processing of color and form in the visual cortex

To what extent does the visual system process color and form separately? Proponents of the segregation view claim that distinct regions of the cortex are dedicated to each of these two dimensions separately. However, evidence is accumulating that color and form processing may, at least to some extent, be intertwined in the brain. In this perspective, we review psychophysical and neurophysiological studies on color and form perception and evaluate their results in light of recent developments in population coding.

Categorical membership modulates crowding: evidence from characters

Visual crowding is generally thought to affect recognition mostly or only at the level of feature combination. Calling this assertion into question, recent studies have shown that if a target object and its flankers belong to different categories crowding is weaker than if they belong to the same category. Nevertheless, these results can be explained in terms of featural differences between categories. The current study tests if category-level (i.e., high-level) interference in crowding occurs when featural differences are controlled for. First, replicating previous results, we found lower critical spacing for targets and flankers belonging to different categories. Second, we observed the same, albeit weaker, category-specific effect when objects in both categories had the exact same feature set, suggesting that category-specific effects persist even when featural differences are fully controlled for. Third, we manipulated the semantic content of the flankers while keeping their feature set constant, by using upright or rotated objects, and found that meaning modulated crowding. An exclusively feature-based account of crowding would predict no differences due to such changes in meaning. We conclude that crowding results from not only the well-documented feature-level interactions but also additional interactions at a level where objects are grouped by meaning.

Deep learning in neural networks: an overview

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

What can fish brains tell us about visual perception?

Fish are a complex taxonomic group, whose diversity and distance from other vertebrates well suits the comparative investigation of brain and behavior: in fish species we observe substantial differences with respect to the telencephalic organization of other vertebrates and an astonishing variety in the development and complexity of pallial structures. We will concentrate on the contribution of research on fish behavioral biology for the understanding of the evolution of the visual system. We shall review evidence concerning perceptual effects that reflect fundamental principles of the visual system functioning, highlighting the similarities and differences between distant fish groups and with other vertebrates. We will focus on perceptual effects reflecting some of the main tasks that the visual system must attain. In particular, we will deal with subjective contours and optical illusions, invariance effects, second order motion and biological motion and, finally, perceptual binding of object properties in a unified higher level representation.

Topological sparse learning of dynamic form patterns

Motion is a crucial source of information for a variety of tasks in social interactions. The process of how humans recognize complex articulated movements such as gestures or face expressions remains largely unclear. There is an ongoing discussion if and how explicit low-level motion information, such as optical flow, is involved in the recognition process. Motivated by this discussion, we introduce a computational model that classifies the spatial configuration of gradient and optical flow patterns. The patterns are learned with an unsupervised learning algorithm based on translation-invariant nonnegative sparse coding called VNMF that extracts prototypical optical flow patterns shaped, for example, as moving heads or limb parts. A key element of the proposed system is a lateral inhibition term that suppresses activations of competing patterns in the learning process, leading to a low number of dominant and topological sparse activations. We analyze the classification performance of the gradient and optical flow patterns on three real-world human action recognition and one face expression recognition data set. The results indicate that the recognition of human actions can be achieved by gradient patterns alone, but adding optical flow patterns increases the classification performance. The combined patterns outperform other biological-inspired models and are competitive with current computer vision approaches.

Higher brain functions served by the lowly rodent primary visual cortex

It has been more than 50 years since the first description of ocular dominance plasticity--the profound modification of primary visual cortex (V1) following temporary monocular deprivation. This discovery immediately attracted the intense interest of neurobiologists focused on the general question of how experience and deprivation modify the brain as a potential substrate for learning and memory. The pace of discovery has quickened considerably in recent years as mice have become the preferred species to study visual cortical plasticity, and new studies have overturned the dogma that primary sensory cortex is immutable after a developmental critical period. Recent work has shown that, in addition to ocular dominance plasticity, adult visual cortex exhibits several forms of response modification previously considered the exclusive province of higher cortical areas. These "higher brain functions" include neural reports of stimulus familiarity, reward-timing prediction, and spatiotemporal sequence learning. Primary visual cortex can no longer be viewed as a simple visual feature detector with static properties determined during early development. Rodent V1 is a rich and dynamic cortical area in which functions normally associated only with "higher" brain regions can be studied at the mechanistic level.

Facilitated detection of social cues conveyed by familiar faces

Recognition of the identity of familiar faces in conditions with poor visibility or over large changes in head angle, lighting and partial occlusion is far more accurate than recognition of unfamiliar faces in similar conditions. Here we used a visual search paradigm to test if one class of social cues transmitted by faces—direction of another's attention as conveyed by gaze direction and head orientation—is perceived more rapidly in personally familiar faces than in unfamiliar faces. We found a strong effect of familiarity on the detection of these social cues, suggesting that the times to process these signals in familiar faces are markedly faster than the corresponding processing times for unfamiliar faces. In the light of these new data, hypotheses on the organization of the visual system for processing faces are formulated and discussed.

Finding and recognizing objects in natural scenes: complementary computations in the dorsal and ventral visual systems

Searching for and recognizing objects in complex natural scenes is implemented by multiple saccades until the eyes reach within the reduced receptive field sizes of inferior temporal cortex (IT) neurons. We analyze and model how the dorsal and ventral visual streams both contribute to this. Saliency detection in the dorsal visual system including area LIP is modeled by graph-based visual saliency, and allows the eyes to fixate potential objects within several degrees. Visual information at the fixated location subtending approximately 9° corresponding to the receptive fields of IT neurons is then passed through a four layer hierarchical model of the ventral cortical visual system, VisNet. We show that VisNet can be trained using a synaptic modification rule with a short-term memory trace of recent neuronal activity to capture both the required view and translation invariances to allow in the model approximately 90% correct object recognition for 4 objects shown in any view across a range of 135° anywhere in a scene. The model was able to generalize correctly within the four trained views and the 25 trained translations. This approach analyses the principles by which complementary computations in the dorsal and ventral visual cortical streams enable objects to be located and recognized in complex natural scenes.

EEG source connectivity analysis: from dense array recordings to brain networks

The recent past years have seen a noticeable increase of interest for electroencephalography (EEG) to analyze functional connectivity through brain sources reconstructed from scalp signals. Although considerable advances have been done both on the recording and analysis of EEG signals, a number of methodological questions are still open regarding the optimal way to process the data in order to identify brain networks. In this paper, we analyze the impact of three factors that intervene in this processing: i) the number of scalp electrodes, ii) the combination between the algorithm used to solve the EEG inverse problem and the algorithm used to measure the functional connectivity and iii) the frequency bands retained to estimate the functional connectivity among neocortical sources. Using High-Resolution (hr) EEG recordings in healthy volunteers, we evaluated these factors on evoked responses during picture recognition and naming task. The main reason for selection this task is that a solid literature background is available about involved brain networks (ground truth). From this a priori information, we propose a performance criterion based on the number of connections identified in the regions of interest (ROI) that belong to potentially activated networks. Our results show that the three studied factors have a dramatic impact on the final result (the identified network in the source space) as strong discrepancies were evidenced depending on the methods used. They also suggest that the combination of weighted Minimum Norm Estimator (wMNE) and the Phase Synchronization (PS) methods applied on High-Resolution EEG in beta/gamma bands provides the best performance in term of topological distance between the identified network and the expected network in the above-mentioned cognitive task.

Curvature-processing network in macaque visual cortex

Our visual environment abounds with curved features. Thus, the goal of understanding visual processing should include the processing of curved features. Using functional magnetic resonance imaging in behaving monkeys, we demonstrated a network of cortical areas selective for the processing of curved features. This network includes three distinct hierarchically organized regions within the ventral visual pathway: a posterior curvature-biased patch (PCP) located in the near-foveal representation of dorsal V4, a middle curvature-biased patch (MCP) located on the ventral lip of the posterior superior temporal sulcus (STS) in area TEO, and an anterior curvature-biased patch (ACP) located just below the STS in anterior area TE. Our results further indicate that the processing of curvature becomes increasingly complex from PCP to ACP. The proximity of the curvature-processing network to the well-known face-processing network suggests a possible functional link between them.

The role of object categories in hybrid visual and memory search

In hybrid search, observers search for any of several possible targets in a visual display containing distracting items and, perhaps, a target. Wolfe (2012) found that response times (RTs) in such tasks increased linearly with increases in the number of items in the display. However, RT increased linearly with the log of the number of items in the memory set. In earlier work, all items in the memory set were unique instances (e.g., this apple in this pose). Typical real-world tasks involve more broadly defined sets of stimuli (e.g., any "apple" or, perhaps, "fruit"). The present experiments show how sets or categories of targets are handled in joint visual and memory search. In Experiment 1, searching for a digit among letters was not like searching for targets from a 10-item memory set, though searching for targets from an N-item memory set of arbitrary alphanumeric characters was like searching for targets from an N-item memory set of arbitrary objects. In Experiment 2, observers searched for any instance of N sets or categories held in memory. This hybrid search was harder than search for specific objects. However, memory search remained logarithmic. Experiment 3 illustrates the interaction of visual guidance and memory search when a subset of visual stimuli are drawn from a target category. Furthermore, we outline a conceptual model, supported by our results, defining the core components that would be necessary to support such categorical hybrid searches.

A Linnaeus NG (TM) interactive key to the Lithocolletinae of North-West Europe aimed at accelerating the accumulation of reliable biodiversity data (Lepidoptera, Gracillariidae)

We present an interactive key that is available online through any web browser without the need to install any additional software, making it an easily accessible tool for the larger public. The key can be found at http://identify.naturalis.nl/lithocolletinae. The key includes all 86 North-West European Lithocolletinae, a subfamily of smaller moths (“micro-moths”) that is commonly not treated in field guides. The user can input data on several external morphological character systems in addition to distribution, host plant and even characteristics of the larval feeding traces to reach an identification. We expect that this will enable more people to contribute with reliable observation data on this group of moths and alleviate the workload of taxonomic specialists, allowing them to focus on other new keys or taxonomic work.

Dimensionality of object representations in monkey inferotemporal cortex

We have calculated the intrinsic dimensionality of visual object representations in anterior inferotemporal (AIT) cortex, based on responses of a large sample of cells stimulated with photographs of diverse objects. Because dimensionality was dependent on data set size, we determined asymptotic dimensionality as both the number of neurons and number of stimulus image approached infinity. Our final dimensionality estimate was 93 (SD: ± 11), indicating that there is basis set of approximately 100 independent features that characterize the dimensions of neural object space. We believe this is the first estimate of the dimensionality of neural visual representations based on single-cell neurophysiological data. The dimensionality of AIT object representations was much lower than the dimensionality of the stimuli. We suggest that there may be a gradual reduction in the dimensionality of object representations in neural populations going from retina to inferotemporal cortex as receptive fields become increasingly complex.

Feedforward object-vision models only tolerate small image variations compared to human

Invariant object recognition is a remarkable ability of primates' visual system that its underlying mechanism has constantly been under intense investigations. Computational modeling is a valuable tool toward understanding the processes involved in invariant object recognition. Although recent computational models have shown outstanding performances on challenging image databases, they fail to perform well in image categorization under more complex image variations. Studies have shown that making sparse representation of objects by extracting more informative visual features through a feedforward sweep can lead to higher recognition performances. Here, however, we show that when the complexity of image variations is high, even this approach results in poor performance compared to humans. To assess the performance of models and humans in invariant object recognition tasks, we built a parametrically controlled image database consisting of several object categories varied in different dimensions and levels, rendered from 3D planes. Comparing the performance of several object recognition models with human observers shows that only in low-level image variations the models perform similar to humans in categorization tasks. Furthermore, the results of our behavioral experiments demonstrate that, even under difficult experimental conditions (i.e., briefly presented masked stimuli with complex image variations), human observers performed outstandingly well, suggesting that the models are still far from resembling humans in invariant object recognition. Taken together, we suggest that learning sparse informative visual features, although desirable, is not a complete solution for future progresses in object-vision modeling. We show that this approach is not of significant help in solving the computational crux of object recognition (i.e., invariant object recognition) when the identity-preserving image variations become more complex.

Functional specialization in rat occipital and temporal visual cortex

Recent studies have revealed a surprising degree of functional specialization in rodent visual cortex. Anatomically, suggestions have been made about the existence of hierarchical pathways with similarities to the ventral and dorsal pathways in primates. Here we aimed to characterize some important functional properties in part of the supposed "ventral" pathway in rats. We investigated the functional properties along a progression of five visual areas in awake rats, from primary visual cortex (V1) over lateromedial (LM), latero-intermediate (LI), and laterolateral (LL) areas up to the newly found lateral occipito-temporal cortex (TO). Response latency increased >20 ms from areas V1/LM/LI to areas LL and TO. Orientation and direction selectivity for the used grating patterns increased gradually from V1 to TO. Overall responsiveness and selectivity to shape stimuli decreased from V1 to TO and was increasingly dependent upon shape motion. Neural similarity for shapes could be accounted for by a simple computational model in V1, but not in the other areas. Across areas, we find a gradual change in which stimulus pairs are most discriminable. Finally, tolerance to position changes increased toward TO. These findings provide unique information about possible commonalities and differences between rodents and primates in hierarchical cortical processing.

The transition in the ventral stream from feature to real-world entity representations

We propose that the ventral visual pathway of human and non-human primates is organized into three levels: (1) ventral retinotopic cortex including what is known as TEO in the monkey but corresponds to V4A and PITd/v, and the phPIT cluster in humans, (2) area TE in the monkey and its homolog LOC and neighboring fusiform regions, and more speculatively, (3) TGv in the monkey and its possible human equivalent, the temporal pole. We attribute to these levels the visual representations of features, partial real-world entities (RWEs), and known, complete RWEs, respectively. Furthermore, we propose that the middle level, TE and its homolog, is organized into three parallel substreams, lower bank STS, dorsal convexity of TE, and ventral convexity of TE, as are their corresponding human regions. These presumably process shape in depth, 2D shape and material properties, respectively, to construct RWE representations.

Early recurrent feedback facilitates visual object recognition under challenging conditions

Standard models of the visual object recognition pathway hold that a largely feedforward process from the retina through inferotemporal cortex leads to object identification. A subsequent feedback process originating in frontoparietal areas through reciprocal connections to striate cortex provides attentional support to salient or behaviorally-relevant features. Here, we review mounting evidence that feedback signals also originate within extrastriate regions and begin during the initial feedforward process. This feedback process is temporally dissociable from attention and provides important functions such as grouping, associational reinforcement, and filling-in of features. Local feedback signals operating concurrently with feedforward processing are important for object identification in noisy real-world situations, particularly when objects are partially occluded, unclear, or otherwise ambiguous. Altogether, the dissociation of early and late feedback processes presented here expands on current models of object identification, and suggests a dual role for descending feedback projections.

One month in the life of a neuron: longitudinal single-unit electrophysiology in the monkey visual system

Conventional recording methods generally preclude following the activity of the same neurons in awake animals across days. This limits our ability to systematically investigate the principles of neuronal specialization, or to study phenomena that evolve over multiple days such as experience-dependent plasticity. To redress this shortcoming, we developed a drivable, chronically implanted microwire recording preparation that allowed us to follow visual responses in inferotemporal (IT) cortex in awake behaving monkeys across multiple days, and in many cases across months. The microwire bundle and other implanted components were MRI compatible and thus permitted in the same animals both functional imaging and long-term recording from multiple neurons in deep structures within a region the approximate size of one voxel (<1 mm). The distinct patterns of stimulus selectivity observed in IT neurons, together with stable features in spike waveforms and interspike interval distributions, allowed us to track individual neurons across weeks and sometimes months. The long-term consistency of visual responses shown here permits large-scale mappings of neuronal properties using massive image libraries presented over the course of days. We demonstrate this possibility by screening the visual responses of single neurons to a set of 10,000 stimuli.

Thinking about seeing: perceptual sources of knowledge are encoded in the theory of mind brain regions of sighted and blind adults

Blind people's inferences about how other people see provide a window into fundamental questions about the human capacity to think about one another's thoughts. By working with blind individuals, we can ask both what kinds of representations people form about others' minds, and how much these representations depend on the observer having had similar mental states themselves. Thinking about others' mental states depends on a specific group of brain regions, including the right temporo-parietal junction (RTPJ). We investigated the representations of others' mental states in these brain regions, using multivoxel pattern analyses (MVPA). We found that, first, in the RTPJ of sighted adults, the pattern of neural response distinguished the source of the mental state (did the protagonist see or hear something?) but not the valence (did the protagonist feel good or bad?). Second, these neural representations were preserved in congenitally blind adults. These results suggest that the temporo-parietal junction contains explicit, abstract representations of features of others' mental states, including the perceptual source. The persistence of these representations in congenitally blind adults, who have no first-person experience with sight, provides evidence that these representations emerge even in the absence of relevant first-person perceptual experiences.

Neural mechanisms of auditory categorization: from across brain areas to within local microcircuits

Categorization enables listeners to efficiently encode and respond to auditory stimuli. Behavioral evidence for auditory categorization has been well documented across a broad range of human and non-human animal species. Moreover, neural correlates of auditory categorization have been documented in a variety of different brain regions in the ventral auditory pathway, which is thought to underlie auditory-object processing and auditory perception. Here, we review and discuss how neural representations of auditory categories are transformed across different scales of neural organization in the ventral auditory pathway: from across different brain areas to within local microcircuits. We propose different neural transformations across different scales of neural organization in auditory categorization. Along the ascending auditory system in the ventral pathway, there is a progression in the encoding of categories from simple acoustic categories to categories for abstract information. On the other hand, in local microcircuits, different classes of neurons differentially compute categorical information.

Natural variability in species-specific vocalizations constrains behavior and neural activity

A listener's capacity to discriminate between sounds is related to the amount of acoustic variability that exists between these sounds. However, a full understanding of how this natural variability impacts neural activity and behavior is lacking. Here, we tested monkeys' ability to discriminate between different utterances of vocalizations from the same acoustic class (i.e., coos and grunts), while neural activity was simultaneously recorded in the anterolateral belt region (AL) of the auditory cortex, a brain region that is a part of a pathway that mediates auditory perception. Monkeys could discriminate between coos better than they could discriminate between grunts. We also found AL activity was more informative about different coos than different grunts. This difference could be attributed, in part, to our finding that coos had more acoustic variability than grunts. Thus, intrinsic acoustic variability constrained the discriminability of AL spike trains and the ability of rhesus monkeys to discriminate between vocalizations.

Tuning and spontaneous spike time synchrony share a common structure in macaque inferior temporal cortex

Investigating the relationship between tuning and spike timing is necessary to understand how neuronal populations in anterior visual cortex process complex stimuli. Are tuning and spontaneous spike time synchrony linked by a common spatial structure (do some cells covary more strongly, even in the absence of visual stimulation?), and what is the object coding capability of this structure? Here, we recorded from spiking populations in macaque inferior temporal (IT) cortex under neurolept anesthesia. We report that, although most nearby IT neurons are weakly correlated, neurons with more similar tuning are also more synchronized during spontaneous activity. This link between tuning and synchrony was not simply due to cell separation distance. Instead, it expands on previous reports that neurons along an IT penetration are tuned to similar but slightly different features. This constraint on possible population firing rate patterns was consistent across stimulus sets, including animate vs. inanimate object categories. A classifier trained on this structure was able to generalize category "read-out" to untrained objects using only a few dimensions (a few patterns of site weightings per electrode array). We suggest that tuning and spike synchrony are linked by a common spatial structure that is highly efficient for object representation.

Short temporal asynchrony disrupts visual object recognition

Humans can recognize objects and scenes in a small fraction of a second. The cascade of signals underlying rapid recognition might be disrupted by temporally jittering different parts of complex objects. Here we investigated the time course over which shape information can be integrated to allow for recognition of complex objects. We presented fragments of object images in an asynchronous fashion and behaviorally evaluated categorization performance. We observed that visual recognition was significantly disrupted by asynchronies of approximately 30 ms, suggesting that spatiotemporal integration begins to break down with even small deviations from simultaneity. However, moderate temporal asynchrony did not completely obliterate recognition; in fact, integration of visual shape information persisted even with an asynchrony of 100 ms. We describe the data with a concise model based on the dynamic reduction of uncertainty about what image was presented. These results emphasize the importance of timing in visual processing and provide strong constraints for the development of dynamical models of visual shape recognition.

Performance-optimized hierarchical models predict neural responses in higher visual cortex

The ventral visual stream underlies key human visual object recognition abilities. However, neural encoding in the higher areas of the ventral stream remains poorly understood. Here, we describe a modeling approach that yields a quantitatively accurate model of inferior temporal (IT) cortex, the highest ventral cortical area. Using high-throughput computational techniques, we discovered that, within a class of biologically plausible hierarchical neural network models, there is a strong correlation between a model's categorization performance and its ability to predict individual IT neural unit response data. To pursue this idea, we then identified a high-performing neural network that matches human performance on a range of recognition tasks. Critically, even though we did not constrain this model to match neural data, its top output layer turns out to be highly predictive of IT spiking responses to complex naturalistic images at both the single site and population levels. Moreover, the model's intermediate layers are highly predictive of neural responses in the V4 cortex, a midlevel visual area that provides the dominant cortical input to IT. These results show that performance optimization--applied in a biologically appropriate model class--can be used to build quantitative predictive models of neural processing.

A model of subjective report and objective discrimination as categorical decisions in a vast representational space

Subliminal perception studies have shown that one can objectively discriminate a stimulus without subjectively perceiving it. We show how a minimalist framework based on Signal Detection Theory and Bayesian inference can account for this dissociation, by describing subjective and objective tasks with similar decision-theoretic mechanisms. Each of these tasks relies on distinct response classes, and therefore distinct priors and decision boundaries. As a result, they may reach different conclusions. By formalizing, within the same framework, forced-choice discrimination responses, subjective visibility reports and confidence ratings, we show that this decision model suffices to account for several classical characteristics of conscious and unconscious perception. Furthermore, the model provides a set of original predictions on the nonlinear profiles of discrimination performance obtained at various levels of visibility. We successfully test one such prediction in a novel experiment: when varying continuously the degree of perceptual ambiguity between two visual symbols presented at perceptual threshold, identification performance varies quasi-linearly when the stimulus is unseen and in an 'all-or-none' manner when it is seen. The present model highlights how conscious and non-conscious decisions may correspond to distinct categorizations of the same stimulus encoded by a high-dimensional neuronal population vector.

The temporal evolution of coarse location coding of objects: evidence for feedback

Objects occupy space. How does the brain represent the spatial location of objects? Retinotopic early visual cortex has precise location information but can only segment simple objects. On the other hand, higher visual areas can resolve complex objects but only have coarse location information. Thus coarse location of complex objects might be represented by either (a) feedback from higher areas to early retinotopic areas or (b) coarse position encoding in higher areas. We tested these alternatives by presenting various kinds of first- (edge-defined) and second-order (texture) objects. We applied multivariate classifiers to the pattern of EEG amplitudes across the scalp at a range of time points to trace the temporal dynamics of coarse location representation. For edge-defined objects, peak classification performance was high and early and thus attributable to the retinotopic layout of early visual cortex. For texture objects, it was low and late. Crucially, despite these differences in peak performance and timing, training a classifier on one object and testing it on others revealed that the topography at peak performance was the same for both first- and second-order objects. That is, the same location information, encoded by early visual areas, was available for both edge-defined and texture objects at different time points. These results indicate that locations of complex objects such as textures, although not represented in the bottom-up sweep, are encoded later by neural patterns resembling the bottom-up ones. We conclude that feedback mechanisms play an important role in coarse location representation of complex objects.

The receptive field is dead. Long live the receptive field?

Advances in experimental techniques, including behavioral paradigms using rich stimuli under closed loop conditions and the interfacing of neural systems with external inputs and outputs, reveal complex dynamics in the neural code and require a revisiting of standard concepts of representation. High-throughput recording and imaging methods along with the ability to observe and control neuronal subpopulations allow increasingly detailed access to the neural circuitry that subserves neural representations and the computations they support. How do we harness theory to build biologically grounded models of complex neural function?

Perceptual expertise and top-down expectation of musical notation engages the primary visual cortex

Most theories of visual processing propose that object recognition is achieved in higher visual cortex. However, we show that category selectivity for musical notation can be observed in the first ERP component called the C1 (measured 40-60 msec after stimulus onset) with music-reading expertise. Moreover, the C1 note selectivity was observed only when the stimulus category was blocked but not when the stimulus category was randomized. Under blocking, the C1 activity for notes predicted individual music-reading ability, and behavioral judgments of musical stimuli reflected music-reading skill. Our results challenge current theories of object recognition, indicating that the primary visual cortex can be selective for musical notation within the initial feedforward sweep of activity with perceptual expertise and with a testing context that is consistent with the expertise training, such as blocking the stimulus category for music reading.

Bandwidths for the perception of head orientation decrease during childhood

Adults use the orientation of people's heads as a cue to the focus of their attention. We examined developmental changes in mechanisms underlying sensitivity to head orientation during childhood. Eight-, 10-, 12-year-olds, and adults were adapted to a frontal face view or a 20° left or right side view before judging the orientation of a face at or near frontal. After frontal adaptation, there were no age differences in judgments of head orientation. However, after adaptation to a 20° left or right side view, aftereffects were larger and sensitivity to head orientation was lower in 8- and 10-year-olds than in adults, with no difference between 12-year-olds and adults. A computational model indicates that these results can be modeled as a consequence of decreasing neural tuning bandwidths and decreasing additive internal noise during childhood, and/or as a consequence of increasing inhibition during childhood. These results provide the first evidence that neural mechanisms underlying sensitivity to head orientation undergo considerable refinement during childhood.

Learned spatiotemporal sequence recognition and prediction in primary visual cortex

Learning to recognize and predict temporal sequences is fundamental to sensory perception, and is impaired in several neuropsychiatric disorders, but little is known about where and how this occurs in the brain. We discovered that repeated presentations of a visual sequence over a course of days causes evoked response potentiation in mouse V1 that is highly specific for stimulus order and timing. Remarkably, after V1 is trained to recognize a sequence, cortical activity regenerates the full sequence even when individual stimulus elements are omitted. This novel neurophysiological report of sequence learning advances the understanding of how the brain makes “intelligent guesses” based on limited information to form visual percepts and suggests that it is possible to study the mechanistic basis of this high–level cognitive ability by studying low–level sensory systems.

Object concepts in the chemical senses

This paper examines the applicability of the object concept to the chemical senses, by evaluating them against a set of criteria for object-hood. Taste and chemesthesis do not generate objects. Their parts, perceptible from birth, never combine. Orthonasal olfaction (sniffing) presents a strong case for generating objects. Odorants have many parts yet they are perceived as wholes, this process is based on learning, and there is figure-ground segregation. While flavors are multimodal representations bound together by learning, there is no functional need for flavor objects in the mouth. Rather, food identification occurs prior to ingestion using the eye and nose, with the latter retrieving multimodal flavor objects via sniffing (e.g., sweet smelling caramel). While there are differences in object perception between vision, audition, and orthonasal olfaction, the commonalities suggest that the brain has adopted the same basic solution when faced with extracting meaning from complex stimulus arrays.

Characterizing the dynamics of mental representations: the temporal generalization method

Parsing a cognitive task into a sequence of operations is a central problem in cognitive neuroscience. We argue that a major advance is now possible owing to the application of pattern classifiers to time-resolved recordings of brain activity [electroencephalography (EEG), magnetoencephalography (MEG), or intracranial recordings]. By testing at which moment a specific mental content becomes decodable in brain activity, we can characterize the time course of cognitive codes. Most importantly, the manner in which the trained classifiers generalize across time, and from one experimental condition to another, sheds light on the temporal organization of information-processing stages. A repertoire of canonical dynamical patterns is observed across various experiments and brain regions. This method thus provides a novel way to understand how mental representations are manipulated and transformed.

Slowness and sparseness have diverging effects on complex cell learning

Following earlier studies which showed that a sparse coding principle may explain the receptive field properties of complex cells in primary visual cortex, it has been concluded that the same properties may be equally derived from a slowness principle. In contrast to this claim, we here show that slowness and sparsity drive the representations towards substantially different receptive field properties. To do so, we present complete sets of basis functions learned with slow subspace analysis (SSA) in case of natural movies as well as translations, rotations, and scalings of natural images. SSA directly parallels independent subspace analysis (ISA) with the only difference that SSA maximizes slowness instead of sparsity. We find a large discrepancy between the filter shapes learned with SSA and ISA. We argue that SSA can be understood as a generalization of the Fourier transform where the power spectrum corresponds to the maximally slow subspace energies in SSA. Finally, we investigate the trade-off between slowness and sparseness when combined in one objective function.

Task context impacts visual object processing differentially across the cortex

Perception reflects an integration of "bottom-up" (sensory-driven) and "top-down" (internally generated) signals. Although models of visual processing often emphasize the central role of feed-forward hierarchical processing, less is known about the impact of top-down signals on complex visual representations. Here, we investigated whether and how the observer's goals modulate object processing across the cortex. We examined responses elicited by a diverse set of objects under six distinct tasks, focusing on either physical (e.g., color) or conceptual properties (e.g., man-made). Critically, the same stimuli were presented in all tasks, allowing us to investigate how task impacts the neural representations of identical visual input. We found that task has an extensive and differential impact on object processing across the cortex. First, we found task-dependent representations in the ventral temporal and prefrontal cortex. In particular, although object identity could be decoded from the multivoxel response within task, there was a significant reduction in decoding across tasks. In contrast, the early visual cortex evidenced equivalent decoding within and across tasks, indicating task-independent representations. Second, task information was pervasive and present from the earliest stages of object processing. However, although the responses of the ventral temporal, prefrontal, and parietal cortex enabled decoding of both the type of task (physical/conceptual) and the specific task (e.g., color), the early visual cortex was not sensitive to type of task and could only be used to decode individual physical tasks. Thus, object processing is highly influenced by the behavioral goal of the observer, highlighting how top-down signals constrain and inform the formation of visual representations.

Algorithms of whisker-mediated touch perception

Comparison of the functional organization of sensory modalities can reveal the specialized mechanisms unique to each modality as well as processing algorithms that are common across modalities. Here we examine the rodent whisker system. The whisker's mechanical properties shape the forces transmitted to specialized receptors. The sensory and motor systems are intimately interconnected, giving rise to two forms of sensation: generative and receptive. The sensory pathway is a test bed for fundamental concepts in computation and coding: hierarchical feature detection, sparseness, adaptive representations, and population coding. The central processing of signals can be considered a sequence of filters. At the level of cortex, neurons represent object features by a coordinated population code which encompasses cells with heterogeneous properties.

Resolving human object recognition in space and time

A comprehensive picture of object processing in the human brain requires combining both spatial and temporal information about brain activity. Here we acquired human magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) responses to 92 object images. Multivariate pattern classification applied to MEG revealed the time course of object processing: whereas individual images were discriminated by visual representations early, ordinate and superordinate category levels emerged relatively late. Using representational similarity analysis, we combined human fMRI and MEG to show content-specific correspondence between early MEG responses and primary visual cortex (V1), and later MEG responses and inferior temporal (IT) cortex. We identified transient and persistent neural activities during object processing with sources in V1 and IT. Finally, we correlated human MEG signals to single-unit responses in monkey IT. Together, our findings provide an integrated space- and time-resolved view of human object categorization during the first few hundred milliseconds of vision.

Kinesthesis can make an invisible hand visible

Self-generated body movements have reliable visual consequences. This predictive association between vision and action likely underlies modulatory effects of action on visual processing. However, it is unknown whether actions can have generative effects on visual perception. We asked whether, in total darkness, self-generated body movements are sufficient to evoke normally concomitant visual perceptions. Using a deceptive experimental design, we discovered that waving one's own hand in front of one's covered eyes can cause visual sensations of motion. Conjecturing that these visual sensations arise from multisensory connectivity, we showed that grapheme-color synesthetes experience substantially stronger kinesthesis-induced visual sensations than nonsynesthetes do. Finally, we found that the perceived vividness of kinesthesis-induced visual sensations predicted participants' ability to smoothly track self-generated hand movements with their eyes in darkness, which indicates that these sensations function like typical retinally driven visual sensations. Evidently, even in the complete absence of external visual input, the brain predicts visual consequences of actions.

Reaction Time for Object Categorization Is Predicted by Representational Distance

How does the brain translate an internal representation of an object into a decision about the object's category? Recent studies have uncovered the structure of object representations in inferior temporal cortex (IT) using multivariate pattern analysis methods. These studies have shown that representations of individual object exemplars in IT occupy distinct locations in a high-dimensional activation space, with object exemplar representations clustering into distinguishable regions based on category (e.g., animate vs. inanimate objects). In this study, we hypothesized that a representational boundary between category representations in this activation space also constitutes a decision boundary for categorization. We show that behavioral RTs for categorizing objects are well described by our activation space hypothesis. Interpreted in terms of classical and contemporary models of decision-making, our results suggest that the process of settling on an internal representation of a stimulus is itself partially constitutive of decision-making for object categorization.

Central auditory neurons display flexible feature recombination functions

Recognition of natural stimuli requires a combination of selectivity and invariance. Classical neurobiological models achieve selectivity and invariance, respectively, by assigning to each cortical neuron either a computation equivalent to the logical "AND" or a computation equivalent to the logical "OR." One powerful OR-like operation is the MAX function, which computes the maximum over input activities. The MAX function is frequently employed in computer vision to achieve invariance and considered a key operation in visual cortex. Here we explore the computations for selectivity and invariance in the auditory system of a songbird, using natural stimuli. We ask two related questions: does the MAX operation exist in auditory system? Is it implemented by specialized "MAX" neurons, as assumed in vision? By analyzing responses of individual neurons to combinations of stimuli we systematically sample the space of implemented feature recombination functions. Although we frequently observe the MAX function, we show that the same neurons that implement it also readily implement other operations, including the AND-like response. We then show that sensory adaptation, a ubiquitous property of neural circuits, causes transitions between these operations in individual neurons, violating the fixed neuron-to-computation mapping posited in the state-of-the-art object-recognition models. These transitions, however, accord with predictions of neural-circuit models incorporating divisive normalization and variable polynomial nonlinearities at the spike threshold. Because these biophysical properties are not tied to a particular sensory modality but are generic, the flexible neuron-to-computation mapping demonstrated in this study in the auditory system is likely a general property.

The neural circuitry of expertise: perceptual learning and social cognition

Amongst the most significant questions we are confronted with today include the integration of the brain's micro-circuitry, our ability to build the complex social networks that underpin society and how our society impacts on our ecological environment. In trying to unravel these issues one place to begin is at the level of the individual: to consider how we accumulate information about our environment, how this information leads to decisions and how our individual decisions in turn create our social environment. While this is an enormous task, we may already have at hand many of the tools we need. This article is intended to review some of the recent results in neuro-cognitive research and show how they can be extended to two very specific and interrelated types of expertise: perceptual expertise and social cognition. These two cognitive skills span a vast range of our genetic heritage. Perceptual expertise developed very early in our evolutionary history and is a highly developed part of all mammals' cognitive ability. On the other hand social cognition is most highly developed in humans in that we are able to maintain larger and more stable long term social connections with more behaviorally diverse individuals than any other species. To illustrate these ideas I will discuss board games as a toy model of social interactions as they include many of the relevant concepts: perceptual learning, decision-making, long term planning and understanding the mental states of other people. Using techniques that have been developed in mathematical psychology, I show that we can represent some of the key features of expertise using stochastic differential equations (SDEs). Such models demonstrate how an expert's long exposure to a particular context influences the information they accumulate in order to make a decision.These processes are not confined to board games, we are all experts in our daily lives through long exposure to the many regularities of daily tasks and social contexts.

The what, where and how of auditory-object perception

The fundamental perceptual unit in hearing is the 'auditory object'. Similar to visual objects, auditory objects are the computational result of the auditory system's capacity to detect, extract, segregate and group spectrotemporal regularities in the acoustic environment; the multitude of acoustic stimuli around us together form the auditory scene. However, unlike the visual scene, resolving the component objects within the auditory scene crucially depends on their temporal structure. Neural correlates of auditory objects are found throughout the auditory system. However, neural responses do not become correlated with a listener's perceptual reports until the level of the cortex. The roles of different neural structures and the contribution of different cognitive states to the perception of auditory objects are not yet fully understood.

Theory of mind: a neural prediction problem

Predictive coding posits that neural systems make forward-looking predictions about incoming information. Neural signals contain information not about the currently perceived stimulus, but about the difference between the observed and the predicted stimulus. We propose to extend the predictive coding framework from high-level sensory processing to the more abstract domain of theory of mind; that is, to inferences about others' goals, thoughts, and personalities. We review evidence that, across brain regions, neural responses to depictions of human behavior, from biological motion to trait descriptions, exhibit a key signature of predictive coding: reduced activity to predictable stimuli. We discuss how future experiments could distinguish predictive coding from alternative explanations of this response profile. This framework may provide an important new window on the neural computations underlying theory of mind.

Interactions between visual and semantic processing during object recognition revealed by modulatory effects of age of acquisition

The age of acquisition (AoA) of objects and their names is a powerful determinant of processing speed in adulthood, with early-acquired objects being recognized and named faster than late-acquired objects. Previous research using fMRI (Ellis et al., 2006. Traces of vocabulary acquisition in the brain: evidence from covert object naming. NeuroImage 33, 958-968) found that AoA modulated the strength of BOLD responses in both occipital and left anterior temporal cortex during object naming. We used magnetoencephalography (MEG) to explore in more detail the nature of the influence of AoA on activity in those two regions. Covert object naming recruited a network within the left hemisphere that is familiar from previous research, including visual, left occipito-temporal, anterior temporal and inferior frontal regions. Region of interest (ROI) analyses found that occipital cortex generated a rapid evoked response (~75-200 ms at 0-40 Hz) that peaked at 95 ms but was not modulated by AoA. That response was followed by a complex of later occipital responses that extended from ~300 to 850 ms and were stronger to early- than late-acquired items from ~325 to 675 ms at 10-20 Hz in the induced rather than the evoked component. Left anterior temporal cortex showed an evoked response that occurred significantly later than the first occipital response (~100-400 ms at 0-10 Hz with a peak at 191 ms) and was stronger to early- than late-acquired items from ~100 to 300 ms at 2-12 Hz. A later anterior temporal response from ~550 to 1050 ms at 5-20 Hz was not modulated by AoA. The results indicate that the initial analysis of object forms in visual cortex is not influenced by AoA. A fastforward sweep of activation from occipital and left anterior temporal cortex then results in stronger activation of semantic representations for early- than late-acquired objects. Top-down re-activation of occipital cortex by semantic representations is then greater for early than late acquired objects resulting in delayed modulation of the visual response.

Wernicke's area revisited: parallel streams and word processing

Auditory word-form recognition was originally proposed by Wernicke to occur within left superior temporal gyrus (STG), later further specified to be in posterior STG. To account for clinical observations (specifically paraphasia), Wernicke proposed his sensory speech center was also essential for correcting output from frontal speech-motor regions. Recent work, in contrast, has established a role for anterior STG, part of the auditory ventral stream, in the recognition of species-specific vocalizations in nonhuman primates and word-form recognition in humans. Recent work also suggests monitoring self-produced speech and motor control are associated with posterior STG, part of the auditory dorsal stream. Working without quantitative methods or evidence of sensory cortex' hierarchical organization, Wernicke co-localized functions that today appear dissociable. "Wernicke's area" thus may be better construed as two cortical modules, an auditory word-form area (AWFA) in the auditory ventral stream and an "inner speech area" in the auditory dorsal stream.

The Convallis rule for unsupervised learning in cortical networks

The phenomenology and cellular mechanisms of cortical synaptic plasticity are becoming known in increasing detail, but the computational principles by which cortical plasticity enables the development of sensory representations are unclear. Here we describe a framework for cortical synaptic plasticity termed the "Convallis rule", mathematically derived from a principle of unsupervised learning via constrained optimization. Implementation of the rule caused a recurrent cortex-like network of simulated spiking neurons to develop rate representations of real-world speech stimuli, enabling classification by a downstream linear decoder. Applied to spike patterns used in in vitro plasticity experiments, the rule reproduced multiple results including and beyond STDP. However STDP alone produced poorer learning performance. The mathematical form of the rule is consistent with a dual coincidence detector mechanism that has been suggested by experiments in several synaptic classes of juvenile neocortex. Based on this confluence of normative, phenomenological, and mechanistic evidence, we suggest that the rule may approximate a fundamental computational principle of the neocortex.

Parallel, multi-stage processing of colors, faces and shapes in macaque inferior temporal cortex

Visual-object processing culminates in inferior temporal cortex (IT). To assess the organization of IT, we measured functional magnetic resonance imaging responses in alert monkeys to achromatic images (faces, fruit, bodies and places) and colored gratings. IT contained multiple color-biased regions, which were typically ventral to face patches and yoked to them, spaced regularly at four locations predicted by known anatomy. Color and face selectivity increased for more anterior regions, indicative of a broad hierarchical arrangement. Responses to non-face shapes were found across IT, but were stronger outside color-biased regions and face patches, consistent with multiple parallel streams. IT also contained multiple coarse eccentricity maps: face patches overlapped central representations, color-biased regions spanned mid-peripheral representations and place-biased regions overlapped peripheral representations. These results show that IT comprises parallel, multi-stage processing networks subject to one organizing principle.