Visual response properties of single neurons in the temporal pole of behaving monkeys

Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries

Cultural similarities and differences in facial expressions have been a controversial issue in the field of facial communications. A key step in addressing the debate regarding the cultural dependency of emotional expression (and perception) is to characterize the visual features of specific facial expressions in individual cultures. Here we developed an image analysis framework for this purpose using convolutional neural networks (CNNs) that through training learned visual features critical for classification. We analyzed photographs of facial expressions derived from two databases, each developed in a different country (Sweden and Japan), in which corresponding emotion labels were available. While the CNNs reached high rates of correct results that were far above chance after training with each database, they showed many misclassifications when they analyzed faces from the database that was not used for training. These results suggest that facial features useful for classifying facial expressions differed between the databases. The selectivity of computational units in the CNNs to action units (AUs) of the face varied across the facial expressions. Importantly, the AU selectivity often differed drastically between the CNNs trained with the different databases. Similarity and dissimilarity of these tuning profiles partly explained the pattern of misclassifications, suggesting that the AUs are important for characterizing the facial features and differ between the two countries. The AU tuning profiles, especially those reduced by principal component analysis, are compact summaries useful for comparisons across different databases, and thus might advance our understanding of universality vs. specificity of facial expressions across cultures.

Rapid processing of threatening faces in the amygdala of nonhuman primates: subcortical inputs and dual roles

A subcortical pathway through the superior colliculus and pulvinar has been proposed to provide the amygdala with rapid but coarse visual information about emotional faces. However, evidence for short-latency, facial expression-discriminating responses from individual amygdala neurons is lacking; even if such a response exists, how it might contribute to stimulus detection is unclear. Also, no definitive anatomical evidence is available for the assumed pathway. Here we showed that ensemble responses of amygdala neurons in monkeys carried robust information about open-mouthed, presumably threatening, faces within 50 ms after stimulus onset. This short-latency signal was not found in the visual cortex, suggesting a subcortical origin. Temporal analysis revealed that the early response contained excitatory and suppressive components. The excitatory component may be useful for sending rapid signals downstream, while the sharpening of the rising phase of later-arriving inputs (presumably from the cortex) by the suppressive component might improve the processing of facial expressions over time. Injection of a retrograde trans-synaptic tracer into the amygdala revealed presumed monosynaptic labeling in the pulvinar and disynaptic labeling in the superior colliculus, including the retinorecipient layers. We suggest that the early amygdala responses originating from the colliculo-pulvino-amygdalar pathway play dual roles in threat detection.

A fast link between face perception and memory in the temporal pole

The question of how the brain recognizes the faces of familiar individuals has been important throughout the history of neuroscience. Cells linking visual processing to person memory have been proposed but not found. Here, we report the discovery of such cells through recordings from an area in the macaque temporal pole identified with functional magnetic resonance imaging. These cells responded to faces that were personally familiar. They responded nonlinearly to stepwise changes in face visibility and detail and holistically to face parts, reflecting key signatures of familiar face recognition. They discriminated between familiar identities, as fast as a general face identity area. The discovery of these cells establishes a new pathway for the fast recognition of familiar individuals.

The temporal pole: From anatomy to function-A literature appraisal

Historically, the anterior part of the temporal lobe was labelled as a unique structure named Brain Area 38 by Brodmann or Temporopolar Area TG by Von Economo, but its functions were unknown at that time. Later on, a few studies proposed to divide the temporal pole in several different subparts, based on distinct cytoarchitectural structure or connectivity patterns, while a still growing number of studies have associated the temporal pole with many cognitive functions. In this review, we provide an overview of the temporal pole anatomical and histological structure and its various functions. We performed a literature review of articles published prior to September 30, 2020 that included 112 articles. The temporal pole has thereby been associated with several high-level cognitive processes: visual processing for complex objects and face recognition, autobiographic memory, naming and word-object labelling, semantic processing in all modalities, and socio-emotional processing, as demonstrated in healthy subjects and in patients with neurological or psychiatric diseases, especially in the field of neurodegenerative disorders. A good knowledge of those functions and the symptoms associated with temporal pole lesions or dysfunctions is helpful to identify these diseases, whose diagnosis may otherwise be difficult.

Automatic Encoding of a View-Centered Background Image in the Macaque Temporal Lobe

Perceptual processing along the ventral visual pathway to the hippocampus (HPC) is hypothesized to be substantiated by signal transformation from retinotopic space to relational space, which represents interrelations among constituent visual elements. However, our visual perception necessarily reflects the first person’s perspective based on the retinotopic space. To investigate this two-facedness of visual perception, we compared neural activities in the temporal lobe (anterior inferotemporal cortex, perirhinal and parahippocampal cortices, and HPC) between when monkeys gazed on an object and when they fixated on the screen center with an object in their peripheral vision. We found that in addition to the spatially invariant object signal, the temporal lobe areas automatically represent a large-scale background image, which specify the subject’s viewing location. These results suggest that a combination of two distinct visual signals on relational space and retinotopic space may provide the first person’s perspective serving for perception and presumably subsequent episodic memory.

Forward Processing of Object-Location Association from the Ventral Stream to Medial Temporal Lobe in Nonhuman Primates

While the hippocampus (HPC) is a prime candidate combining object identity and location due to its strong connections to the ventral and dorsal pathways via surrounding medial temporal lobe (MTL) areas, recent physiological studies have reported spatial information in the ventral pathway and its downstream target in MTL. However, it remains unknown whether the object-location association proceeds along the ventral MTL pathway before HPC. To address this question, we recorded neuronal activity from MTL and area anterior inferotemporal cortex (TE) of two macaques gazing at an object to retain its identity and location in each trial. The results showed significant effects of object-location association at a single-unit level in TE, perirhinal cortex (PRC), and HPC, but not in the parahippocampal cortex. Notably, a clear area difference emerged in the association form: 1) representations of object identity were added to those of subjects' viewing location in TE; 2) PRC signaled both the additive form and the conjunction of the two inputs; and 3) HPC signaled only the conjunction signal. These results suggest that the object and location signals are combined stepwise at TE and PRC each time primates view an object, and PRC may provide HPC with the conjunctional signal, which might be used for encoding episodic memory.

Laminar Pattern of Projections Indicates the Hierarchical Organization of the Anterior Cingulate-Temporal Lobe Emotion System

The anterior cingulate cortex (ACC), surrounding the genu of the corpus callosum, plays important roles in emotional processing and is functionally divided into the dorsal, perigenual, and subgenual subregions (dACC, pgACC, and sgACC, respectively). Previous studies have suggested that the pgACC and sgACC have distinctive roles in the regulation of emotion. In order to elicit appropriate emotional responses, these ACC regions require sensory information from the environment. Anatomically, the ACC has rich connections with the temporal lobe, where the higher-order processing of sensory information takes place. To clarify the organization of sensory inputs into the ACC subregions, we injected neuronal tracers into the pgACC, sgACC, and dACC and compared the afferent connections. Previously, we analyzed the afferent projections from the amygdala and found a distinct pattern for the sgACC. In the present study, the patterns of the afferent projections were analyzed in the temporal cortex, especially the temporal pole (TP) and medial temporal areas. After tracers were injected into the sgACC, we observed labeled neurons in the TP and the subiculum of the hippocampal formation. The majority of the labeled cell bodies were found in the superficial layers of the TP ("feedforward" type projections). The pgACC received afferent projections from the TP, the entorhinal cortex (EC), and the parahippocampal cortex (PHC), but not from the hippocampus. In each area, the labeled cells were mainly found in the deep layers ("feedback" type projection). The pattern for the dACC was similar to that for the pgACC. Previous studies suggested that the pgACC, but not the sgACC receive projections from the dorsolateral prefrontal cortex (DLPFC). These data suggest that the sgACC plays crucial roles for emotional responses based on sensory and mnemonic inputs from the anterior temporal lobe, whereas the pgACC is more related to the cognitive control of emotion.

Age-Related Changes in Perirhinal Cortex Sensitivity to Configuration and Part Familiarity and Connectivity to Visual Cortex

The perirhinal cortex (PRC) is a medial temporal lobe (MTL) structure known to be involved in assessing whether an object is familiar (i.e., meaningful) or novel. Recent evidence shows that the PRC is sensitive to the familiarity of both whole object configurations and their parts, and suggests the PRC may modulate part familiarity responses in V2. Here, using functional magnetic resonance imaging (fMRI), we investigated age-related decline in the PRC's sensitivity to part/configuration familiarity and assessed its functional connectivity to visual cortex in young and older adults. Participants categorized peripherally presented silhouettes as familiar ("real-world") or novel. Part/configuration familiarity was manipulated via three silhouette configurations: Familiar (parts/configurations familiar), Control Novel (parts/configurations novel), and Part-Rearranged Novel (parts familiar, configurations novel). "Real-world" judgments were less accurate than "novel" judgments, although accuracy did not differ between age groups. The fMRI data revealed differential neural activity, however: In young adults, a linear pattern of activation was observed in left hemisphere (LH) PRC, with Familiar > Control Novel > Part-Rearranged Novel. Older adults did not show this pattern, indicating age-related decline in the PRC's sensitivity to part/configuration familiarity. A functional connectivity analysis revealed a significant coupling between the PRC and V2 in the LH in young adults only. Older adults showed a linear pattern of activation in the temporopolar cortex (TPC), but no evidence of TPC-V2 connectivity. This is the first study to demonstrate age-related decline in the PRC's representations of part/configuration familiarity and its covariance with visual cortex.

Neural Correlates underlying Size Constancy in Virtual Three-Dimensional Space

The perceived size of an object remains relatively constant although its retinal size keeps decreasing as the object moves away along the depth dimension of the 3D space, i.e. size constancy. Neural mechanisms generating size constancy in virtual 3D space, however, remain poorly understood. By constructing a virtual 3D world in the MR scanner, we positioned the same 3D objects either near or far from the observers so that the near and far objects were perceived as having the same physical size despite their differences in retinal size. To control for the effect of differential retinal size, an additional 2D condition was introduced: a large and a small object, with matched retinal images as the near and far objects in the 3D condition, respectively, were presented on a 2D screen. Differences in retinal size activated overlapped areas in bilateral inferior occipital gyrus (IOG) in both experiments. The overlapped areas in IOG, however, showed different patterns of functional connectivity with different neural networks, depending on the perceived size of objects. In particular, IOG showed enhanced connectivity with bilateral superior parietal cortex in the 2D condition, but with inferior temporal and prefrontal cortex in the virtual 3D condition, i.e., size constancy.

Brain Abnormalities in Congenital Fibrosis of the Extraocular Muscles Type 1: A Multimodal MRI Imaging Study

PURPOSE

To explore the possible brain structural and functional alterations in congenital fibrosis of extraocular muscles type 1 (CFEOM1) patients using multimodal MRI imaging.

METHODS

T1-weighted, diffusion tensor images and functional MRI data were obtained from 9 KIF21A positive patients and 19 age- and gender-matched healthy controls. Voxel based morphometry and tract based spatial statistics were applied to the T1-weighted and diffusion tensor images, respectively. Amplitude of low frequency fluctuations and regional homogeneity were used to process the functional MRI data. We then compared these multimodal characteristics between CFEOM1 patients and healthy controls.

RESULTS

Compared with healthy controls, CFEOM1 patients demonstrated increased grey matter volume in bilateral frontal orbital cortex and in the right temporal pole. No diffusion indices changes were detected, indicating unaffected white matter microstructure. In addition, from resting state functional MRI data, trend of amplitude of low-frequency fluctuations increases were noted in the right inferior parietal lobe and in the right frontal cortex, and a trend of ReHo increase (p<0.001 uncorrected) in the left precentral gyrus, left orbital frontal cortex, temporal pole and cingulate gyrus.

CONCLUSIONS

CFEOM1 patients had structural and functional changes in grey matter, but the white matter was unaffected. These alterations in the brain may be due to the abnormality of extraocular muscles and their innervating nerves. Future studies should consider the possible correlations between brain morphological/functional findings and clinical data, especially pertaining to eye movements, to obtain more precise answers about the role of brain area changes and their functional consequence in CFEOM1.

Conjunctive Coding of Complex Object Features

Critical to perceiving an object is the ability to bind its constituent features into a cohesive representation, yet the manner by which the visual system integrates object features to yield a unified percept remains unknown. Here, we present a novel application of multivoxel pattern analysis of neuroimaging data that allows a direct investigation of whether neural representations integrate object features into a whole that is different from the sum of its parts. We found that patterns of activity throughout the ventral visual stream (VVS), extending anteriorly into the perirhinal cortex (PRC), discriminated between the same features combined into different objects. Despite this sensitivity to the unique conjunctions of features comprising objects, activity in regions of the VVS, again extending into the PRC, was invariant to the viewpoints from which the conjunctions were presented. These results suggest that the manner in which our visual system processes complex objects depends on the explicit coding of the conjunctions of features comprising them.

Medial temporal lobe coding of item and spatial information during relational binding in working memory

Several models have proposed that different medial temporal lobe (MTL) regions represent different kinds of information in the service of long-term memory. For instance, it has been proposed that perirhinal cortex (PRC), parahippocampal cortex (PHC), and hippocampus differentially support long-term memory for item information, spatial context, and item-context relations present during an event, respectively. Recent evidence has indicated that, in addition to long-term memory, MTL subregions may similarly contribute to processes that support the retention of complex spatial arrangements of objects across short delays. Here, we used functional magnetic resonance imaging and multivoxel pattern similarity analysis to investigate the extent to which human MTL regions independently code for object and spatial information, as well as the conjunction of this information, during working memory encoding and active maintenance. Voxel activity patterns in PRC, temporopolar cortex, and amygdala carried information about individual objects, whereas activity patterns in the PHC and posterior hippocampus carried information about the configuration of spatial locations that was to be remembered. Additionally, the integrity of multivoxel patterns in the right anterior hippocampus across encoding and delay periods was predictive of accurate short-term memory for object-location relationships. These results are consistent with parallel processing of item and spatial context information by PRC and PHC, respectively, and the binding of item and context by the hippocampus.

The timing and directional connectivity of human frontoparietal and ventral visual attention networks in emotional scene perception

Electrocortical and hemodynamic measures reliably identify enhanced activity in the ventral and dorsal visual cortices during the perception of emotionally arousing versus neutral images, an effect that may reflect directive feedback from the subcortical amygdala. However, other brain regions strongly modulate visual attention, such as frontal eye fields (FEF) and intraparietal sulcus (IPS). Here we employ rapid sampling of BOLD signal (4 Hz) in the amygdala, fusiform gyrus (FG), FEF and IPS in 42 human participants as they viewed a series of emotional and neutral natural scene photographs balanced for luminosity and complexity, to test whether emotional discrimination is evident in dorsal structures prior to such discrimination in the amygdala and FG. Granger causality analyses were used to assess directional connectivity within dorsal and ventral networks. Results demonstrate emotionally-enhanced peak BOLD signal in the amygdala, FG, FEF, and IPS, with the onset of BOLD signal discrimination occurring between 2 and 3s after stimulus onset in ventral structures, and between 4 and 5s in FEF and IPS. Granger causality estimates yield stronger directional connectivity from IPS to FEF than the reverse in this emotional picture paradigm. Consistent with a reentrant perspective of emotional scene perception, greater directional connectivity was found from the amygdala to FG compared to the reverse. These data support a perspective in which the registration of emotional scene content is orchestrated by the amygdala and rostral inferotemporal visual cortex.

Beyond the FFA: The role of the ventral anterior temporal lobes in face processing

Extensive research has supported the existence of a specialized face-processing network that is distinct from the visual processing areas used for general object recognition. The majority of this work has been aimed at characterizing the response properties of the fusiform face area (FFA) and the occipital face area (OFA), which together are thought to constitute the core network of brain areas responsible for facial identification. Although accruing evidence has shown that face-selective patches in the ventral anterior temporal lobes (vATLs) are interconnected with the FFA and OFA, and that they play a role in facial identification, the relative contribution of these brain areas to the core face-processing network has remained unarticulated. Here we review recent research critically implicating the vATLs in face perception and memory. We propose that current models of face processing should be revised such that the ventral anterior temporal lobes serve a centralized role in the visual face-processing network. We speculate that a hierarchically organized system of face processing areas extends bilaterally from the inferior occipital gyri to the vATLs, with facial representations becoming increasingly complex and abstracted from low-level perceptual features as they move forward along this network. The anterior temporal face areas may serve as the apex of this hierarchy, instantiating the final stages of face recognition. We further argue that the anterior temporal face areas are ideally suited to serve as an interface between face perception and face memory, linking perceptual representations of individual identity with person-specific semantic knowledge.

Neural correlates of auditory recognition memory in the primate dorsal temporal pole

Temporal pole (TP) cortex is associated with higher-order sensory perception and/or recognition memory, as human patients with damage in this region show impaired performance during some tasks requiring recognition memory (Olson et al. 2007). The underlying mechanisms of TP processing are largely based on examination of the visual nervous system in humans and monkeys, while little is known about neuronal activity patterns in the auditory portion of this region, dorsal TP (dTP; Poremba et al. 2003). The present study examines single-unit activity of dTP in rhesus monkeys performing a delayed matching-to-sample task utilizing auditory stimuli, wherein two sounds are determined to be the same or different. Neurons of dTP encode several task-relevant events during the delayed matching-to-sample task, and encoding of auditory cues in this region is associated with accurate recognition performance. Population activity in dTP shows a match suppression mechanism to identical, repeated sound stimuli similar to that observed in the visual object identification pathway located ventral to dTP (Desimone 1996; Nakamura and Kubota 1996). However, in contrast to sustained visual delay-related activity in nearby analogous regions, auditory delay-related activity in dTP is transient and limited. Neurons in dTP respond selectively to different sound stimuli and often change their sound response preferences between experimental contexts. Current findings suggest a significant role for dTP in auditory recognition memory similar in many respects to the visual nervous system, while delay memory firing patterns are not prominent, which may relate to monkeys' shorter forgetting thresholds for auditory vs. visual objects.

Is Granger causality a viable technique for analyzing fMRI data?

Multivariate neural data provide the basis for assessing interactions in brain networks. Among myriad connectivity measures, Granger causality (GC) has proven to be statistically intuitive, easy to implement, and generate meaningful results. Although its application to functional MRI (fMRI) data is increasing, several factors have been identified that appear to hinder its neural interpretability: (a) latency differences in hemodynamic response function (HRF) across different brain regions, (b) low-sampling rates, and (c) noise. Recognizing that in basic and clinical neuroscience, it is often the change of a dependent variable (e.g., GC) between experimental conditions and between normal and pathology that is of interest, we address the question of whether there exist systematic relationships between GC at the fMRI level and that at the neural level. Simulated neural signals were convolved with a canonical HRF, down-sampled, and noise-added to generate simulated fMRI data. As the coupling parameters in the model were varied, fMRI GC and neural GC were calculated, and their relationship examined. Three main results were found: (1) GC following HRF convolution is a monotonically increasing function of neural GC; (2) this monotonicity can be reliably detected as a positive correlation when realistic fMRI temporal resolution and noise level were used; and (3) although the detectability of monotonicity declined due to the presence of HRF latency differences, substantial recovery of detectability occurred after correcting for latency differences. These results suggest that Granger causality is a viable technique for analyzing fMRI data when the questions are appropriately formulated.

A review of cell assemblies

Since the cell assembly (CA) was hypothesised, it has gained substantial support and is believed to be the neural basis of psychological concepts. A CA is a relatively small set of connected neurons, that through neural firing can sustain activation without stimulus from outside the CA, and is formed by learning. Extensive evidence from multiple single unit recording and other techniques provides support for the existence of CAs that have these properties, and that their neurons also spike with some degree of synchrony. Since the evidence is so broad and deep, the review concludes that CAs are all but certain. A model of CAs is introduced that is informal, but is broad enough to include, e.g. synfire chains, without including, e.g. holographic reduced representation. CAs are found in most cortical areas and in some sub-cortical areas, they are involved in psychological tasks including categorisation, short-term memory and long-term memory, and are central to other tasks including working memory. There is currently insufficient evidence to conclude that CAs are the neural basis of all concepts. A range of models have been used to simulate CA behaviour including associative memory and more process- oriented tasks such as natural language parsing. Questions involving CAs, e.g. memory persistence, CAs' complex interactions with brain waves and learning, remain unanswered. CA research involves a wide range of disciplines including biology and psychology, and this paper reviews literature directly related to the CA, providing a basis of discussion for this interdisciplinary community on this important topic. Hopefully, this discussion will lead to more formal and accurate models of CAs that are better linked to neuropsychological data.

Functional connectivity patterns of medial and lateral macaque frontal eye fields reveal distinct visuomotor networks

It has been previously shown that small- and large-amplitude saccades have different functions during vision in natural environments. Large saccades are associated with reaching movements toward objects, whereas small saccades facilitate the identification of more detailed object features necessary for successful grasping and manual manipulation. To determine whether these represent dichotomous processing streams, we used resting-state functional MRI to examine the functional connectivity patterns of the medial and lateral frontal eye field (FEF) regions that encode large- and small-amplitude saccades, respectively. We found that the spontaneous blood oxygen level-dependent signals of the medial FEF were functionally correlated with areas known to be involved in reaching movements and executive control processes, whereas lateral FEF was functionally correlated with cortical areas involved in object processing and in grasping, fixation, and manipulation of objects. The results provide strong evidence for two distinct visuomotor network systems in the primate brain that likely reflect the alternating phases of vision for action in natural environments.

Single neuron activity and theta modulation in postrhinal cortex during visual object discrimination

Postrhinal cortex, rodent homolog of the primate parahippocampal cortex, processes spatial and contextual information. Our hypothesis of postrhinal function is that it serves to encode context, in part, by forming representations that link objects to places. To test this hypothesis, we recorded postrhinal neurons and local field potentials (LFPs) in rats trained on a two-choice, visual discrimination task. As predicted, many postrhinal neurons signaled object-location conjunctions. Another large proportion encoded egocentric motor responses. In addition, postrhinal LFPs exhibited strong oscillatory rhythms in the theta band, and many postrhinal neurons were phase locked to theta. Although correlated with running speed, theta power was lower than predicted by speed alone immediately before and after choice. However, theta power was significantly increased following incorrect decisions, suggesting a role in signaling error. These findings provide evidence that postrhinal cortex encodes representations that link objects to places and suggest postrhinal theta modulation extends to cognitive as well as spatial functions.

Anterior temporal face patches: a meta-analysis and empirical study

Evidence suggests the anterior temporal lobe (ATL) plays an important role in person identification and memory. In humans, neuroimaging studies of person memory report consistent activations in the ATL to famous and personally familiar faces and studies of patients report resection or damage of the ATL causes an associative prosopagnosia in which face perception is intact but face memory is compromised. In addition, high-resolution fMRI studies of non-human primates and electrophysiological studies of humans also suggest regions of the ventral ATL are sensitive to novel faces. The current study extends previous findings by investigating whether similar subregions in the dorsal, ventral, lateral, or polar aspects of the ATL are sensitive to personally familiar, famous, and novel faces. We present the results of two studies of person memory: a meta-analysis of existing fMRI studies and an empirical fMRI study using optimized imaging parameters. Both studies showed left-lateralized ATL activations to familiar individuals while novel faces activated the right ATL. Activations to famous faces were quite ventral, similar to what has been reported in previous high-resolution fMRI studies of non-human primates. These findings suggest that face memory-sensitive patches in the human ATL are in the ventral/polar ATL.

Visual response of neurons in the lateral intraparietal area and saccadic reaction time during a visual detection task

During visual detection with saccades, a target with higher luminance is detected with reduced reaction times. In such visual detection behaviors, luminance-related sensory signals should be converted into movement-related signals for saccade initiation. At the site where the visuomotor transformation takes place, there is the possibility that visual activity not only encodes the target luminance but also affects the generation of an upcoming saccade. To assess this possibility, we recorded single-cell activity from visually responsive neurons in the lateral intraparietal area (LIP) when monkeys made a saccade to an isolated target over five luminance levels. We found that as stimulus luminance increased, visual response strength increased, and response onset latency decreased. These luminance-related changes in activity were significantly correlated with changes in reaction time. In particular, changes in response onset latency accounted for a substantial part of the observed changes in reaction time, suggesting that luminance-related changes in response onset latency may propagate to the saccade generation process. However, the length of time from response onset to saccade onset was not constant but increased as luminance was reduced, suggesting the existence of other luminance-dependent processing in downstream and/or parallel pathways before saccade generation. Additionally, we failed to find strong covariance between response strength or latency and reaction time when the effect of luminance changes was removed. Thus, the present results reveal how visually responsive LIP neurons contribute to saccade generation in visual detection.

Feature binding of a continuously changing object

Consider a feature of a stimulus (such as color, luminance, or spatial frequency) that changes over time along a continuum. When a second stimulus is briefly pulsed with the same feature value as the first stimulus, the two stimuli are not perceived to match. Instead, the continuously changing stimulus is perceived to be further ahead on the feature continuum than the pulsed stimulus [Nat. Neurosci. 3, 489 (2000)]. This shift is quantified by the amount of time ahead on the changing continuum, which is different for various types of features. A basic question is how our percepts are affected when an object has two continuously changing features (such as color and orientation) with different magnitudes of time ahead. This was addressed using a bar continuously changing in both color and orientation. Even though the two features were part of the same object, each feature maintained a distinctly different time ahead. This implies that observers perceived at each moment a combination of color and orientation that never was presented to the eye.

Sensory and semantic category subdivisions within the anterior temporal lobes

In the semantic memory literature the anterior temporal lobe (ATL) is frequently discussed as one homogeneous region when in fact, anatomical studies indicate that it is likely that there are discrete subregions within this area. Indeed, the influential Hub Account of semantic memory has proposed that this region is a sensory-amodal, general-purpose semantic processing region. However review of the literature suggested two potential demarcations: sensory subdivisions and a social/nonsocial subdivision. To test this, participants were trained to associate social or non-social words with novel auditory, visual, or audiovisual stimuli. Later, study participants underwent an fMRI scan where they were presented with the sensory stimuli and the task was to recall the semantic associate. The results showed that there were sensory specific subdivisions within the ATL - that the perceptual encoding of auditory stimuli preferentially activated the superior ATL, visual stimuli the inferior ATL, and multisensory stimuli the polar ATL. Moreover, our data showed that there is stimulus-specific sensitivity within the ATL - the superior and polar ATLs were more sensitive to the retrieval of social knowledge as compared to non-social knowledge. No ATL regions were more sensitive to the retrieval of non-social knowledge. These findings indicate that the retrieval of newly learned semantic associations activates the ATL. In addition, superior and polar aspects of the ATL are sensitive to social stimuli but relatively insensitive to non-social stimuli, a finding that is predicted by anatomical connectivity and single-unit studies in non-human primates. And lastly, the ATL contains sensory processing subdivisions that fall along superior (auditory), inferior (visual), polar (audiovisual) subdivisions.

Role of temporal processing stages by inferior temporal neurons in facial recognition

In this review, we focus on the role of temporal stages of encoded facial information in the visual system, which might enable the efficient determination of species, identity, and expression. Facial recognition is an important function of our brain and is known to be processed in the ventral visual pathway, where visual signals are processed through areas V1, V2, V4, and the inferior temporal (IT) cortex. In the IT cortex, neurons show selective responses to complex visual images such as faces, and at each stage along the pathway the stimulus selectivity of the neural responses becomes sharper, particularly in the later portion of the responses. In the IT cortex of the monkey, facial information is represented by different temporal stages of neural responses, as shown in our previous study: the initial transient response of face-responsive neurons represents information about global categories, i.e., human vs. monkey vs. simple shapes, whilst the later portion of these responses represents information about detailed facial categories, i.e., expression and/or identity. This suggests that the temporal stages of the neuronal firing pattern play an important role in the coding of visual stimuli, including faces. This type of coding may be a plausible mechanism underlying the temporal dynamics of recognition, including the process of detection/categorization followed by the identification of objects. Recent single-unit studies in monkeys have also provided evidence consistent with the important role of the temporal stages of encoded facial information. For example, view-invariant facial identity information is represented in the response at a later period within a region of face-selective neurons. Consistent with these findings, temporally modulated neural activity has also been observed in human studies. These results suggest a close correlation between the temporal processing stages of facial information by IT neurons and the temporal dynamics of face recognition.

Six principles of visual cortical dynamics

A fundamental goal in vision science is to determine how many neurons in how many areas are required to compute a coherent interpretation of the visual scene. Here I propose six principles of cortical dynamics of visual processing in the first 150 ms following the appearance of a visual stimulus. Fast synaptic communication between neurons depends on the driving neurons and the biophysical history and driving forces of the target neurons. Under these constraints, the retina communicates changes in the field of view driving large populations of neurons in visual areas into a dynamic sequence of feed-forward communication and integration of the inward current of the change signal into the dendrites of higher order area neurons (30-70 ms). Simultaneously an even larger number of neurons within each area receiving feed-forward input are pre-excited to sub-threshold levels. The higher order area neurons communicate the results of their computations as feedback adding inward current to the excited and pre-excited neurons in lower areas. This feedback reconciles computational differences between higher and lower areas (75-120 ms). This brings the lower area neurons into a new dynamic regime characterized by reduced driving forces and sparse firing reflecting the visual areas interpretation of the current scene (140 ms). The population membrane potentials and net-inward/outward currents and firing are well behaved at the mesoscopic scale, such that the decoding in retinotopic cortical space shows the visual areas' interpretation of the current scene. These dynamics have plausible biophysical explanations. The principles are theoretical, predictive, supported by recent experiments and easily lend themselves to experimental tests or computational modeling.

The Kuleshov Effect: the influence of contextual framing on emotional attributions

Filmmakers have long recognized the importance of editing techniques to guide the audiences' perceptions and enhance the impact of a scene. We demonstrate behaviorally that pairing identical faces with either neutral or emotionally salient contextual movies, an editing technique referred to as the 'Kuleshov Effect', results in both altered attributions of facial expression and mental-state. Using functional neuroimaging (fMRI), we show that faces paired with emotional movies enhance BOLD responses in the bilateral temporal pole, anterior cingulate cortices, amygdala and bilateral superior temporal sulcus relative to identical faces juxtaposed with neutral movies. An interaction was observed in the right amygdala when subtle happy and fear faces were juxtaposed with positive and negative movies, respectively. An interaction between happy faces and negative context was also observed in bilateral amygdala suggesting that the amygdala may act to prime or tag affective value to faces. A parametric modulation of BOLD signal by attribution ratings indicated a dissociation between ventrolateral and the ventromedial prefrontal cortex for negative and positive contextually evoked attributions, respectively. These prefrontal regions may act to guide appropriate choices across altering contexts. Together, these findings offer a neurobiological basis for contextual framing effects on social attributions.

A computational model of Basal Ganglia and its role in memory retrieval in rewarded visual memory tasks

Visual working memory (WM) tasks involve a network of cortical areas such as inferotemporal, medial temporal and prefrontal cortices. We suggest here to investigate the role of the basal ganglia (BG) in the learning of delayed rewarded tasks through the selective gating of thalamocortical loops. We designed a computational model of the visual loop linking the perirhinal cortex, the BG and the thalamus, biased by sustained representations in prefrontal cortex. This model learns concurrently different delayed rewarded tasks that require to maintain a visual cue and to associate it to itself or to another visual object to obtain reward. The retrieval of visual information is achieved through thalamic stimulation of the perirhinal cortex. The input structure of the BG, the striatum, learns to represent visual information based on its association to reward, while the output structure, the substantia nigra pars reticulata, learns to link striatal representations to the disinhibition of the correct thalamocortical loop. In parallel, a dopaminergic cell learns to associate striatal representations to reward and modulates learning of connections within the BG. The model provides testable predictions about the behavior of several areas during such tasks, while providing a new functional organization of learning within the BG, putting emphasis on the learning of the striatonigral connections as well as the lateral connections within the substantia nigra pars reticulata. It suggests that the learning of visual WM tasks is achieved rapidly in the BG and used as a teacher for feedback connections from prefrontal cortex to posterior cortices.

The timing of emotional discrimination in human amygdala and ventral visual cortex

Models of visual emotional perception suggest a reentrant organization of the ventral visual system with the amygdala. Using focused functional magnetic resonance imaging in humans with a sampling rate of 100 ms, here we determine the relative timing of emotional discrimination in amygdala and ventral visual cortical structures during emotional perception. Results show that amygdala and inferotemporal visual cortex differentiate emotional from nonemotional scenes approximately 1 s before extrastriate occipital cortex, whereas primary occipital cortex shows consistent activity across all scenes. This pattern of discrimination is consistent with a reentrant organization of emotional perception in visual processing, in which transaction between rostral ventral visual cortex and amygdala originates the identification of emotional relevance.

Spatial frequency components influence cell activity in the inferotemporal cortex

We studied the correlation between the spatial frequency of complex stimuli and neuronal activity in the monkey inferotemporal (IT) cortex while performing a task that required visual recognition. Single-cell activity was recorded from the right IT cortex. The frequency components of the images used as stimuli were analyzed by using a fast Fourier transform, and a modulus was obtained for 40 spatial frequency ranges from 0.3 to 11.1 cycles/deg. We recorded 82 cells showing statistically significant responses (analysis of variance, P < 0.05) to at least one of the images used as a stimulus. Seventy-eight percent of these cells (n = 64) showed significant responses to at least three images, and in two thirds of them (n = 42), we found a statistically significant correlation (P < 0.05) between cell response and the modulus amplitude of at least one frequency range present in the images. Our results suggest that information about spatial frequency of the visual images is present in the IT cortex.

Latency and selectivity of single neurons indicate hierarchical processing in the human medial temporal lobe

Neurons in the temporal lobe of both monkeys and humans show selective responses to classes of visual stimuli and even to specific individuals. In this study, we investigate the latency and selectivity of visually responsive neurons recorded from microelectrodes in the parahippocampal cortex, entorhinal cortex, hippocampus, and amygdala of human subjects during a visual object presentation task. During 96 experimental sessions in 35 subjects, we recorded from a total of 3278 neurons. Of these units, 398 responded selectively to one or more of the presented stimuli. Mean response latencies were substantially larger than those reported in monkeys. We observed a highly significant correlation between the latency and the selectivity of these neurons: the longer the latency the greater the selectivity. Particularly, parahippocampal neurons were found to respond significantly earlier and less selectively than those in the other three regions. Regional analysis showed significant correlations between latency and selectivity within the parahippocampal cortex, entorhinal cortex, and hippocampus, but not within the amygdala. The later and more selective responses tended to be generated by cells with sparse baseline firing rates and vice versa. Our results provide direct evidence for hierarchical processing of sensory information at the interface between the visual pathway and the limbic system, by which increasingly refined and specific representations of stimulus identity are generated over time along the anatomic pathways of the medial temporal lobe.

Visual prediction: psychophysics and neurophysiology of compensation for time delays

A necessary consequence of the nature of neural transmission systems is that as change in the physical state of a time-varying event takes place, delays produce error between the instantaneous registered state and the external state. Another source of delay is the transmission of internal motor commands to muscles and the inertia of the musculoskeletal system. How does the central nervous system compensate for these pervasive delays? Although it has been argued that delay compensation occurs late in the motor planning stages, even the earliest visual processes, such as phototransduction, contribute significantly to delays. I argue that compensation is not an exclusive property of the motor system, but rather, is a pervasive feature of the central nervous system (CNS) organization. Although the motor planning system may contain a highly flexible compensation mechanism, accounting not just for delays but also variability in delays (e.g., those resulting from variations in luminance contrast, internal body temperature, muscle fatigue, etc.), visual mechanisms also contribute to compensation. Previous suggestions of this notion of "visual prediction" led to a lively debate producing re-examination of previous arguments, new analyses, and review of the experiments presented here. Understanding visual prediction will inform our theories of sensory processes and visual perception, and will impact our notion of visual awareness.

An fMRI study of the functional distinction of neuronal circuits at the sites on ventral visual stream co-activated by visual stimuli of different objects

In functional studies of the human brain using functional magnetic resonance imaging (fMRI) we often observe some functional areas that are commonly activated by different stimulus inputs even when the inputs are of different categories. It is difficult to distinguish by fMRI whether the neuronal circuits activated for processing these inputs are separate and independent ones within the functional area or whether they are mutually interactive and possibly sharing a part of the circuits for processing some common features of the input information. In order to elucidate such property of the neuronal circuits, we used a novel paradigm in which paired input stimuli with varied inter stimulus interval (ISI) were presented during a stimulation period in fMRI experiments. The refractory suppression induced by an input pair depends on ISI as well as the differing characteristics of the input pair. The extent of suppression, an indicator for the interaction between the processing activities of the input pair, can be evaluated by the shape of the time-course of fMRI responses. We examined the functional characteristics of the neuronal circuits in areas that were activated by different inputs of inter-category types, namely face stimuli and building stimuli and also by intra-category input pairs such as different faces. In the ventral visual occipitotemporal region, we found functional areas where neuronal circuits were interacting to process these inputs rather than functioning as separate independent circuits. With this approach, one can probe functional system activity at the neuronal circuit level to learn the characteristics that determine the functional roles of certain brain areas.

Impacts of facial identity and type of emotion on responses of amygdala neurons

The amygdala has been implicated in the processing of emotional expressions. Who makes the emotion and the type of emotion are important in producing appropriate responses. How amygdala neurons are affected by facial identity and type of emotion, however, has not yet been systematically examined. We examined the activity of amygdala neurons using nine monkey stimuli: 3 monkeys x 3 types of emotion. Of the 227 neurons tested, 77 responded to the monkey stimuli. The effects of facial identity and type of emotion on the response magnitude were significant in 48 and 57 neurons, respectively. Both effects were significant in 38 neurons. These results indicate that both facial identity and type of emotion have strong impacts on amygdala functions.

Working memory for visual objects: complementary roles of inferior temporal, medial temporal, and prefrontal cortex

Humans have an extraordinary ability to maintain and manipulate visual image information in the absence of perceptual stimulation. The neural substrates of visual working memory have been extensively researched, but there have been few attempts to integrate these findings into a model of how different cortical areas interact to form and maintain visual memories. In this paper, I review findings from neurophysiological, neuropsychological, and neuroimaging studies of visual working memory in human and nonhuman primates. These data support a model in which visual working memory operations rely on activation of object representations in inferior temporal cortex, via top-down feedback from neocortical areas in the prefrontal and medial temporal cortex, and also from the hippocampus.

Selectivity of inferior temporal neurons for realistic pictures predicted by algorithms for image database navigation

Primates have a remarkable ability to perceive, recognize, and discriminate among the plethora of people, places, and things that they see, and neural selectivity in the primate inferotemporal (IT) cortex is thought to underlie this ability. Here we investigated the relationship between neural response and perception by recording from IT neurons in monkeys while they viewed realistic images. We then compared the similarity of neural responses elicited by images to the quantitative similarity of the images. Image similarity was approximated using several algorithms, two of which were designed to search image databases for perceptually similar images. Some algorithms for image similarity correlated well with human perception, and these algorithms explained part of the stimulus selectivity of IT neurons. Images that elicited similar neural responses were ranked as more similar by these algorithms than images that elicited different neural responses, and images ranked as similar by the algorithms elicited similar responses from neurons. Neural selectivity was predicted more accurately when the reference images for algorithm similarity elicited either very strong or very weak responses from the neuron. The degree to which algorithms for image similarity were correlated with human perception was related to the degree to which algorithms explained the selectivity of IT neurons, providing support for the proposal that the selectivity of IT neurons is related to perceptual similarity of images.

Quantitative analysis of functional clustering of neurons in the macaque inferior temporal cortex

Neurons with similar preferences for two-dimensional shapes of intermediate complexity cluster in area TE of the monkey inferior temporal cortex. To further characterize the functional structure of area TE, we quantitatively analyzed various aspects of the visual responses of closely located neurons by applying multiple single-unit recording techniques in anesthetized monkeys. Examination of the visual responses elicited with a large, predetermined set of visual stimuli confirmed previous findings that nearby neurons, on average, exhibited positively correlated preferences for a set of visual stimuli. Nearby neurons also tended to be similar in their receptive-field organization and contrast-polarity preference. In contrast, no correlation was found in the size tuning of neighboring neurons. Pooling or subtraction of activities between a pair of nearby neurons was shown to improve stimulus discriminability, if the neuron pair had positively or negatively correlated stimulus preferences, respectively. These results indicate that nearby TE neurons share some aspects of stimulus preference, but their response selectivity differ in other aspects. Both pooling and subtraction between nearby neurons can reduce across-trial response variability, if these decoding strategies are applied to appropriate neuronal pools.

Comparison between monkey and human visual fields using a personal computer system

To use monkeys as models for eye diseases that may lead to blindness, we need to develop a method to precisely measure its visual field and to understand similarities and differences in visual field properties between monkeys and humans. The visual field of monkey was not measured precisely although the necessity. We established a new system with personal computers for precise measurement of the monkey visual field. Four monkeys and three humans served as subjects. The luminance-contrast sensitivity of the central 24 degrees field was measured while the subject was fixating a small spot. During the measurement, we continuously recorded the eye position, and discarded the data when fixation was broken. Reliability indices demonstrated high and stable behavioral performance by both monkeys and humans. The luminance-contrast sensitivity was highest around the fovea, and declined as eccentricity increased. The blind spot was clearly detected 15 degrees temporally. The overall sensitivity was higher in humans than in monkeys and the sensitivity dropped more sharply in the periphery in monkeys than in humans. We recommend this system as a convenient and reliable way to measure visual functions in monkeys in basic ophthalmologic research or in assessment of the drug effects on the visual field.

Natural image profiles are most likely to be step edges

We introduce Geometric Texton Theory (GTT), a theory of categorical visual feature classification that arises through consideration of the metamerism that affects families of co-localised linear receptive-field operators. A refinement of GTT that uses maximum likelihood (ML) to resolve this metamerism is presented. We describe a method for discovering the ML element of a metamery class by analysing a database of natural images. We apply the method to the simplest case--the ML element of a canonical metamery class defined by co-registering the location and orientation of profiles from images, and affinely scaling their intensities so that they have identical responses to 1-D, zeroth- and first-order, derivative of Gaussian operators. We find that a step edge is the ML profile. This result is consistent with our proposed theory of feature classification.

Differential connections of the temporal pole with the orbital and medial prefrontal networks in macaque monkeys

Previous studies indicate that the orbital and medial prefrontal cortex (OMPFC) is organized into "orbital" and "medial" networks, which have distinct connections with cortical, limbic, and subcortical structures. In this study, retrograde and anterograde tracer experiments in monkeys demonstrated differential connections between the medial and orbital networks and the dorsal and ventral parts of the temporal pole. The dorsal part, including dysgranular and granular areas (TGdd and TGdg), is reciprocally connected with the medial network areas on the medial wall and gyrus rectus (areas 10m, 10o, 11m, 13a, 14c, 14r, 25, and 32) and on the lateral orbital surface (areas Iai and 12o). The strongest connections are with areas 10m (caudal part), 14c, 14r, 25, 32, and Iai. The agranular temporal pole (TGa) is connected with several areas, but most strongly with medial network area 25. The granular area around the superior temporal sulcus (TGsts) and the ventral dysgranular and granular areas (TGvd and TGvg) are reciprocally connected with the orbital network (especially areas 11l, 13b, 13l, 13m, Ial, Iam, and Iapm). TGsts is strongly connected with the entire orbital network, whereas areas TGvd and TGvg have lighter and more limited connections. Intrinsic connections within the temporal pole are also restricted to dorsal or ventral parts. Together with evidence that the dorsal and ventral temporal pole are differentially connected to auditory and visual areas of the superior and inferior temporal cortex, the results indicate separate connections between these systems and the medial and orbital prefrontal networks.

Visual response properties of neurons in the parahippocampal cortex of monkeys

We examined visual response properties of single neurons in the parahippocampal (PH) cortex of alert monkeys using various visual stimuli (bars, geometrical shapes such as a circle, and images such as a human face) while the monkey fixated a spot for a juice reward. Of the investigated PH neurons 104 of 359 (29%) were found to be visually responsive. The investigation was focused on spatial and object aspects of visual processing. We investigated a visual receptive field (RF) property and a direction selectivity for a moving bar with respect to spatial processing. For half of these PH neurons (53%), the optimal stimulus position, where a visual stimulus elicited the maximal response, located peripherally, that is, with an eccentricity of more than 10 deg. More than 20% of these PH neurons had an RF that does not include the center of gaze. There were neurons in the PH cortex that appeared to convey motion signals. In addition, some PH neurons showed eye-position-dependent activity. With respect to object processing, we investigated selectivities for images, geographical shapes, orientations of a bar, and colors. For comparison purposes, we also examined responses of perirhinal (PR) neurons. PH neurons showed selective responses to these stimuli, but PR neurons were found to be more selective for images than PH neurons. These results suggest that the PH cortex is involved in both spatial and object processing, but less involved than the PR cortex in processing of complex images.

Forward processing of long-term associative memory in monkey inferotemporal cortex

The macaque inferotemporal (IT) cortex, which serves as the storehouse of visual long-term memory, consists of two distinct but mutually interconnected areas: area TE (TE) and area 36 (A36). In the present study, we tested whether memory encoding is put forward at this stage, i.e., whether association between the representations of different but semantically linked objects proceeds forward from TE to A36. To address this question, we trained monkeys in a pair-association (PA) memory task, after which single-unit activities were recorded from TE and A36 during PA trials. Neurons in both areas showed stimulus-selective cue responses (347 in TE, 76 in A36; "cue-selective neurons") that provided, at the population level, mnemonic linkage between the paired associates. The percentage of neurons in which responses to the paired associates were significantly (p < 0.01) correlated at the single-neuron level ("pair-coding neuron") dramatically increased from TE (4.9% of the cue-selective neurons) to A36 (33%). The pair-coding neurons in A36 were further separable into Type1 (68%) and Type2 (32%) on the basis of their initial transient responses after cue stimulus presentation. Type1 neurons, but not Type2 neurons, began to encode association between paired stimuli as soon as they exhibited stimulus selectivity. Thus, the representation of long-term memory encoded by Type1 neurons in A36 is likely substantiated without feedback input from other higher centers. Therefore, we conclude that association between the representations of the paired associates proceeds forward at this critical step within IT cortex, suggesting selective convergence onto a single A36 neuron from two TE neurons that encode separate visual objects.

The hole precedence in face but not figure discrimination and its neuronal correlates

The discriminative tasks of face and geometrical figure were trained in three rhesus monkeys. The hole feature speeds up learning of face discrimination, but hampers learning of figure discrimination. By reducing presentation duration of the stimuli, the detecting precedence appears to be the hole feature of the face component, while it appears to be shape feature of figures. The patterns of neuron firings in inferior temporal cortex (IT) are consistent with the context-dependent precedence of hole feature. The results might suggest that the neural correlates exist not only in IT neurons, but also in combination with executive mechanism.

On the neural correlates of object recognition awareness: relationship to computational activities and activities mediating perceptual awareness

Based on theoretical considerations of Aurell (1979) and Block (1995), we argue that object recognition awareness is distinct from purely sensory awareness and that the former is mediated by neuronal activities in areas that are separate and distinct from cortical sensory areas. We propose that two of the principal functions of neuronal activities in sensory cortex, which are to provide sensory awareness and to effect the computations that are necessary for object recognition, are dissociated. We provide examples of how this dissociation might be achieved and argue that the components of the neuronal activities which carry the computations do not directly enter the awareness of the subject. The results of these computations are sparse representations (i.e., vector or distributed codes) which are activated by the presentation of particular sensory objects and are essentially engrams for the recognition of objects. These final representations occur in the highest order areas of sensory cortex; in the visual analyzer, the areas include the anterior part of the inferior temporal cortex and the perirhinal cortex. We propose, based on lesion and connectional data, that the two areas in which activities provide recognition awareness are the temporopolar cortex and the medial orbitofrontal cortex. Activities in the temporopolar cortex provide the recognition awareness of objects learned in the remote past (consolidated object recognition), and those in the medial orbitofrontal cortex provide the recognition awareness of objects learned in the recent past. The activation of the sparse representation for a particular sensory object in turn activates neurons in one or both of these regions of cortex, and it is the activities of these neurons that provide the awareness of recognition of the object in question. The neural circuitry involved in the activation of these representations is discussed.