Tolerance of macaque middle STS body patch neurons to shape-preserving stimulus transformations

Natural scenes reveal diverse representations of 2D and 3D body pose in the human brain

Human pose, defined as the spatial relationships between body parts, carries instrumental information supporting the understanding of motion and action of a person. A substantial body of previous work has identified cortical areas responsive to images of bodies and different body parts. However, the neural basis underlying the visual perception of body part relationships has received less attention. To broaden our understanding of body perception, we analyzed high-resolution fMRI responses to a wide range of poses from over 4,000 complex natural scenes. Using ground-truth annotations and an application of three-dimensional (3D) pose reconstruction algorithms, we compared similarity patterns of cortical activity with similarity patterns built from human pose models with different levels of depth availability and viewpoint dependency. Targeting the challenge of explaining variance in complex natural image responses with interpretable models, we achieved statistically significant correlations between pose models and cortical activity patterns (though performance levels are substantially lower than the noise ceiling). We found that the 3D view-independent pose model, compared with two-dimensional models, better captures the activation from distinct cortical areas, including the right posterior superior temporal sulcus (pSTS). These areas, together with other pose-selective regions in the LOTC, form a broader, distributed cortical network with greater view-tolerance in more anterior patches. We interpret these findings in light of the computational complexity of natural body images, the wide range of visual tasks supported by pose structures, and possible shared principles for view-invariant processing between articulated objects and ordinary, rigid objects.

Configuration-sensitive face-body interactions in primate visual cortex

Traditionally, the neural processing of faces and bodies is studied separately, although they are encountered together, as parts of an agent. Despite its social importance, it is poorly understood how faces and bodies interact, particularly at the single-neuron level. Here, we examined the interaction between faces and bodies in the macaque inferior temporal (IT) cortex, targeting an fMRI-defined patch. We recorded responses of neurons to monkey images in which the face was in its natural location (natural face-body configuration), or in which the face was mislocated with respect to the upper body (unnatural face-body configuration). On average, the neurons did not respond stronger to the natural face-body configurations compared to the summed responses to their faces and bodies, presented in isolation. However, the neurons responded stronger to the natural compared to the unnatural face-body configurations. This configuration effect was present for face- and monkey-centered images, did not depend on local feature differences between configurations, and was present when the face was replaced by a small object. The face-body interaction rules differed between natural and unnatural configurations. In sum, we show for the first time that single IT neurons process faces and bodies in a configuration-specific manner, preferring natural face-body configurations.

Bodies in motion: Unraveling the distinct roles of motion and shape in dynamic body responses in the temporal cortex

The temporal cortex represents social stimuli, including bodies. We examine and compare the contributions of dynamic and static features to the single-unit responses to moving monkey bodies in and between a patch in the anterior dorsal bank of the superior temporal sulcus (dorsal patch [DP]) and patches in the anterior inferotemporal cortex (ventral patch [VP]), using fMRI guidance in macaques. The response to dynamics varies within both regions, being higher in DP. The dynamic body selectivity of VP neurons correlates with static features derived from convolutional neural networks and motion. DP neurons' dynamic body selectivity is not predicted by static features but is dominated by motion. Whereas these data support the dominance of motion in the newly proposed "dynamic social perception" stream, they challenge the traditional view that distinguishes DP and VP processing in terms of motion versus static features, underscoring the role of inferotemporal neurons in representing body dynamics.

Categorization learning induced changes in action representations in the macaque STS

Neuroimaging and single cell recordings have demonstrated the presence of STS body category-selective regions (body patches) containing neurons responding to presentation of static bodies and body parts. To date, it remains unclear if these body patches and additional STS regions respond during observation of different categories of dynamic actions and to what extent categorization learning influences representations of observed actions in the STS. In the present study, we trained monkeys to discriminate videos depicting three different actions categories (grasping, touching and reaching) with a forced-choice action categorization task. Before and after categorization training, we performed fMRI recordings while monkeys passively observed the same action videos. At the behavioral level, after categorization training, monkeys generalized to untrained action exemplars, in particular for grasping actions. Before training, uni- and/or multivariate fMRI analyses suggest a broad representation of dynamic action categories in particular in posterior and middle STS. Univariate analysis further suggested action category specific training effects in middle and anterior body patches, face patch ML and posterior STS region MT and FST. Overall, our fMRI experiments suggest a widespread representation of observed dynamic bodily actions in the STS that can be modulated by visual learning, supporting its proposed role in action recognition.

A broadly tuned network for affective body language in the macaque brain

Body language is a powerful tool that we use to communicate how we feel, but it is unclear whether other primates also communicate in this way. Here, we use functional magnetic resonance imaging to show that the body-selective patches in macaques are activated by affective body language. Unexpectedly, we found these regions to be tolerant of naturalistic variation in posture as well as species; the bodies of macaques, humans, and domestic cats all evoked a stronger response when they conveyed fear than when they conveyed no affect. Multivariate analyses confirmed that the neural representation of fear-related body expressions was species-invariant. Collectively, these findings demonstrate that, like humans, macaques have body-selective brain regions in the ventral visual pathway for processing affective body language. These data also indicate that representations of body stimuli in these regions are built on the basis of emergent properties, such as socio-affective meaning, and not just putative image properties.

More Than the Face: Representations of Bodies in the Inferior Temporal Cortex

Visual representations of bodies, in addition to those of faces, contribute to the recognition of con- and heterospecifics, to action recognition, and to nonverbal communication. Despite its importance, the neural basis of the visual analysis of bodies has been less studied than that of faces. In this article, I review what is known about the neural processing of bodies, focusing on the macaque temporal visual cortex. Early single-unit recording work suggested that the temporal visual cortex contains representations of body parts and bodies, with the dorsal bank of the superior temporal sulcus representing bodily actions. Subsequent functional magnetic resonance imaging studies in both humans and monkeys showed several temporal cortical regions that are strongly activated by bodies. Single-unit recordings in the macaque body patches suggest that these represent mainly body shape features. More anterior patches show a greater viewpoint-tolerant selectivity for body features, which may reflect a processing principle shared with other object categories, including faces. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Curvature domains in V4 of macaque monkey

An important aspect of visual object recognition is the ability to perceive object shape. Two basic components of complex shapes are straight and curved contours. A large body of evidence suggests a modular hierarchy for shape representation progressing from simple and complex orientation in early areas V1 and V2, to increasingly complex stages of curvature representation in V4, TEO, and TE. Here, we reinforce and extend the concept of modular representation. Using intrinsic signal optical imaging in Macaque area V4, we find sub-millimeter sized modules for curvature representation that are organized from low to high curvatures as well as domains with complex curvature preference. We propose a possible 'curvature hypercolumn' within V4. In combination with previous studies, we suggest that the key emergent functions at each stage of cortical processing are represented in systematic, modular maps.

The neurons that mistook a hat for a face

Despite evidence that context promotes the visual recognition of objects, decades of research have led to the pervasive notion that the object processing pathway in primate cortex consists of multiple areas that each process the intrinsic features of a few particular categories (e.g. faces, bodies, hands, objects, and scenes). Here we report that such category-selective neurons do not in fact code individual categories in isolation but are also sensitive to object relationships that reflect statistical regularities of the experienced environment. We show by direct neuronal recording that face-selective neurons respond not just to an image of a face, but also to parts of an image where contextual cues—for example a body—indicate a face ought to be, even if what is there is not a face.

The impact of stimulus size and orientation on individual face coding in monkey face-selective cortex

Face-selective neurons in the monkey temporal cortex discharge at different rates in response to pictures of different individual faces. Here we tested whether this pattern of response across single neurons in the face-selective area ML (located in the middle Superior Temporal Sulcus) tolerates two affine transformations; picture-plane inversion, known to decrease the average response of face-selective neurons and the other, stimulus size. We recorded the response of 57 ML neurons in two awake and fixating monkeys. Face stimuli were presented at two sizes (10 and 5 degrees of visual angle) and two orientations (upright and inverted). Different faces elicited distinct patterns of activity across ML neurons that were reliable (i.e., predictable with a classifier) within a specific size and orientation condition. Despite observing a reduction in the average response magnitude of face-selective neurons to inverted faces, compared to upright faces, classifier performance was above chance for both upright and inverted faces. While decoding was largely preserved across changes in stimulus size, a classifier trained with one orientation condition and tested on the other did not lead to performance above chance level. We conclude that different individual faces can be decoded from patterns of responses in the monkey area ML regardless of orientation or size, but with qualitatively different patterns of responses for upright and inverted faces.

Frontier of Self and Impact Prediction

The construction of a coherent representation of our body and the mapping of the space immediately surrounding it are of the highest ecological importance. This space has at least three specificities: it is a space where actions are planned in order to interact with our environment; it is a space that contributes to the experience of self and self-boundaries, through tactile processing and multisensory interactions; last, it is a space that contributes to the experience of body integrity against external events. In the last decades, numerous studies have been interested in peripersonal space (PPS), defined as the space directly surrounding us and which we can interact with (for reviews, see Cléry et al., 2015b; de Vignemont and Iannetti, 2015; di Pellegrino and Làdavas, 2015). These studies have contributed to the understanding of how this space is constructed, encoded and modulated. The majority of these studies focused on subparts of PPS (the hand, the face or the trunk) and very few of them investigated the interaction between PPS subparts. In the present review, we summarize the latest advances in this research and we discuss the new perspectives that are set forth for futures investigations on this topic. We describe the most recent methods used to estimate PPS boundaries by the means of dynamic stimuli. We then highlight how impact prediction and approaching stimuli modulate this space by social, emotional and action-related components involving principally a parieto-frontal network. In a next step, we review evidence that there is not a unique representation of PPS but at least three sub-sections (hand, face and trunk PPS). Last, we discuss how these subspaces interact, and we question whether and how bodily self-consciousness (BSC) is functionally and behaviorally linked to PPS.

Binocular depth processing in the ventral visual pathway

One of the most powerful forms of depth perception capitalizes on the small relative displacements, or binocular disparities, in the images projected onto each eye. The brain employs these disparities to facilitate various computations, including sensori-motor transformations (reaching, grasping), scene segmentation and object recognition. In accordance with these different functions, disparity activates a large number of regions in the brain of both humans and monkeys. Here, we review how disparity processing evolves along different regions of the ventral visual pathway of macaques, emphasizing research based on both correlational and causal techniques. We will discuss the progression in the ventral pathway from a basic absolute disparity representation to a more complex three-dimensional shape code. We will show that, in the course of this evolution, the underlying neuronal activity becomes progressively more bound to the global perceptual experience. We argue that these observations most probably extend beyond disparity processing per se, and pertain to object processing in the ventral pathway in general. We conclude by posing some important unresolved questions whose answers may significantly advance the field, and broaden its scope.This article is part of the themed issue 'Vision in our three-dimensional world'.

Shape Selectivity of Middle Superior Temporal Sulcus Body Patch Neurons

Functional MRI studies in primates have demonstrated cortical regions that are strongly activated by visual images of bodies. The presence of such body patches in macaques allows characterization of the stimulus selectivity of their single neurons. Middle superior temporal sulcus body (MSB) patch neurons showed similar stimulus selectivity for natural, shaded, and textured images compared with their silhouettes, suggesting that shape is an important determinant of MSB responses. Here, we examined and modeled the shape selectivity of single MSB neurons. We measured the responses of single MSB neurons to a variety of shapes producing a wide range of responses. We used an adaptive stimulus sampling procedure, selecting and modifying shapes based on the responses of the neuron. Forty percent of shapes that produced the maximal response were rated by humans as animal-like, but the top shape of many MSB neurons was not judged as resembling a body. We fitted the shape selectivity of MSB neurons with a model that parameterizes shapes in terms of curvature and orientation of contour segments, with a pixel-based model, and with layers of units of convolutional neural networks (CNNs). The deep convolutional layers of CNNs provided the best goodness-of-fit, with a median explained explainable variance of the neurons’ responses of 77%. The goodness-of-fit increased along the convolutional layers’ hierarchy but was lower for the fully connected layers. Together with demonstrating the successful modeling of single unit shape selectivity with deep CNNs, the data suggest that semantic or category knowledge determines only slightly the single MSB neuron’s shape selectivity.

Stimulus features coded by single neurons of a macaque body category selective patch

Body category-selective regions of the primate temporal cortex respond to images of bodies, but it is unclear which fragments of such images drive single neurons' responses in these regions. Here we applied the Bubbles technique to the responses of single macaque middle superior temporal sulcus (midSTS) body patch neurons to reveal the image fragments the neurons respond to. We found that local image fragments such as extremities (limbs), curved boundaries, and parts of the torso drove the large majority of neurons. Bubbles revealed the whole body in only a few neurons. Neurons coded the features in a manner that was tolerant to translation and scale changes. Most image fragments were excitatory but for a few neurons both inhibitory and excitatory fragments (opponent coding) were present in the same image. The fragments we reveal here in the body patch with Bubbles differ from those suggested in previous studies of face-selective neurons in face patches. Together, our data indicate that the majority of body patch neurons respond to local image fragments that occur frequently, but not exclusively, in bodies, with a coding that is tolerant to translation and scale. Overall, the data suggest that the body category selectivity of the midSTS body patch depends more on the feature statistics of bodies (e.g., extensions occur more frequently in bodies) than on semantics (bodies as an abstract category).

Neural representation for object recognition in inferotemporal cortex

We suggest that population representation of objects in inferotemporal cortex lie on a continuum between a purely structural, parts-based description and a purely holistic description. The intrinsic dimensionality of object representation is estimated to be around 100, perhaps with lower dimensionalities for object representations more toward the holistic end of the spectrum. Cognitive knowledge in the form of semantic information and task information feed back to inferotemporal cortex from perirhinal and prefrontal cortex respectively, providing high-level multimodal-based expectations that assist in the interpretation of object stimuli. Integration of object information across eye movements may also contribute to object recognition through a process of active vision.