Decoding of the other's focus of attention by a temporal cortex module

Dynamic modulation of social gaze by sex and familiarity in marmoset dyads

Social communication relies on the ability to perceive and interpret the direction of others' attention, and is commonly conveyed through head orientation and gaze direction in humans and nonhuman primates. However, traditional social gaze experiments in nonhuman primates require restraining head movements, significantly limiting their natural behavioral repertoire. Here, we developed a novel framework for accurately tracking facial features and three-dimensional head gaze orientations of multiple freely moving common marmosets ( Callithrix jacchus ). By combining deep learning-based computer vision tools with triangulation algorithms, we were able to track the facial features of marmoset dyads within an arena. This method effectively generates dynamic 3D geometrical facial frames while overcoming common challenges like occlusion. To detect the head gaze direction, we constructed a virtual cone, oriented perpendicular to the facial frame. Using this pipeline, we quantified different types of interactive social gaze events, including partner-directed gaze and joint gaze to a shared spatial location. We observed clear effects of sex and familiarity on both interpersonal distance and gaze dynamics in marmoset dyads. Unfamiliar pairs exhibited more stereotyped patterns of arena occupancy, more sustained levels of social gaze across social distance, and increased social gaze monitoring. On the other hand, familiar pairs exhibited higher levels of joint gazes. Moreover, males displayed significantly elevated levels of gazes toward females' faces and the surrounding regions, irrespective of familiarity. Our study reveals the importance of two key social factors in driving the gaze behaviors of a prosocial primate species and lays the groundwork for a rigorous quantification of primate behaviors in naturalistic settings.

The big mixup: Neural representation during natural modes of primate visual behavior

The primate brain has evolved specialized visual capacities to navigate complex physical and social environments. Researchers studying cortical circuits underlying these capacities have traditionally favored the use of simplified tasks and brief stimulus presentations in order to isolate cognitive variables with tight experimental control. As a result, operational theories about visual brain function have come to emphasize feature detection, hierarchical stimulus encoding, top-down task modulation, and functional segregation in distinct cortical areas. Recently, however, experimental paradigms combining natural behavior with electrophysiological recordings have begun to offer a distinctly different portrait of how the brain takes in and analyzes its visual surroundings. The present article reviews recent work in this area, highlighting some of the more surprising findings in domains of social vision and spatial navigation along with shifts in thinking that have begun to emanate from this approach.

Low-cost, portable, easy-to-use kiosks to facilitate home-cage testing of nonhuman primates during vision-based behavioral tasks

Nonhuman primates (NHPs), especially rhesus macaques, have significantly contributed to our understanding of the neural computations underlying human vision. Besides the established homologies in the visual brain areas between these species and our ability to probe detailed neural mechanisms in monkeys at multiple scales, NHPs' ability to perform human-like visual behavior makes them an extremely appealing animal model of human vision. Traditionally, such behavioral studies have been conducted in controlled laboratory settings, offering experimenters tight control over variables like luminance, eye movements, and auditory interference. However, in-lab experiments have several constraints, including limited experimental time, the need for dedicated human experimenters, additional lab space requirements, invasive surgeries for headpost implants, and extra time and training for chairing and head restraints. To overcome these limitations, we propose adopting home-cage behavioral training and testing of NHPs, enabling the administration of many vision-based behavioral tasks simultaneously across multiple monkeys with reduced human personnel requirements, no NHP head restraint, and monkeys' unrestricted access to experiments. In this article, we present a portable, low-cost, easy-to-use kiosk system developed to conduct home-cage vision-based behavioral tasks in NHPs. We provide details of its operation and build to enable more open-source development of this technology. Furthermore, we present validation results using behavioral measurements performed in the lab and in NHP home cages, demonstrating the system's reliability and potential to enhance the efficiency and flexibility of NHP behavioral research.NEW & NOTEWORTHY Training nonhuman primates (NHPs) for vision-based behavioral tasks in a laboratory setting is a time-consuming process and comes with many limitations. To overcome these challenges, we have developed an affordable, open-source, wireless, touchscreen training system that can be placed in the NHPs' housing environment. This system enables NHPs to work at their own pace. It provides a platform to implement continuous behavioral training protocols without major experimenter intervention and eliminates the need for other standard practices like NHP chair training, collar placement, and head restraints. Hence, these kiosks ultimately contribute to animal welfare and therefore better-quality neuroscience in the long run. In addition, NHPs quickly learn complex behavioral tasks using this system, making it a promising tool for wireless electrophysiological research in naturalistic, unrestricted environments to probe the relation between brain and behavior.

Closed-loop microstimulations of the orbitofrontal cortex during real-life gaze interaction enhance dynamic social attention

Neurons from multiple prefrontal areas encode several key variables of social gaze interaction. To explore the causal roles of the primate prefrontal cortex in real-life gaze interaction, we applied weak closed-loop microstimulations that were precisely triggered by specific social gaze events. Microstimulations of the orbitofrontal cortex, but not the dorsomedial prefrontal cortex or the anterior cingulate cortex, enhanced momentary dynamic social attention in the spatial dimension by decreasing the distance of fixations relative to a partner's eyes and in the temporal dimension by reducing the inter-looking interval and the latency to reciprocate the other's directed gaze. By contrast, on a longer timescale, microstimulations of the dorsomedial prefrontal cortex modulated inter-individual gaze dynamics relative to one's own gaze positions. These findings demonstrate that multiple regions in the primate prefrontal cortex may serve as functionally accessible nodes in controlling different aspects of dynamic social attention and suggest their potential for a therapeutic brain interface.

Gaze and Arrows: Does the Gaze-Following Patch in the Posterior Temporal Cortex Differentiate Social and Symbolic Spatial Cues?

The gaze-following patch (GFP) is located in the posterior temporal cortex and has been described as a cortical module dedicated to processing other people's gaze-direction in a domain-specific manner. Thus, it appears to be the neural correlate of Baron-Cohen's eye direction detector (EDD) which is one of the core modules in his mindreading system-a neurocognitive model for the theory of mind concept. Inspired by Jerry Fodor's ideas on the modularity of the mind, Baron-Cohen proposed that, among other things, the individual modules are domain specific. In the case of the EDD, this means that it exclusively processes eye-like stimuli to extract gaze-direction and that other stimuli, which may carry directional information as well, are processed elsewhere. If the GFP is indeed EDD's neural correlate, it must meet this expectation. To test this, we compared the GFP's BOLD activity during gaze-direction following with the activity during arrow-direction following in the present human fMRI study. Contrary to the expectation based on the assumption of domain specificity, we did not find a differentiation between gaze- and arrow-direction following. In fact, we were not able to reproduce the GFP as presented in the previous studies. A possible explanation is that in the present study-unlike the previous work-the gaze stimuli did not contain an obvious change of direction that represented a visual motion. Hence, the critical stimulus component responsible for the identification of the GFP in the previous experiments might have been visual motion.

Closed-loop microstimulations of the orbitofrontal cortex during real-life gaze interaction enhance dynamic social attention

The prefrontal cortex is extensively involved in social exchange. During dyadic gaze interaction, multiple prefrontal areas exhibit neuronal encoding of social gaze events and context-specific mutual eye contact, supported by a widespread neural mechanism of social gaze monitoring. To explore causal manipulation of real-life gaze interaction, we applied weak closed-loop microstimulations that were precisely triggered by specific social gaze events to three prefrontal areas in monkeys. Microstimulations of orbitofrontal cortex (OFC), but not dorsomedial prefrontal or anterior cingulate cortex, enhanced momentary dynamic social attention in the spatial dimension by decreasing distance of one's gaze fixations relative to partner monkey's eyes. In the temporal dimension, microstimulations of OFC reduced the inter-looking interval for attending to another agent and the latency to reciprocate other's directed gaze. These findings demonstrate that primate OFC serves as a functionally accessible node in controlling dynamic social attention and suggest its potential for a therapeutic brain interface.

Causal Manipulation of Gaze-Following in the Macaque Temporal Cortex

Gaze-following, the ability to shift one's own attention to places or objects others are looking at, is essential for social interactions. Single unit recordings from the monkey cortex and neuroimaging work on the human and monkey brain suggest that a distinct region in the temporal cortex, the gaze-following patch (GFP), underpins this ability. Since previous studies of the GFP have relied on correlational techniques, it remains unclear whether gaze-following related activity in the GFP indicates a causal role rather than being just a reverberation of behaviorally relevant information produced elsewhere. To answer this question, we applied focal electrical and pharmacological perturbation to the GFP. Both approaches, when applied to the GFP, disrupted gaze-following if the monkeys had been instructed to follow gaze, along with the ability to suppress it if vetoed by the context. Hence the GFP is necessary for gaze-following as well as its cognitive control.

A frontoparietal network for volitional control of gaze following

Gaze following is a major element of non-verbal communication and important for successful social interactions. Human gaze following is a fast and almost reflex-like behaviour, yet it can be volitionally controlled and suppressed to some extent if inappropriate or unnecessary, given the social context. In order to identify the neural basis of the cognitive control of gaze following, we carried out an event-related fMRI experiment, in which human subjects' eye movements were tracked while they were exposed to gaze cues in two distinct contexts: A baseline gaze following condition in which subjects were instructed to use gaze cues to shift their attention to a gazed-at spatial target and a control condition in which the subjects were required to ignore the gaze cue and instead to shift their attention to a distinct spatial target to be selected based on a colour mapping rule, requiring the suppression of gaze following. We could identify a suppression-related blood-oxygen-level-dependent (BOLD) response in a frontoparietal network comprising dorsolateral prefrontal cortex (dlPFC), orbitofrontal cortex (OFC), the anterior insula, precuneus, and posterior parietal cortex (PPC). These findings suggest that overexcitation of frontoparietal circuits in turn suppressing the gaze following patch might be a potential cause of gaze following deficits in clinical populations.

Widespread implementations of interactive social gaze neurons in the primate prefrontal-amygdala networks

Social gaze interaction powerfully shapes interpersonal communication. However, compared with social perception, very little is known about the neuronal underpinnings of real-life social gaze interaction. Here, we studied a large number of neurons spanning four regions in primate prefrontal-amygdala networks and demonstrate robust single-cell foundations of interactive social gaze in the orbitofrontal, dorsomedial prefrontal, and anterior cingulate cortices, in addition to the amygdala. Many neurons in these areas exhibited high temporal heterogeneity for social discriminability, with a selectivity bias for looking at a conspecific compared with an object. Notably, a large proportion of neurons in each brain region parametrically tracked the gaze of self or other, providing substrates for social gaze monitoring. Furthermore, several neurons displayed selective encoding of mutual eye contact in an agent-specific manner. These findings provide evidence of widespread implementations of interactive social gaze neurons in the primate prefrontal-amygdala networks during social gaze interaction.

The role of temporal cortex in the control of attention

Attention is an indispensable component of active vision. Contrary to the widely accepted notion that temporal cortex processing primarily focusses on passive object recognition, a series of very recent studies emphasize the role of temporal cortex structures, specifically the superior temporal sulcus (STS) and inferotemporal (IT) cortex, in guiding attention and implementing cognitive programs relevant for behavioral tasks. The goal of this theoretical paper is to advance the hypothesis that the temporal cortex attention network (TAN) entails necessary components to actively participate in attentional control in a flexible task-dependent manner. First, we will briefly discuss the general architecture of the temporal cortex with a focus on the STS and IT cortex of monkeys and their modulation with attention. Then we will review evidence from behavioral and neurophysiological studies that support their guidance of attention in the presence of cognitive control signals. Next, we propose a mechanistic framework for executive control of attention in the temporal cortex. Finally, we summarize the role of temporal cortex in implementing cognitive programs and discuss how they contribute to the dynamic nature of visual attention to ensure flexible behavior.

Variability of neuronal responses in the posterior superior temporal sulcus predicts choice behavior during social interactions

Recent studies have shown that neural activity in a well-defined patch in the posterior superior temporal sulcus (the "gaze-following patch," GFP) of the primate brain is strongly modulated when the other's gaze attracts the observer's attention to locations/objects, the other is looking at. Changes of the mean discharge rate of neurons in the monkey GFP indicate that they are involved in two distinct computations: the allocation of spatial attention guided by the other's gaze vector and the suppression of gaze following if inappropriate in a given situation. Here, we asked if and how the discharge variability of neurons in the GFP is related to the task and if it carries information on behavioral performance. To this end, we calculated the Fano factor as a measure of across-trial discharge variability as a function of time. Our results show that all neurons exhibiting a task-related discharge-rate modulation also exhibit a stimulus onset-dependent drop in the Fano factor. Furthermore, the amplitude of the Fano factor reduction is modulated by task condition and the neuron's selectivity in this regard. We found that these effects are directly related to the monkeys' behavioral performance in that the Fano factor is predictive about upcoming correct or wrong decisions. Our results indicate that neuronal discharge variability as gauged by the Fano factor, hitherto primarily studied in the context of visual perception or motor control, is an informative measure also in studies of the neural underpinnings of complex social behavior.NEW & NOTEWORTHY Quenching of neural variability following stimulus onset is a widely accepted phenomenon. However, the relevance of quenching for the shaping of complex social behaviors remains to be explored. Here, we show that task selective neurons in the GFP exhibit a higher degree of variability quenching than their neighboring unselective neurons. Furthermore, we demonstrate that behavioral errors are not only associated with lower firing rates but also less variability quenching, suggesting that both facilitate optimal performance.

Does the temporal cortex make us human? A review of structural and functional diversity of the primate temporal lobe

Temporal cortex is a primate specialization that shows considerable variation in size, morphology, and connectivity across species. Human temporal cortex is involved in many behaviors that are considered especially well developed in humans, including semantic processing, language, and theory of mind. Here, we ask whether the involvement of temporal cortex in these behaviors can be explained in the context of the 'general' primate organization of the temporal lobe or whether the human temporal lobe contains unique specializations indicative of a 'step change' in the lineage leading to modern humans. We propose that many human behaviors can be explained as elaborations of temporal cortex functions observed in other primates. However, changes in temporal lobe white matter suggest increased integration of information within temporal cortex and between posterior temporal cortex and other association areas, which likely enable behaviors not possible in other species.

Levels of naturalism in social neuroscience research

In order to understand ecologically meaningful social behaviors and their neural substrates in humans and other animals, researchers have been using a variety of social stimuli in the laboratory with a goal of extracting specific processes in real-life scenarios. However, certain stimuli may not be sufficiently effective at evoking typical social behaviors and neural responses. Here, we review empirical research employing different types of social stimuli by classifying them into five levels of naturalism. We describe the advantages and limitations while providing selected example studies for each level. We emphasize the important trade-off between experimental control and ecological validity across the five levels of naturalism. Taking advantage of newly emerging tools, such as real-time videos, virtual avatars, and wireless neural sampling techniques, researchers are now more than ever able to adopt social stimuli at a higher level of naturalism to better capture the dynamics and contingency of real-life social interaction.

From the field to the lab and back: neuroethology of primate social behavior

Social mammals with more numerous and stronger social relationships live longer, healthier lives. Despite the established importance of social relationships, our understanding of the neurobiological mechanisms by which they are pursued, formed, and maintained in primates remains largely confined to highly controlled laboratory settings which do not allow natural, dynamic social interactions to unfold. In this review, we argue that the neurobiological study of primate social behavior would benefit from adopting a neuroethological approach, that is, a perspective grounded in natural, species-typical behavior, with careful selection of animal models according to the scientific question at hand. We highlight macaques and marmosets as key animal models for human social behavior and summarize recent findings in the social domain for both species. We then review pioneering studies of dynamic social behaviors in small animals, which can inspire studies in larger primates where the technological landscape is now ripe for an ethological overhaul.

Social interaction networks in the primate brain

Primate brains have evolved to understand and engage with their social world. Much about the structure of this world can be gleaned from social interactions. Circuits for the analysis of and participation in social interactions have now been mapped. Increased knowledge about their functional specializations and relative spatial locations promises to greatly improve the understanding of the functional organization of the primate social brain. Detailed electrophysiology, as in the case of the face-processing network, of local operations and functional interactions between areas is necessary to uncover neural mechanisms and computation principles of social cognition. New naturalistic behavioral paradigms, behavioral tracking, and new analytical approaches for parallel non-stationary data will be important components toward a neuroscientific theory of primates' interactive minds.

Frontal, Parietal, and Temporal Brain Areas Are Differentially Activated When Disambiguating Potential Objects of Joint Attention

Humans establish joint attention with others by following the other's gaze. Previous work has suggested that a cortical patch (gaze-following patch, GFP) close to the posterior superior temporal sulcus (pSTS) may serve as a link between the extraction of the other's gaze direction and the resulting shifts of attention, mediated by human lateral intraparietal area (hLIP). However, it is not clear how the brain copes with situations in which information on gaze direction alone is insufficient to identify the target object because more than one may lie along the gaze vector. In this fMRI study, we tested human subjects on a paradigm that allowed the identification of a target object based on the integration of the other's gaze direction and information provided by an auditory cue on the relevant object category. Whereas the GFP activity turned out to be fully determined by the use of gaze direction, activity in hLIP reflected the total information needed to pinpoint the target. Moreover, in an exploratory analysis, we found that a region in the inferior frontal junction (IFJ) was sensitive to the total information on the target. An examination of the BOLD time courses in the three identified areas suggests functionally complementary roles. Although the GFP seems to primarily process directional information stemming from the other's gaze, the IFJ may help to analyze the scene when gaze direction and auditory information are not sufficient to pinpoint the target. Finally, hLIP integrates both streams of information to shift attention to distinct spatial locations.

A Naturalistic Dynamic Monkey Head Avatar Elicits Species-Typical Reactions and Overcomes the Uncanny Valley

Research on social perception in monkeys may benefit from standardized, controllable, and ethologically valid renditions of conspecifics offered by monkey avatars. However, previous work has cautioned that monkeys, like humans, show an adverse reaction toward realistic synthetic stimuli, known as the "uncanny valley" effect. We developed an improved naturalistic rhesus monkey face avatar capable of producing facial expressions (fear grin, lip smack and threat), animated by motion capture data of real monkeys. For validation, we additionally created decreasingly naturalistic avatar variants. Eight rhesus macaques were tested on the various videos and avoided looking at less naturalistic avatar variants, but not at the most naturalistic or the most unnaturalistic avatar, indicating an uncanny valley effect for the less naturalistic avatar versions. The avoidance was deepened by motion and accompanied by physiological arousal. Only the most naturalistic avatar evoked facial expressions comparable to those toward the real monkey videos. Hence, our findings demonstrate that the uncanny valley reaction in monkeys can be overcome by a highly naturalistic avatar.