Seeing the future: Natural image sequences produce “anticipatory” neuronal activity and bias perceptual report

Temporal continuity shapes visual responses of macaque face patch neurons

Macaque inferior temporal cortex neurons respond selectively to complex visual images, with recent work showing that they are also entrained reliably by the evolving content of natural movies. To what extent does temporal continuity itself shape the responses of high-level visual neurons? We addressed this question by measuring how cells in face-selective regions of the macaque visual cortex were affected by the manipulation of a movie's temporal structure. Sampling a 5-min movie at 1 s intervals, we measured neural responses to randomized, brief stimuli of different lengths, ranging from 800 ms dynamic movie snippets to 100 ms static frames. We found that the disruption of temporal continuity strongly altered neural response profiles, particularly in the early response period after stimulus onset. The results suggest that models of visual system function based on discrete and randomized visual presentations may not translate well to the brain's natural modes of operation.

Deficits in multi-scale top-down processes distorting auditory perception in schizophrenia

Cognitive models postulate that impaired source monitoring incorrectly weights the top-down prediction and bottom-up sensory processes and causes hallucinations. However, the underlying mechanisms of the interaction, such as whether the incorrectly weighting is ubiquitously on all levels of sensory features and whether different top-down processes have distinct effects in subgroups of schizophrenia are still unclear. This study investigates how multi-scale predictions influence perception of basic tonal features in schizophrenia. Sixty-three schizophrenia patients with and without symptoms of auditory verbal hallucinations (AVHs), and thirty healthy controls identified target tones in noise at the end of tone sequences. Predictions of different timescales were manipulated by either an alternating pattern in the preceding tone sequences (long-term regularity) or a repetition between the target tone and the tone immediately before (short-term repetition). The sensitivity index, d prime (d'), was obtained to assess the modulation of predictions on tone identification. Patients with AVHs showed higher d' when the target tones conformed to the long-term regularity of alternating pattern in the preceding tone sequence than when the target tones were inconsistent with the pattern. Whereas, the short-term repetition modulated the tone identification in patients without AVHs. Predictions did not influence tone identification in healthy controls. Our results suggest that impaired source monitoring in schizophrenia patients with AVHs heavily weights top-down predictions over bottom-up perceptual processes to form incorrect perception. The weighting function in source monitoring can extend to the processes of basic tonal features, and predictions at multiple timescales could differentially modulate perception in different clinical populations. The impaired interaction between top-down and bottom-up processes might underlie the development of hallucination symptoms in schizophrenia.

All-or-none face categorization in the human brain

Visual categorization is integral for our interaction with the natural environment. In this process, similar selective responses are produced to a class of variable visual inputs. Whether categorization is supported by partial (graded) or absolute (all-or-none) neural responses in high-level human brain regions is largely unknown. We address this issue with a novel frequency-sweep paradigm probing the evolution of face categorization responses between the minimal and optimal stimulus presentation times. In a first experiment, natural images of variable non-face objects were progressively swept from 120 to 3 ​Hz (8.33-333 ​ms duration) in rapid serial visual presentation sequences. Widely variable face exemplars appeared every 1 ​s, enabling an implicit frequency-tagged face-categorization electroencephalographic (EEG) response at 1 ​Hz. Face-categorization activity emerged with stimulus durations as brief as 17 ​ms (17-83 ​ms across individual participants) but was significant with 33 ​ms durations at the group level. The face categorization response amplitude increased until 83 ​ms stimulus duration (12 ​Hz), implying graded categorization responses. In a second EEG experiment, faces appeared non-periodically throughout such sequences at fixed presentation rates, while participants explicitly categorized faces. A strong correlation between response amplitude and behavioral accuracy across frequency rates suggested that dilution from missed categorizations, rather than a decreased response to each face stimulus, accounted for the graded categorization responses as found in Experiment 1. This was supported by (1) the absence of neural responses to faces that participants failed to categorize explicitly in Experiment 2 and (2) equivalent amplitudes and spatio-temporal signatures of neural responses to behaviorally categorized faces across presentation rates. Overall, these observations provide original evidence that high-level visual categorization of faces, starting at about 100 ​ms following stimulus onset in the human brain, is variable across observers tested under tight temporal constraints, but occurs in an all-or-none fashion.

Interactor's body shape does not affect visuo-motor interference effects during motor coordination

The biological-tuning of the Action Observation Network is highly debated. A current open question relates to whether the morphological appearance (body shape) and/or the biological motion of the observed agent triggers action simulation processes. Motor simulation of the partner's action is critical for motor interactions, where two partners coordinate their actions in space and time. It supports interpersonal alignment and facilitates online coordination. However, motor simulation also leads to visuo-motor interference effects when people are required to coordinate with complementary actions, i.e. incongruent movements as compared to the observed ones. Movement kinematics of interactive partners allows us to capture their automatic tendency to simulate and imitate the partner's complementary movements. In an online reach-to-grasp task, we investigated whether visuo-motor interference effects, visible in the kinematics of complementary movements, are modulated by the visual presence of the interactor's body shape. We asked participants to interact with 1) a humanoid agent with a human-like body shape and with real human, biological, movement kinematics, or 2) a non-humanoid agent, which did not resemble the human body-shape but moved with the same real kinematics. Through the combination of inferential and Bayesian statistics, the results show no effect of interactor's body shape on visuo-motor interference in reaching and grasping kinematics during online motor coordination. We discuss the results and propose that the kinematics of the observed movements, during motor interactions, might be the key factor for visuo-motor interference to take place independently from the morphological appearance of the partner. This is particularly relevant in a technological society that constantly asks humans to interact with artificial agents.

Neural Basis of Action Observation and Understanding From First- and Third-Person Perspectives: An fMRI Study

Understanding the intentions of others while observing their actions is a fundamental aspect of social behavior. However, the differences in neural and functional mechanisms between observing actions from the first-person perspective (1PP) and third-person perspective (3PP) are poorly understood. The present study had two aims: (1) to delineate the neural basis of action observation and understanding from the 1PP and 3PP; and (2) to identify whether there are different activation patterns during action observation and understanding from 1PP and 3PP. We used a blocked functional magnetic resonance imaging (fMRI) experimental design. Twenty-six right-handed participants observed interactions between the right hand and a cup from 1PP and 3PP. The results indicated that both 1PP and 3PP were associated with similar patterns of activation in key areas of the mirror neuron system underlying action observation and understanding. Importantly, besides of the core network of mirror neuron system, we also found that parts of the basal ganglia and limbic system were involved in action observation in both the 1PP and 3PP tasks, including the putamen, insula and hippocampus, providing a more complete understanding of the neural basis for action observation and understanding. Moreover, compared with the 3PP, the 1PP task caused more extensive and stronger activation. In contrast, the opposite comparison revealed that no regions exhibited significantly more activation in the 3PP compared with the 1PP condition. The current results have important implications for understanding the role of the core network underlying the mirror neuron system, as well as parts of the basal ganglia and limbic system, during action observation and understanding.

Neural Mechanisms of Social Cognition in Primates

Activity in a network of areas spanning the superior temporal sulcus, dorsomedial frontal cortex, and anterior cingulate cortex is concerned with how nonhuman primates negotiate the social worlds in which they live. Central aspects of these circuits are retained in humans. Activity in these areas codes for primates' interactions with one another, their attempts to find out about one another, and their attempts to prevent others from finding out too much about themselves. Moreover, important features of the social world, such as dominance status, cooperation, and competition, modulate activity in these areas. We consider the degree to which activity in these regions is simply encoding an individual's own actions and choices or whether this activity is especially and specifically concerned with social cognition. Recent advances in comparative anatomy and computational modeling may help us to gain deeper insights into the nature and boundaries of primate social cognition.

Rapid face adaptation distributes representation in inferior-temporal cortex across time and neuronal dimensions

Neuronal networks of the brain adapt their information processing according to the history of stimuli. Whereas most studies have linked adaptation to repetition suppression, recurrent connections within a network and disinhibition due to adaptation predict more complex response patterns. The main questions of this study are as follows: what is the effect of the selectivity of neurons on suppression/enhancement of neural responses? What are the consequences of adaptation on information representation in neural population and the temporal structure of response patterns? We studied rapid face adaptation using spiking activities of neurons in the inferior-temporal (IT) cortex. Investigating the responses of neurons, within a wide range from negative to positive face selectivity, showed that despite the peak amplitude suppression in highly positive selective neurons, responses were enhanced in most other neurons. This enhancement can be attributed to disinhibition due to adaptation. Delayed and distributed responses were observed for positive selective neurons. Principal component analysis of the IT population responses over time revealed that repetition of face stimuli resulted in temporal decorrelation of the network activity. The contributions of the main and higher neuronal dimensions were changed under an adaptation condition, where more neuronal dimensions were used to encode repeated face stimuli.

Hierarchical process memory: memory as an integral component of information processing

Models of working memory (WM) commonly focus on how information is encoded into and retrieved from storage at specific moments. However, in the majority of real-life processes, past information is used continuously to process incoming information across multiple timescales. Considering single-unit, electrocorticography, and functional imaging data, we argue that (i) virtually all cortical circuits can accumulate information over time, and (ii) the timescales of accumulation vary hierarchically, from early sensory areas with short processing timescales (10s to 100s of milliseconds) to higher-order areas with long processing timescales (many seconds to minutes). In this hierarchical systems perspective, memory is not restricted to a few localized stores, but is intrinsic to information processing that unfolds throughout the brain on multiple timescales.

Decoding the future from past experience: learning shapes predictions in early visual cortex

Learning the structure of the environment is critical for interpreting the current scene and predicting upcoming events. However, the brain mechanisms that support our ability to translate knowledge about scene statistics to sensory predictions remain largely unknown. Here we provide evidence that learning of temporal regularities shapes representations in early visual cortex that relate to our ability to predict sensory events. We tested the participants' ability to predict the orientation of a test stimulus after exposure to sequences of leftward- or rightward-oriented gratings. Using fMRI decoding, we identified brain patterns related to the observers' visual predictions rather than stimulus-driven activity. Decoding of predicted orientations following structured sequences was enhanced after training, while decoding of cued orientations following exposure to random sequences did not change. These predictive representations appear to be driven by the same large-scale neural populations that encode actual stimulus orientation and to be specific to the learned sequence structure. Thus our findings provide evidence that learning temporal structures supports our ability to predict future events by reactivating selective sensory representations as early as in primary visual cortex.

Virtual lesion of right posterior superior temporal sulcus modulates conscious visual perception of fearful expressions in faces and bodies

The posterior Superior Temporal Suclus (pSTS) represents a central hub in the complex cerebral network for person perception and emotion recognition as also suggested by its heavy connections with face- and body-specific cortical (e.g., the fusiform face area, FFA and the extrastriate body area, EBA) and subcortical structures (e.g., amygdala). Information on whether pSTS is causatively involved in sustaining conscious visual perception of emotions expressed by faces and bodies is lacking. We explored this issue by combining a binocular rivalry procedure (where emotional and neutral face and body postures rivaled with house images) with off-line, 1-Hz repetitive transcranial magnetic stimulation (rTMS). We found that temporary inhibition of the right pSTS reduced perceptual dominance of fearful faces and increased perceptual dominance of fearful bodies, while leaving unaffected the perception of neutral face and body images. Inhibition of the vertex had no effect on conscious visual perception of neutral or emotional face or body stimuli. Thus, the right pSTS plays a causal role in shortening conscious vision of fearful faces and in prolonging conscious vision of fearful bodies. These results suggest that pSTS selectively modulates the activity of segregated networks involved in the conscious visual perception of emotional faces or bodies. We speculate that the opposite role of the right pSTS for conscious perception of fearful face and body may be explained by the different connections that this region entertains with face- and body-selective visual areas as well as with amygdalae and premotor regions.

Contrasting specializations for facial motion within the macaque face-processing system

Facial motion transmits rich and ethologically vital information, but how the brain interprets this complex signal is poorly understood. Facial form is analyzed by anatomically distinct face patches in the macaque brain, and facial motion activates these patches and surrounding areas. Yet, it is not known whether facial motion is processed by its own distinct and specialized neural machinery, and if so, what that machinery's organization might be. To address these questions, we used fMRI to monitor the brain activity of macaque monkeys while they viewed low- and high-level motion and form stimuli. We found that, beyond classical motion areas and the known face patch system, moving faces recruited a heretofore unrecognized face patch. Although all face patches displayed distinctive selectivity for face motion over object motion, only two face patches preferred naturally moving faces, while three others preferred randomized, rapidly varying sequences of facial form. This functional divide was anatomically specific, segregating dorsal from ventral face patches, thereby revealing a new organizational principle of the macaque face-processing system.

Visual and Sensorimotor Contributions to the Esthetic Appraisal of Body Form, Motion, and Emotion

Recent neuroscience studies indicate that the visual processing of human bodies relies on a cortical network comprising different sensorimotor regions (extrastriate body area [EBA], superior temporal sulcus [STS], parietal cortex [PC], and premotor cortex [PM]). These regions seem to be specifically involved in the processing of morphological (form) and dynamic (movement) cues of the body. Importantly, the integrated activity within the network dedicated to body processing seems to underpin the unified perception of the body and its movements via simulation-like mechanisms (“cold embodiment”). Studies also suggest that regions within the body-related network are involved in the esthetic appreciation of human bodies together with a variety of cortical and subcortical regions associated to the emotional reward coding of stimuli (e.g., the amygdala for fear/disgust and the nucleus accumbens, the insula, and the cingulate cortex for pleasure reward), which may drive a form of “hot embodiment.” Thus, the esthetic evaluation of human bodies may rely upon a large cortico-subcortical network. Here we review evidence concerning the role of specific sensorimotor cortical and subcortical regions in the perception of beauty and attractiveness of the body. We conclude that exploring the way in which visual, sensorimotor, affective, and multisensory information in art and ecological life in general perturb our body representations is crucial for understanding the neural foundations of esthetic body appreciation.

Emotional and movement-related body postures modulate visual processing

Human body postures convey useful information for understanding others' emotions and intentions. To investigate at which stage of visual processing emotional and movement-related information conveyed by bodies is discriminated, we examined event-related potentials elicited by laterally presented images of bodies with static postures and implied-motion body images with neutral, fearful or happy expressions. At the early stage of visual structural encoding (N190), we found a difference in the sensitivity of the two hemispheres to observed body postures. Specifically, the right hemisphere showed a N190 modulation both for the motion content (i.e. all the observed postures implying body movements elicited greater N190 amplitudes compared with static postures) and for the emotional content (i.e. fearful postures elicited the largest N190 amplitude), while the left hemisphere showed a modulation only for the motion content. In contrast, at a later stage of perceptual representation, reflecting selective attention to salient stimuli, an increased early posterior negativity was observed for fearful stimuli in both hemispheres, suggesting an enhanced processing of motivationally relevant stimuli. The observed modulations, both at the early stage of structural encoding and at the later processing stage, suggest the existence of a specialized perceptual mechanism tuned to emotion- and action-related information conveyed by human body postures.

From tones in tinnitus to sensed social interaction in schizophrenia: how understanding cortical organization can inform the study of hallucinations and psychosis

The content, modality, and perceptual attributes of hallucinations and other psychotic symptoms may be related to neural representation at a single cell and population level in the cerebral cortex. A brief survey of some principles and examples of cortical representation and organization will be presented together with evidence for a correspondence between the neurobiology of brain areas activated at the time of a hallucination and the content of the corresponding hallucinatory and psychotic experiences. Contrasting the hallucinations of schizophrenia with other conditions, we highlight phenomenological aspects of hallucinations that are ignored in clinical practice but carry potentially important information about the brain regions and dysfunctions underlying them. Knowledge of cortical representation and organization are being used to develop animal models of hallucination and to test treatments that are now beginning to translate to the clinical domain.

Neuroanatomical substrates of action perception and understanding: an anatomic likelihood estimation meta-analysis of lesion-symptom mapping studies in brain injured patients

Several neurophysiologic and neuroimaging studies suggested that motor and perceptual systems are tightly linked along a continuum rather than providing segregated mechanisms supporting different functions. Using correlational approaches, these studies demonstrated that action observation activates not only visual but also motor brain regions. On the other hand, brain stimulation and brain lesion evidence allows tackling the critical question of whether our action representations are necessary to perceive and understand others’ actions. In particular, recent neuropsychological studies have shown that patients with temporal, parietal, and frontal lesions exhibit a number of possible deficits in the visual perception and the understanding of others’ actions. The specific anatomical substrates of such neuropsychological deficits however, are still a matter of debate. Here we review the existing literature on this issue and perform an anatomic likelihood estimation meta-analysis of studies using lesion-symptom mapping methods on the causal relation between brain lesions and non-linguistic action perception and understanding deficits. The meta-analysis encompassed data from 361 patients tested in 11 studies and identified regions in the inferior frontal cortex, the inferior parietal cortex and the middle/superior temporal cortex, whose damage is consistently associated with poor performance in action perception and understanding tasks across studies. Interestingly, these areas correspond to the three nodes of the action observation network that are strongly activated in response to visual action perception in neuroimaging research and that have been targeted in previous brain stimulation studies. Thus, brain lesion mapping research provides converging causal evidence that premotor, parietal and temporal regions play a crucial role in action recognition and understanding.

Neural underpinnings of superior action prediction abilities in soccer players

The ability to form anticipatory representations of ongoing actions is crucial for effective interactions in dynamic environments. In sports, elite athletes exhibit greater ability than novices in predicting other players' actions, mainly based on reading their body kinematics. This superior perceptual ability has been associated with a modulation of visual and motor areas by visual and motor expertise. Here, we investigated the causative role of visual and motor action representations in experts' ability to predict the outcome of soccer actions. We asked expert soccer players (outfield players and goalkeepers) and novices to predict the direction of the ball after perceiving the initial phases of penalty kicks that contained or not incongruent body kinematics. During the task, we applied repetitive transcranial magnetic stimulation (rTMS) over the superior temporal sulcus (STS) and the dorsal premotor cortex (PMd). Results showed that STS-rTMS disrupted performance in both experts and novices, especially in those with greater visual expertise (i.e. goalkeepers). Conversely, PMd-rTMS impaired performance only in expert players (i.e. outfield players and goalkeepers), who exhibit strong motor expertise into facing domain-specific actions in soccer games. These results provide causative evidence of the complimentary functional role of visual and motor action representations in experts' action prediction.

Does that look heavy to you? Perceived weight judgment in lifting actions in younger and older adults

When interpreting other people's movements or actions, observers may not only rely on the visual cues available in the observed movement, but they may also be able to "put themselves in the other person's shoes" by engaging brain systems involved in both "mentalizing" and motor simulation. The ageing process brings changes in both perceptual and motor abilities, yet little is known about how these changes may affect the ability to accurately interpret other people's actions. Here we investigated the effect of ageing on the ability to discriminate the weight of objects based on the movements of actors lifting these objects. Stimuli consisted of videos of an actor lifting a small box weighing 0.05-0.9 kg or a large box weighting 3-18 kg. In a four-alternative forced-choice task, younger and older participants reported the perceived weight of the box in each video. Overall, older participants were less sensitive than younger participants in discriminating the perceived weight of lifted boxes, an effect that was especially pronounced in the small box condition. Weight discrimination performance was better for the large box compared to the small box in both groups, due to greater saliency of the visual cues in this condition. These results suggest that older adults may require more salient visual cues to interpret the actions of others accurately. We discuss the potential contribution of age-related changes in visual and motor function on the observed effects and suggest that older adults' decline in the sensitivity to subtle visual cues may lead to greater reliance on visual analysis of the observed scene and its semantic context.

Reflections on agranular architecture: predictive coding in the motor cortex

•

Predictive coding explains the recursive hierarchical structure of cortical processes.

•

Granular layer 4, which relays ascending cortical pathways, is absent from motor cortex.

•

Perceptual inference results if ascending sensory data modify sensory predictions action, if spinal reflexes enact descending motor and/or proprioceptive predictions.

•

Motor layer 4 regresses as motor predictions inherently require less modification.

The agranular architecture of motor cortex lacks a functional interpretation. Here, we consider a ‘predictive coding’ account of this unique feature based on asymmetries in hierarchical cortical connections. In sensory cortex, layer 4 (the granular layer) is the target of ascending pathways. We theorise that the operation of predictive coding in the motor system (a process termed ‘active inference’) provides a principled rationale for the apparent recession of the ascending pathway in motor cortex. The extension of this theory to interlaminar circuitry also accounts for a sub-class of ‘mirror neuron’ in motor cortex – whose activity is suppressed when observing an action –explaining how predictive coding can gate hierarchical processing to switch between perception and action.

Hippocampal physiology, structure and function and the neuroscience of schizophrenia: a unified account of declarative memory deficits, working memory deficits and schizophrenic symptoms

Memory impairment is a consistent feature of the schizophrenic syndrome. Hippocampal dysfunction has also been consistently demonstrated. This review will discuss neurophysiological and neuroanatomical aspects of memory formation and how they relate to memory impairment in schizophrenia. An understanding of the cellular physiology and connectivity of the hippocampus with other regions can also aid in understanding the relationship between schizophrenic declarative or relational memory deficits, working memory deficits and the clinical symptoms of the syndrome.

Action prediction in younger versus older adults: neural correlates of motor familiarity

Generating predictions during action observation is essential for efficient navigation through our social environment. With age, the sensitivity in action prediction declines. In younger adults, the action observation network (AON), consisting of premotor, parietal and occipitotemporal cortices, has been implicated in transforming executed and observed actions into a common code. Much less is known about age-related changes in the neural representation of observed actions. Using fMRI, the present study measured brain activity in younger and older adults during the prediction of temporarily occluded actions (figure skating elements and simple movement exercises). All participants were highly familiar with the movement exercises whereas only some participants were experienced figure skaters. With respect to the AON, the results confirm that this network was preferentially engaged for the more familiar movement exercises. Compared to younger adults, older adults recruited visual regions to perform the task and, additionally, the hippocampus and caudate when the observed actions were familiar to them. Thus, instead of effectively exploiting the sensorimotor matching properties of the AON, older adults seemed to rely predominantly on the visual dynamics of the observed actions to perform the task. Our data further suggest that the caudate played an important role during the prediction of the less familiar figure skating elements in better-performing groups. Together, these findings show that action prediction engages a distributed network in the brain, which is modulated by the content of the observed actions and the age and experience of the observer.

Time perception during apparent biological motion reflects subjective speed of movement, not objective rate of visual stimulation

We have investigated links between biological motion perception and time perception. Participants compared the durations of two paired visual frames, inside which task-irrelevant sequences of static body postures were presented. The sequences produced apparent movements of shorter and longer path lengths, depending on the sequential order of body postures (ABC or ACB). Shorter and longer path lengths were paired with shorter and longer interstimulus intervals (ISIs) to produce path/ISI congruent sequences with intermediate subjective speeds and path/ISI incongruent sequences with slowest and fastest subjective speeds. Participants compared the duration of the visual frames surrounding these sequences; body postures and biological motion were irrelevant. The ability to discriminate the duration of the frames (as measured by the just noticeable difference, JND) was reduced for pairs of path/ISI congruent sequences as compared to pairs of path/ISI incongruent sequences. That is, duration discrimination improved when implied speed differed between the two sequences of a pair compared to when the implied speed was the same. Since stimuli showed no actual movement and were fully matched for lower-level visual input and objective stimulus durations, our findings suggest an involvement of higher-order visual or even motor areas in temporal biases during apparent biological motion perception. We show that apparent speed is the primary dimension of such percepts consistent with a dominant role of movement dynamics in the perception of other people's actions. Our results also confirm an intimate relation between time perception and processing of human movement.

Effect of adaptation on object representation accuracy in macaque inferior temporal cortex

Stimulus repetition produces a decrease of the response in many cortical areas and different modalities. This adaptation is highly prominent in macaque inferior temporal (IT) neurons. Here we ask how these repetition-induced changes in IT responses affect the accuracy by which IT neurons encode objects. This question bears on the functional consequences of adaptation, which are still unclear. We recorded the responses of single IT neurons to sequences of familiar shapes, each shown for 300 msec with an ISI of the same duration. The difference in shape between the two successively presented stimuli,that is, adapter and test, varied parametrically. The discriminability of the test stimuli was reduced for repeated compared with nonrepeated stimuli. In some conditions for which adapter and test shapes differed, the cross-adaptation resulted in an enhanced discriminability. These single cell results were confirmed in a second experiment in which we recorded multiunit spiking activity using a laminar microelectrode in macaque IT. Two familiar stimuli were presented successively for 500 msec each and separated with an ISI of the same duration. Trials consisted either of a repetition of the same stimulus or of their alternation. Small neuronal populations showed decreased classification accuracy for repeated compared with nonrepeated test stimuli, but classification was enhanced for the test compared with adapter stimuli when the test stimulus differed from recently seen stimuli. These findings suggest that short-term, stimulus-specific adaptation in IT supports efficient coding of stimuli that differ from recently seen ones while impairing the coding of repeated stimuli.

Beyond face value: does involuntary emotional anticipation shape the perception of dynamic facial expressions?

Emotional facial expressions are immediate indicators of the affective dispositions of others. Recently it has been shown that early stages of social perception can already be influenced by (implicit) attributions made by the observer about the agent's mental state and intentions. In the current study possible mechanisms underpinning distortions in the perception of dynamic, ecologically-valid, facial expressions were explored. In four experiments we examined to what extent basic perceptual processes such as contrast/context effects, adaptation and representational momentum underpinned the perceptual distortions, and to what extent 'emotional anticipation', i.e. the involuntary anticipation of the other's emotional state of mind on the basis of the immediate perceptual history, might have played a role. Neutral facial expressions displayed at the end of short video-clips, in which an initial facial expression of joy or anger gradually morphed into a neutral expression, were misjudged as being slightly angry or slightly happy, respectively (Experiment 1). This response bias disappeared when the actor's identity changed in the final neutral expression (Experiment 2). Videos depicting neutral-to-joy-to-neutral and neutral-to-anger-to-neutral sequences again produced biases but in opposite direction (Experiment 3). The bias survived insertion of a 400 ms blank (Experiment 4). These results suggested that the perceptual distortions were not caused by any of the low-level perceptual mechanisms (adaptation, representational momentum and contrast effects). We speculate that especially when presented with dynamic, facial expressions, perceptual distortions occur that reflect 'emotional anticipation' (a low-level mindreading mechanism), which overrules low-level visual mechanisms. Underpinning neural mechanisms are discussed in relation to the current debate on action and emotion understanding.

Neural responses to expression and gaze in the posterior superior temporal sulcus interact with facial identity

Neural models of human face perception propose parallel pathways. One pathway (including posterior superior temporal sulcus, pSTS) is responsible for processing changeable aspects of faces such as gaze and expression, and the other pathway (including the fusiform face area, FFA) is responsible for relatively invariant aspects such as identity. However, to be socially meaningful, changes in expression and gaze must be tracked across an individual face. Our aim was to investigate how this is achieved. Using functional magnetic resonance imaging, we found a region in pSTS that responded more to sequences of faces varying in gaze and expression in which the identity was constant compared with sequences in which the identity varied. To determine whether this preferential response to same identity faces was due to the processing of identity in the pSTS or was a result of interactions between pSTS and other regions thought to code face identity, we measured the functional connectivity between face-selective regions. We found increased functional connectivity between the pSTS and FFA when participants viewed same identity faces compared with different identity faces. Together, these results suggest that distinct neural pathways involved in expression and identity interact to process the changeable features of the face in a socially meaningful way.

Task contingencies and perceptual strategies shape behavioral effects on neuronal response profiles

We presented optic flow simulating eight directions of self-movement in the ground plane, while monkeys performed delayed match-to-sample tasks, and we recorded dorsal medial superior temporal (MSTd) neuronal activity. Randomly selected sample headings yield smaller test responses to the neuron's preferred heading when it is near the sample's heading direction and larger test responses to the preferred heading when it is far from the sample's heading. Limiting test stimuli to matching or opposite headings suppresses responses to preferred stimuli in both test conditions, whereas focusing on each neuron's preferred vs. antipreferred stimuli enhances responses to the antipreferred stimulus. Match vs. opposite paradigms create bimodal heading profiles shaped by interactions with late delay-period activity. We conclude that task contingencies, determining the prior probabilities of specific stimuli, interact with the monkeys' perceptual strategy for optic flow analysis. These influences shape attentional and working memory effects on the heading direction selectivities and preferences of MSTd neurons.

Hippocampal temporal-parietal junction interaction in the production of psychotic symptoms: a framework for understanding the schizophrenic syndrome

A framework is described for understanding the schizophrenic syndrome at the brain systems level. It is hypothesized that over-activation of dynamic gesture and social perceptual processes in the temporal-parietal occipital junction (TPJ), posterior superior temporal sulcus (PSTS) and surrounding regions produce the syndrome (including positive and negative symptoms, their prevalence, prodromal signs, and cognitive deficits). Hippocampal system hyper-activity and atrophy have been consistently found in schizophrenia. Hippocampal activity is highly correlated with activity in the TPJ and may be a source of over-excitation of the TPJ and surrounding regions. Strong evidence for this comes from in-vivo recordings in humans during psychotic episodes. Many positive symptoms of schizophrenia can be reframed as the erroneous sense of a presence or other who is observing, acting, speaking, or controlling; these qualia are similar to those evoked during abnormal activation of the TPJ. The TPJ and PSTS play a key role in the perception (and production) of dynamic social, emotional, and attentional gestures for the self and others (e.g., body/face/eye gestures, audiovisual speech and prosody, and social attentional gestures such as eye gaze). The single cell representation of dynamic gestures is multimodal (auditory, visual, tactile), matching the predominant hallucinatory categories in schizophrenia. Inherent in the single cell perceptual signal of dynamic gesture representations is a computation of intention, agency, and anticipation or expectancy (for the self and others). Stimulation of the TPJ resulting in activation of the self representation has been shown to result a feeling of a presence or multiple presences (due to heautoscopy) and also bizarre tactile experiences. Neurons in the TPJ are also tuned, or biased to detect threat related emotions. Abnormal over-activation in this system could produce the conscious hallucination of a voice (audiovisual speech), a person or a touch. Over-activation could interfere with attentional/emotional gesture perception and production (negative symptoms). It could produce the unconscious feeling of being watched, followed, or of a social situation unfolding along with accompanying abnormal perception of intent and agency (delusions). Abnormal activity in the TPJ would also be predicted to create several cognitive disturbances that are characteristic of schizophrenia, including abnormalities in attention, predictive social processing, working memory, and a bias to erroneously perceive threat.

Event-related repetitive transcranial magnetic stimulation of posterior superior temporal sulcus improves the detection of threatening postural changes in human bodies

Perceiving others' emotions through their body movements and postures is crucial for successful social interaction. While imaging studies indicate that perceiving body emotions relies upon a wide network of subcortico-cortical neural regions, little is known on the causative role of different nodes of this network. We applied event-related repetitive transcranial magnetic stimulation (rTMS) over nonfacial, body- and action-related extrastriate (EBA), temporal (pSTS), and premotor (vPM) cortices to test their active contribution in perceiving changes between two successive images of either threatening or neutral human body or animal postures. While stimulation of EBA and vPM showed no selective effect on threatening stimuli with respect to neutral ones, rTMS over pSTS selectively impaired neutral posture detection and increased the accuracy in detecting changes of threatening human postures with respect to all other experimental conditions. No such effect was found for animal stimuli. These results support the notion that pSTS is crucially devoted to the detection of socially relevant information concerning others' actions, fostering the notion that amygdalo-temporo-cortical modulatory connections mediate perception of emotionally salient body postures.

Schizophrenia as a disorder of social communication

Evidence is reviewed for the existence of a core system for moment-to-moment social communication that is based on the perception of dynamic gestures and other social perceptual processes in the temporal-parietal occipital junction (TPJ), including the posterior superior temporal sulcus (PSTS) and surrounding regions. Overactivation of these regions may produce the schizophrenic syndrome. The TPJ plays a key role in the perception and production of dynamic social, emotional, and attentional gestures for the self and others. These include dynamic gestures of the body, face, and eyes as well as audiovisual speech and prosody. Many negative symptoms are characterized by deficits in responding within these domains. Several properties of this system have been discovered through single neuron recording, brain stimulation, neuroimaging, and the study of neurological impairment. These properties map onto the schizophrenic syndrome. The representation of dynamic gestures is multimodal (auditory, visual, and tactile), matching the predominant hallucinatory categories in schizophrenia. Inherent in the perceptual signal of gesture representation is a computation of intention, agency, and anticipation or expectancy (for the self and others). The neurons are also tuned or biased to rapidly detect threat-related emotions. I review preliminary evidence that overactivation of this system can result in schizophrenia.

Representing others' actions: the role of expertise in the aging mind

A large body of evidence suggests that action execution and action observation share a common representational domain. To date, little is known about age-related changes in these action representations that are assumed to support various abilities such as the prediction of observed actions. The purpose of the present study was to investigate (a) how age affects the ability to predict the time course of observed actions; and (b) whether and to what extent sensorimotor expertise attenuates age-related declines in prediction performance. In a first experiment, older adults predicted the time course of familiar everyday actions less precisely than younger adults. In a second experiment, younger and older figure skating experts as well as age-matched novices were asked to predict the time course of figure skating elements and simple movement exercises. Both young age and sensorimotor expertise had a positive influence on prediction performance of figure skating elements. The expertise-related benefit did not show a transfer to movement exercises. Together, the results suggest a specific decline of action representations in the aging mind. However, extensive sensorimotor experience seems to enable experts to represent actions from their domain of expertise more precisely even in older age.

Predictive Context Influences Perceptual Selection during Binocular Rivalry

Prediction may be a fundamental principle of sensory processing: it has been proposed that the brain continuously generates predictions about forthcoming sensory information. However, little is known about how prediction contributes to the selection of a conscious percept from among competing alternatives. Here, we used binocular rivalry to investigate the effects of prediction on perceptual selection. In binocular rivalry, incompatible images presented to the two eyes result in a perceptual alternation between the images, even though the visual stimuli remain constant. If predictive signals influence the competition between neural representations of rivalrous images, this influence should generate a bias in perceptual selection that depends on predictive context. To manipulate predictive context, we developed a novel binocular rivalry paradigm in which rivalrous test images were immediately preceded by a sequence of context images presented identically to the two eyes. One of the test images was consistent with the preceding image sequence (it was the expected next image in the series), and the other was inconsistent (non-predicted). We found that human observers were more likely to perceive the consistent image at the onset of rivalry, suggesting that predictive context biased selection in favor of the predicted percept. This prediction effect was distinct from the effects of adaptation to stimuli presented before the binocular rivalry test. In addition, perceptual reports were speeded for predicted percepts relative to non-predicted percepts. These results suggest that predictive signals related to visual stimulus history exist at neural sites that can bias conscious perception during binocular rivalry. Our paradigm provides a new way to study how prior information and incoming sensory information combine to generate visual percepts.

Statistical learning of visual transitions in monkey inferotemporal cortex

One of the most fundamental functions of the brain is to predict upcoming events on the basis of the recent past. A closely related function is to signal when a prediction has been violated. The identity of the brain regions that mediate these functions is not known. We set out to determine whether they are implemented at the level of single neurons in the visual system. We gave monkeys prolonged exposure to pairs of images presented in fixed sequence so that each leading image became a strong predictor for the corresponding trailing image. We then monitored the responses of neurons in the inferotemporal cortex to image sequences that obeyed or violated the transitional rules imposed during training. Inferotemporal neurons exhibited a transitional surprise effect, responding much more strongly to unpredicted transitions than to predicted transitions. Thus, neurons even in the visual system make experience-based predictions and react when they fail.

From single cells to social perception

Research describing the cellular coding of faces in non-human primates often provides the underlying physiological framework for our understanding of face processing in humans. Models of face perception, explanations of perceptual after-effects from viewing particular types of faces, and interpretation of human neuroimaging data rely on monkey neurophysiological data and the assumption that neurophysiological responses of humans are comparable to those recorded in the non-human primate. Here, we review studies that describe cells that preferentially respond to faces, and assess the link between the physiological characteristics of single cells and social perception. Principally, we describe cells recorded from the non-human primate, although a limited number of cells have been recorded in humans, and are included in order to appraise the validity of non-human physiological data for our understanding of human face and social perception.

Detecting temporal reversals in human locomotion

An experiment investigated the ability by human observers to detect temporal reversals in dynamic displays of human locomotion. We video-taped the lower portion of the body of actors walking at their preferred speed either in the normal, forward direction (FW) or in the backward direction (BW). The videos were presented in a random order either as recorded (N) or in reverse (R). In one session, we presented both normal and time-reversed stimuli in the original upright orientation. In a second session, the stimuli were rotated by 180° around the horizontal axis. Observers were informed that the real recorded movement was either forward or backward and were asked to decide whether or not the movement had been time-reversed prior to the presentation. Although the kinematics of forward and backward human locomotion is quite similar, the detection of temporal reversals followed a consistent pattern showing a good accuracy in condition FW-N and a reduced but still above-chance performance in condition BW-R (by design, in both conditions actors appeared to walk forward). Performance was instead at chance level in the other two conditions where the apparent direction of the movement was backward. Inverting the spatial orientation of the stimuli reduced but did not suppress the ability to detect temporal reversals in the two conditions with apparent forward direction of movement. It is argued that implicit motor competence is at least in part instrumental for extracting the subtle discriminal information from the stimuli.

Involvement of low-level visual areas in hemispheric superiority for face processing

Previous studies on laterality in face processing have indicated superiority of the right hemisphere in discriminating and recognizing faces; however, the reasons for this feature are poorly understood. We employed functional MRI (fMRI) to elucidate the origin of this feature and used a paired-stimulus paradigm in which face pairs were presented unilaterally at the left or right visual hemifield of the participants. Each face in a pair was at a different position in the visual field. Refractory suppression in the fMRI response was observed bilaterally at the fusiform face area (FFA) for the same face pairs when pictures were presented in the left visual hemifield. In contrast, suppression was observed bilaterally at the FFA for the different as well as for the same face pairs when pictures were presented in the right visual hemifield. This pattern indicated inferior discrimination ability for paired stimuli presented to the right visual hemifield. These observations, along with changes in bilaterally interlocked responses at the FFA, suggest that low-level visual areas, and not high-level face areas, are strongly associated with the superiority of the right hemisphere in face processing.

Distinct mechanisms for coding of visual actions in macaque temporal cortex

Temporal cortical neurons are known to respond to visual dynamic-action displays. Many human psychophysical and functional imaging studies examining biological motion perception have used treadmill walking, in contrast to previous macaque single-cell studies. We assessed the coding of locomotion in rhesus monkey (Macaca mulatta) temporal cortex using movies of stationary walkers, varying both form and motion (i.e., different facing directions) or varying only the frame sequence (i.e., forward vs backward walking). The majority of superior temporal sulcus and inferior temporal neurons were selective for facing direction, whereas a minority distinguished forward from backward walking. Support vector machines using the temporal cortical population responses as input classified facing direction well, but forward and backward walking less so. Classification performance for the latter improved markedly when the within-action response modulation was considered, reflecting differences in momentary body poses within the locomotion sequences. Responses to static pose presentations predicted the responses during the course of the action. Analyses of the responses to walking sequences wherein the start frame was varied across trials showed that some neurons also carried a snapshot sequence signal. Such sequence information was present in neurons that responded to static snapshot presentations and in neurons that required motion. Our data suggest that actions are analyzed by temporal cortical neurons using distinct mechanisms. Most neurons predominantly signal momentary pose. In addition, temporal cortical neurons, including those responding to static pose, are sensitive to pose sequence, which can contribute to the signaling of learned action sequences.

Visual Aftereffects for Walking Actions Reveal Underlying Neural Mechanisms for Action Recognition

The results of this study illustrate a new high-level visual aftereffect: Observing actors walking forward, without horizontal translation, makes subsequent actors appear to walk backward, and the opposite effect is obtained after observing backward walking. We used this aftereffect, which cannot be explained by simple low-level adaptation to motion direction, to investigate the properties of neural mechanisms underlying recognition of walking actions. Our results suggest that the perception of walking and the perception of static images of actors in walking postures rely on common brain mechanisms that are primarily object centered, rather than viewer centered, and that are blind to the identity of the actor. These results, obtained with human psychophysical adaptation techniques, support previous evidence accumulated using single-unit recording in nonhuman primates. In addition, these results provide evidence that current models of human action recognition require an object-centered processing stage.

Visual stimulation decorrelates neuronal activity

The accuracy of neuronal encoding depends on the response statistics of individual neurons and the correlation of the activity between different neurons. Here, the dynamics of the neuronal response statistics in the anterior superior temporal sulcus of the macaque monkey is described. A transient reduction in the normalized trial-by-trial variability and decorrelation of the responses with both the activity of other neurons and previous activity of the same neuron are found at response onset. The variability of neuronal activity and its correlation structure return to the levels observed in the resting state 50-100 ms after response onset, except for marked increases in the signal correlation between neurons. The transient changes in the response statistics are seen even if there is little or no stimulus-elicited activity, indicating the effect is due to network properties rather than to activity changes per se. Modeling also indicates that the observed variations in response variability and correlation structure of the neuronal activity over time cannot be attributed to changes in firing rate. However, a reset of the underlying spike-generating process, possibly due to the driving input changing from recurrent to feedforward inputs, captures most of the observed changes. The nonstationarity indicated by the changes in correlation structure around response onset increases coding efficiency: compared with the mutual information calculated without regard to the transitory changes, the decorrelation increases the information conveyed by the initial response of modeled neuronal pairs by ≤ 4% and suggests that an integration time of as little as 50 ms is sufficient to extract 95% the available information during the initial response period.

Stimulus predictability reduces responses in primary visual cortex

In this functional magnetic resonance imaging study we tested whether the predictability of stimuli affects responses in primary visual cortex (V1). The results of this study indicate that visual stimuli evoke smaller responses in V1 when their onset or motion direction can be predicted from the dynamics of surrounding illusory motion. We conclude from this finding that the human brain anticipates forthcoming sensory input that allows predictable visual stimuli to be processed with less neural activation at early stages of cortical processing.

fMRI–Adaptation Studies of Viewpoint Tuning in the Extrastriate and Fusiform Body Areas

People are easily able to perceive the human body across different viewpoints, but the neural mechanisms underpinning this ability are currently unclear. In three experiments, we used functional MRI (fMRI) adaptation to study the view-invariance of representations in two cortical regions that have previously been shown to be sensitive to visual depictions of the human body—the extrastriate and fusiform body areas (EBA and FBA). The BOLD response to sequentially presented pairs of bodies was treated as an index of view invariance. Specifically, we compared trials in which the bodies in each image held identical poses (seen from different views) to trials containing different poses. EBA and FBA adapted to identical views of the same pose, and both showed a progressive rebound from adaptation as a function of the angular difference between views, up to ∼30°. However, these adaptation effects were eliminated when the body stimuli were followed by a pattern mask. Delaying the mask onset increased the response (but not the adaptation effect) in EBA, leaving FBA unaffected. We interpret these masking effects as evidence that view-dependent fMRI adaptation is driven by later waves of neuronal responses in the regions of interest. Finally, in a whole brain analysis, we identified an anterior region of the left inferior temporal sulcus (l-aITS) that responded linearly to stimulus rotation, but showed no selectivity for bodies. Our results show that body-selective cortical areas exhibit a similar degree of view-invariance as other object selective areas—such as the lateral occipitotemporal area (LO) and posterior fusiform gyrus (pFs).