Where and how our brain represents the temporal structure of observed action

Predictive coding for the actions and emotions of others and its deficits in autism spectrum disorders

Traditionally, the neural basis of social perception has been studied by showing participants brief examples of the actions or emotions of others presented in randomized order to prevent participants from anticipating what others do and feel. This approach is optimal to isolate the importance of information flow from lower to higher cortical areas. The degree to which feedback connections and Bayesian hierarchical predictive coding contribute to how mammals process more complex social stimuli has been less explored, and will be the focus of this review. We illustrate paradigms that start to capture how participants predict the actions and emotions of others under more ecological conditions, and discuss the brain activity measurement methods suitable to reveal the importance of feedback connections in these predictions. Together, these efforts draw a richer picture of social cognition in which predictive coding and feedback connections play significant roles. We further discuss how the notion of predicting coding is influencing how we think of autism spectrum disorder.

Computational components of visual predictive coding circuitry

If a full visual percept can be said to be a 'hypothesis', so too can a neural 'prediction' - although the latter addresses one particular component of image content (such as 3-dimensional organisation, the interplay between lighting and surface colour, the future trajectory of moving objects, and so on). And, because processing is hierarchical, predictions generated at one level are conveyed in a backward direction to a lower level, seeking to predict, in fact, the neural activity at that prior stage of processing, and learning from errors signalled in the opposite direction. This is the essence of 'predictive coding', at once an algorithm for information processing and a theoretical basis for the nature of operations performed by the cerebral cortex. Neural models for the implementation of predictive coding invoke specific functional classes of neuron for generating, transmitting and receiving predictions, and for producing reciprocal error signals. Also a third general class, 'precision' neurons, tasked with regulating the magnitude of error signals contingent upon the confidence placed upon the prediction, i.e., the reliability and behavioural utility of the sensory data that it predicts. So, what is the ultimate source of a 'prediction'? The answer is multifactorial: knowledge of the current environmental context and the immediate past, allied to memory and lifetime experience of the way of the world, doubtless fine-tuned by evolutionary history too. There are, in consequence, numerous potential avenues for experimenters seeking to manipulate subjects' expectation, and examine the neural signals elicited by surprising, and less surprising visual stimuli. This review focuses upon the predictive physiology of mouse and monkey visual cortex, summarising and commenting on evidence to date, and placing it in the context of the broader field. It is concluded that predictive coding has a firm grounding in basic neuroscience and that, unsurprisingly, there remains much to learn.

Predictability alters information flow during action observation in human electrocorticographic activity

The action observation network (AON) has been extensively studied using short, isolated motor acts. How activity in the network is altered when these isolated acts are embedded in meaningful sequences of actions remains poorly understood. Here we utilized intracranial electrocorticography to characterize how the exchange of information across key nodes of the AON-the precentral, supramarginal, and visual cortices-is affected by such embedding and the resulting predictability. We found more top-down beta oscillation from precentral to supramarginal contacts during the observation of predictable actions in meaningful sequences compared to the same actions in randomized, and hence less predictable, order. In addition, we find that expectations enabled by the embedding lead to a suppression of bottom-up visual responses in the high-gamma range in visual areas. These results, in line with predictive coding, inform how nodes of the AON integrate information to process the actions of others.

Temporal hierarchy of observed goal-directed actions

During social interactions, we continuously integrate current and previous information over varying timescales to infer other people's action intentions. Motor cognition theories argue for a hierarchical organization of goal-directed actions based on temporal scales. Accordingly, transient motor primitives are represented at lower levels of the hierarchy, a combination of primitives building motor sequences at subordinate levels, and more stable overarching action goals at superordinate levels. A neural topography of hierarchal timescales for information accumulation was previously shown in the visual and auditory domains. However, whether such a temporal hierarchy can also account for observed goal-directed action representations in motor pathways remains to be determined. Thus, the current study examined the neural architecture underlying the processing of observed goal-directed actions using inter-subject correlation (ISC) of fMRI activity. Observers (n = 24) viewed sequential hand movements presented in their intact order or piecewise scrambled at three timescales pertaining to goal-directed action evolution (Primitives: ± 1.5 s, Sub-Goals: ± 4 s, and High-Goals: ± 10 s). The results revealed differential intrinsic temporal capacities for integrating goal-directed action information across brain areas engaged in action observation. Longer timescales (> ± 10 s) were found in the posterior parietal and dorsal premotor compared to the ventral premotor (± 4 s) and anterior parietal (± 1.5 s) cortex. Moreover, our results revealed a hemispheric bias with more extended timescales in the right MT+, primary somatosensory, and early visual cortices compared to their homotopic regions in the left hemisphere. Our findings corroborate a hierarchical neural mapping of observed actions based on temporal scales of goals and provide further support for a ubiquitous time-dependent neural organization of information processing across multiple modalities.

Body Form Modulates the Prediction of Human and Artificial Behaviour from Gaze Observation

The future of human–robot collaboration relies on people’s ability to understand and predict robots' actions. The machine-like appearance of robots, as well as contextual information, may influence people’s ability to anticipate the behaviour of robots. We conducted six separate experiments to investigate how spatial cues and task instructions modulate people’s ability to understand what a robot is doing. Participants observed goal-directed and non-goal directed gaze shifts made by human and robot agents, as well as directional cues displayed by a triangle. We report that biasing an observer's attention, by showing just one object an agent can interact with, can improve people’s ability to understand what humanoid robots will do. Crucially, this cue had no impact on people’s ability to predict the upcoming behaviour of the triangle. Moreover, task instructions that focus on the visual and motor consequences of the observed gaze were found to influence mentalising abilities. We suggest that the human-like shape of an agent and its physical capabilities facilitate the prediction of an upcoming action. The reported findings expand current models of gaze perception and may have important implications for human–human and human–robot collaboration.

Human mirror neuron system responsivity to unimodal and multimodal presentations of action

This study aims to clarify unresolved questions from two earlier studies by McGarry et al. Exp Brain Res 218(4): 527-538, 2012 and Kaplan and Iacoboni Cogn Process 8: 103-113, 2007 on human mirror neuron system (hMNS) responsivity to multimodal presentations of actions. These questions are: (1) whether the two frontal areas originally identified by Kaplan and Iacoboni (ventral premotor cortex [vPMC] and inferior frontal gyrus [IFG]) are both part of the hMNS (i.e., do they respond to execution as well as observation), (2) whether both areas yield effects of biologicalness (biological, control) and modality (audio, visual, audiovisual), and (3) whether the vPMC is preferentially responsive to multimodal input. To resolve these questions about the hMNS, we replicated and extended McGarry et al.'s electroencephalography (EEG) study, while incorporating advanced source localization methods. Participants were asked to execute movements (ripping paper) as well as observe those movements across the same three modalities (audio, visual, and audiovisual), all while 64-channel EEG data was recorded. Two frontal sources consistent with those identified in prior studies showed mu event-related desynchronization (mu-ERD) under execution and observation conditions. These sources also showed a greater response to biological movement than to control stimuli as well as a distinct visual advantage, with greater responsivity to visual and audiovisual compared to audio conditions. Exploratory analyses of mu-ERD in the vPMC under visual and audiovisual observation conditions suggests that the hMNS tracks the magnitude of visual movement over time.

20 Hz Transcranial Alternating Current Stimulation Inhibits Observation-Execution-Related Motor Cortex Excitability

The present study aimed to investigate the effect of transcranial alternating current stimulation (tACS) on the primary motor cortex (M1) during action observation, and subsequent action execution, on motor cortex excitability. The participants received tACS at 10 Hz or 20 Hz, or a sham stimulation over the left M1 for 10 min while they observed a video displaying a repeated button-tapping task using the right hand, and then performed an identical task with their right hand. Motor-evoked potential (MEP) amplitudes were measured before (T0) and after the action observation paired with tACS or a sham stimulation (T1), and after the performance of the action (T2). The results showed that MEPs were significantly reduced at time point T1 (p = 0.042, Cohen’s d = 0.611) and T2 (p = 0.0003, Cohen’s d = 0.852) in the 20 Hz tACS condition, in contrast with the sham stimulation. There was a significantly smaller MEP amplitude at time point T2 in the 20 Hz tACS condition, as compared to the 10 Hz tACS condition (p = 0.01, Cohen’s d = 0.622), but the MEP amplitude did not significantly change at time point T1 between the 20 Hz and 10 Hz tACS conditions (p = 0.136, Cohen’s d = 0.536). There were no significant differences at time point T1 and T2 between the 10 Hz tACS condition and the sham stimulation. We conclude that 20 Hz tACS during action observation inhibited motor cortex excitability and subsequently inhibited execution-related motor cortex excitability. The effects of tACS on task-related motor cortex excitability are frequency-dependent.

The human mirror neuron system-A common neural basis for social cognition?

According to the theory of embodied simulation, mirror neurons (MN) in our brain's motor system are the neuronal basis of all social-cognitive processes. The assumption of such a mirroring process in humans could be supported by results showing that within one person the same region is involved in different social cognition tasks. We conducted an fMRI-study with 75 healthy participants who completed three tasks: imitation, empathy, and theory of mind. We analyzed the data using group conjunction analyses and individual shared voxel counts. Across tasks, across and within participants, we find common activation in inferior frontal gyrus, inferior parietal cortex, fusiform gyrus, posterior superior temporal sulcus, and amygdala. Our results provide evidence for a shared neural basis for different social-cognitive processes, indicating that interpersonal understanding might occur by embodied simulation.

Action perception recruits the cerebellum and is impaired in patients with spinocerebellar ataxia

Our cerebellum has been proposed to generate prediction signals that may help us plan and execute our motor programmes. However, to what extent our cerebellum is also actively involved in perceiving the action of others remains to be elucidated. Using functional MRI, we show here that observing goal-directed hand actions of others bilaterally recruits lobules VI, VIIb and VIIIa in the cerebellar hemispheres. Moreover, whereas healthy subjects (n = 31) were found to be able to discriminate subtle differences in the kinematics of observed limb movements of others, patients suffering from spinocerebellar ataxia type 6 (SCA6; n = 21) were severely impaired in performing such tasks. Our data suggest that the human cerebellum is actively involved in perceiving the kinematics of the hand actions of others and that SCA6 patients’ deficits include a difficulty in perceiving the actions of other individuals. This finding alerts us to the fact that cerebellar disorders can alter social cognition.

The last chance to pass the ball: investigating the role of temporal expectation and motor resonance in processing temporal errors in motor actions

Humans can acquire information on others' motor outputs (action prediction) and intentions (action understanding) according to their individual motor repertoire and to the detected gesture's features (e.g. temporal patterns). We aimed at dissociating between action prediction and action understanding abilities in soccer players and novices observing soccer action videos including correct timing pass (CTP) or delayed pass (DP). First, we used an occluding paradigm to evaluate participants' ability to predict the correct time to pass the ball. Although soccer players showed reduced reaction times, all subjects showed a similar pattern of performance: during DP observation, responses appeared delayed with respect to the other conditions but anticipated with respect to the observed DP. In a separate experiment, we investigated the ability to recognize CTP vs DP and the modulation of primary motor cortex (M1) excitability associated to video observation. Only soccer players showed selective modulation of M1 according to the plausibility of the observed action, with increased excitability during the observation of the CTP and in a phase preceding the DP. In conclusion, action prediction ability seems to be independent from the individual motor repertoire. By contrast, only subjects with previously acquired sensorimotor skills are able to infer the observed action's long-term intention.

Measuring shared responses across subjects using intersubject correlation

Our capacity to jointly represent information about the world underpins our social experience. By leveraging one individual's brain activity to model another's, we can measure shared information across brains-even in dynamic, naturalistic scenarios where an explicit response model may be unobtainable. Introducing experimental manipulations allows us to measure, for example, shared responses between speakers and listeners or between perception and recall. In this tutorial, we develop the logic of intersubject correlation (ISC) analysis and discuss the family of neuroscientific questions that stem from this approach. We also extend this logic to spatially distributed response patterns and functional network estimation. We provide a thorough and accessible treatment of methodological considerations specific to ISC analysis and outline best practices.