Putting actions in context: visual action adaptation aftereffects are modulated by social contexts

Isolating Action Prediction from Action Integration in the Perception of Social Interactions

Previous research suggests that predictive mechanisms are essential in perceiving social interactions. However, these studies did not isolate action prediction (a priori expectations about how partners in an interaction react to one another) from action integration (a posteriori processing of both partner's actions). This study investigated action prediction during social interactions while controlling for integration confounds. Twenty participants viewed 3D animations depicting an action-reaction interaction between two actors. At the start of each action-reaction interaction, one actor performs a social action. Immediately after, instead of presenting the other actor's reaction, a black screen covers the animation for a short time (occlusion duration) until a still frame depicting a precise moment of the reaction is shown (reaction frame). The moment shown in the reaction frame is either temporally aligned with the occlusion duration or deviates by 150 ms or 300 ms. Fifty percent of the action-reaction trials were semantically congruent, and the remaining were incongruent, e.g., one actor offers to shake hands, and the other reciprocally shakes their hand (congruent action-reaction) versus one actor offers to shake hands, and the other leans down (incongruent action-reaction). Participants made fast congruency judgments. We hypothesized that judging the congruency of action-reaction sequences is aided by temporal predictions. The findings supported this hypothesis; linear speed-accuracy scores showed that congruency judgments were facilitated by a temporally aligned occlusion duration, and reaction frames compared to 300 ms deviations, thus suggesting that observers internally simulate the temporal unfolding of an observed social interction. Furthermore, we explored the link between participants with higher autistic traits and their sensitivity to temporal deviations. Overall, the study offers new evidence of prediction mechanisms underpinning the perception of social interactions in isolation from action integration confounds.

Are adaptation aftereffects for facial emotional expressions affected by prior knowledge about the emotion?

Accurate perception of the emotional signals conveyed by others is crucial for successful social interaction. Such perception is influenced not only by sensory input, but also by knowledge we have about the others' emotions. This study addresses the issue of whether knowing that the other's emotional state is congruent or incongruent with their displayed emotional expression ("genuine" and "fake", respectively) affects the neural mechanisms underpinning the perception of their facial emotional expressions. We used a visual adaptation paradigm to investigate this question in three experiments employing increasing adaptation durations. The adapting stimuli consisted of photographs of emotional facial expressions of joy and anger, purported to reflect (in-)congruency between felt and expressed emotion, displayed by professional actors. A Validity checking procedure ensured participants had the correct knowledge about the (in-)congruency. Significantly smaller adaptation aftereffects were obtained when participants knew that the displayed expression was incongruent with the felt emotion, following all tested adaptation periods. This study shows that knowledge relating to the congruency between felt and expressed emotion modulates face expression aftereffects. We argue that this reflects that the neural substrate responsible for the perception of facial expressions of emotion incorporates the presumed felt emotion underpinning the expression.

Visual appearance modulates motor control in social interactions

The goal of new adaptive technologies is to allow humans to interact with technical devices, such as robots, in natural ways akin to human interaction. Essential for achieving this goal, is the understanding of the factors that support natural interaction. Here, we examined whether human motor control is linked to the visual appearance of the interaction partner. Motor control theories consider kinematic-related information but not visual appearance as important for the control of motor movements (Flash & Hogan, 1985; Harris & Wolpert, 1998; Viviani & Terzuolo, 1982). We investigated the sensitivity of motor control to visual appearance during the execution of a social interaction, i.e. a high-five. In a novel mixed reality setup participants executed a high-five with a three-dimensional life-size human- or a robot-looking avatar. Our results demonstrate that movement trajectories and adjustments to perturbations depended on the visual appearance of the avatar despite both avatars carrying out identical movements. Moreover, two well-known motor theories (minimum jerk, two-thirds power law) better predict robot than human interaction trajectories. The dependence of motor control on the human likeness of the interaction partner suggests that different motor control principles might be at work in object and human directed interactions.

Body Size Adaptation Alters Perception of Test Stimuli, Not Internal Body Image

Recent studies have reported that adaptation to extreme body types produces aftereffects on judgments of body normality and attractiveness, and also judgments of the size and shape of the viewer's own body. This latter effect suggests that adaptation could constitute an experimental model of media influences on body image. Alternatively, adaptation could affect perception of test stimuli, which should produce the same aftereffects for judgments about participant's own body or someone else's body. Here, we investigated whether adaptation similarly affects judgments about one's body and other bodies. We were interested in participants' own body image judgments, i.e., we wanted to measure the mental representations to which the test stimuli were compared and not the perception of test stimuli per se. Participants were adapted to pictures of thin or fat bodies and then rated whether bodies were fatter or thinner than either: their own body, an average body (Experiment 1), or the body of another person (Experiments 2 and 3). By keeping the visual stimuli constant but changing the task/type of judgment, i.e., the internal criterion participants are asked to judge the bodies against, we investigated how adaptation affects different stored representations of bodies, specifically own body image vs. representations of others. After adaptation, a classic aftereffect was found, with judgments biased away from the adapting stimulus. Critically, aftereffects were nearly identical for judgments of one's own body and for other people's bodies. These results suggest that adaptation affects body representations in a generic way and may not be specific to the own body image.

Visual perception of joint stiffness from multijoint motion

Humans have an astonishing ability to extract hidden information from the movements of others. For example, even with limited kinematic information, humans can distinguish between biological and nonbiological motion, identify the age and gender of a human demonstrator, and recognize what action a human demonstrator is performing. It is unknown, however, whether they can also estimate hidden mechanical properties of another's limbs simply by observing their motions. Strictly speaking, identifying an object's mechanical properties, such as stiffness, requires contact. With only motion information, unambiguous measurements of stiffness are fundamentally impossible, since the same limb motion can be generated with an infinite number of stiffness values. However, we show that humans can readily estimate the stiffness of a simulated limb from its motion. In three experiments, we found that participants linearly increased their rating of arm stiffness as joint stiffness parameters in the arm controller increased. This was remarkable since there was no physical contact with the simulated limb. Moreover, participants had no explicit knowledge of how the simulated arm was controlled. To successfully map nontrivial changes in multijoint motion to changes in arm stiffness, participants likely drew on prior knowledge of human neuromotor control. Having an internal representation consistent with the behavior of the controller used to drive the simulated arm implies that this control policy competently captures key features of veridical biological control. Finding that humans can extract latent features of neuromotor control from kinematics also provides new insight into how humans interpret the motor actions of others. NEW & NOTEWORTHY Humans can visually perceive another's overt motion, but it is unknown whether they can also perceive the hidden dynamic properties of another's limbs from their motions. Here, we show that humans can correctly infer changes in limb stiffness from nontrivial changes in multijoint limb motion without force information or explicit knowledge of the underlying limb controller. Our findings suggest that humans presume others control motor behavior in such a way that limb stiffness influences motion.

Adaptation aftereffects reveal representations for encoding of contingent social actions

A hallmark of human social behavior is the effortless ability to relate one's own actions to that of the interaction partner, e.g., when stretching out one's arms to catch a tripping child. What are the behavioral properties of the neural substrates that support this indispensable human skill? Here we examined the processes underlying the ability to relate actions to each other, namely the recognition of spatiotemporal contingencies between actions (e.g., a "giving" that is followed by a "taking"). We used a behavioral adaptation paradigm to examine the response properties of perceptual mechanisms at a behavioral level. In contrast to the common view that action-sensitive units are primarily selective for one action (i.e., primary action, e.g., 'throwing"), we demonstrate that these processes also exhibit sensitivity to a matching contingent action (e.g., "catching"). Control experiments demonstrate that the sensitivity of action recognition processes to contingent actions cannot be explained by lower-level visual features or amodal semantic adaptation. Moreover, we show that action recognition processes are sensitive only to contingent actions, but not to noncontingent actions, demonstrating their selective sensitivity to contingent actions. Our findings show the selective coding mechanism for action contingencies by action-sensitive processes and demonstrate how the representations of individual actions in social interactions can be linked in a unified representation.

Action Recognition in a Crowded Environment

So far, action recognition has been mainly examined with small point-light human stimuli presented alone within a narrow central area of the observer's visual field. Yet, we need to recognize the actions of life-size humans viewed alone or surrounded by bystanders, whether they are seen in central or peripheral vision. Here, we examined the mechanisms in central vision and far periphery (40° eccentricity) involved in the recognition of the actions of a life-size actor (target) and their sensitivity to the presence of a crowd surrounding the target. In Experiment 1, we used an action adaptation paradigm to probe whether static or idly moving crowds might interfere with the recognition of a target's action (hug or clap). We found that this type of crowds whose movements were dissimilar to the target action hardly affected action recognition in central and peripheral vision. In Experiment 2, we examined whether crowd actions that were more similar to the target actions affected action recognition. Indeed, the presence of that crowd diminished adaptation aftereffects in central vision as wells as in the periphery. We replicated Experiment 2 using a recognition task instead of an adaptation paradigm. With this task, we found evidence of decreased action recognition accuracy, but this was significant in peripheral vision only. Our results suggest that the presence of a crowd carrying out actions similar to that of the target affects its recognition. We outline how these results can be understood in terms of high-level crowding effects that operate on action-sensitive perceptual channels.

State-Dependent TMS Reveals Representation of Affective Body Movements in the Anterior Intraparietal Cortex

In humans, recognition of others' actions involves a cortical network that comprises, among other cortical regions, the posterior superior temporal sulcus (pSTS), where biological motion is coded and the anterior intraparietal sulcus (aIPS), where movement information is elaborated in terms of meaningful goal-directed actions. This action observation system (AOS) is thought to encode neutral voluntary actions, and possibly some aspects of affective motor repertoire, but the role of the AOS' areas in processing affective kinematic information has never been examined. Here we investigated whether the AOS plays a role in representing dynamic emotional bodily expressions. In the first experiment, we assessed behavioral adaptation effects of observed affective movements. Participants watched series of happy or fearful whole-body point-light displays (PLDs) as adapters and were then asked to perform an explicit categorization of the emotion expressed in test PLDs. Participants were slower when categorizing any of the two emotions as long as it was congruent with the emotion in the adapter sequence. We interpreted this effect as adaptation to the emotional content of PLDs. In the second experiment, we combined this paradigm with TMS applied over either the right aIPS, pSTS, and the right half of the occipital pole (corresponding to Brodmann's area 17 and serving as control) to examine the neural locus of the adaptation effect. TMS over the aIPS (but not over the other sites) reversed the behavioral cost of adaptation, specifically for fearful contents. This demonstrates that aIPS contains an explicit representation of affective body movements.SIGNIFICANCE STATEMENT In humans, a network of areas, the action observation system, encodes voluntary actions. However, the role of these brain regions in processing affective kinematic information has not been investigated. Here we demonstrate that the aIPS contains a representation of affective body movements. First, in a behavioral experiment, we found an adaptation after-effect for emotional PLDs, indicating the existence of a neural representation selective for affective information in biological motion. To examine the neural locus of this effect, we then combined the adaptation paradigm with TMS. Stimulation of the aIPS (but not over pSTS and control site) reversed the behavioral cost of adaptation, specifically for fearful contents, demonstrating that aIPS contains a representation of affective body movements.

Action recognition is sensitive to the identity of the actor

Recognizing who is carrying out an action is essential for successful human interaction. The cognitive mechanisms underlying this ability are little understood and have been subject of discussions in embodied approaches to action recognition. Here we examine one solution, that visual action recognition processes are at least partly sensitive to the actor's identity. We investigated the dependency between identity information and action related processes by testing the sensitivity of neural action recognition processes to clothing and facial identity information with a behavioral adaptation paradigm. Our results show that action adaptation effects are in fact modulated by both clothing information and the actor's facial identity. The finding demonstrates that neural processes underlying action recognition are sensitive to identity information (including facial identity) and thereby not exclusively tuned to actions. We suggest that such response properties are useful to help humans in knowing who carried out an action.

Sensorimotor body-environment interaction serves to regulate emotional experience and exploratory behavior

Almost all living species regularly explore environments that they experience as pleasant, aversive, arousing or frightening. We postulate that such exploratory behavior and emotional experience both are regulated based on the interdependent perception of one’s body and stimuli that collectively define a spatial context such as a cliff. Here we examined this by testing if the interaction of the sensory input on one’s gait and the sensory input on the spatial context is modulating both the emotional experience of the environment and its exploration through head motion. To this end, we asked healthy humans to explore a life-sized Virtual Reality simulation of a forest glade by physically walking around in this environment on two narrow rectangular platforms connected by a plank. The platforms and the plank were presented such that they were either placed on ground or on the top of two high bridge piers. Hence, the forest glade was presented either as a “ground” or as a “height” context. Within these two spatial contexts the virtual plank was projected either on the rigid physical floor or onto a bouncy physical plank. Accordingly, the gait of our participants while they crossed the virtual plank was either “smooth” or “bouncy.” We found that in the height context bouncy gait compared to smooth gait increased the orientation of the head below the horizon and intensified the experience of the environment as negative. Whereas, within the ground context bouncy gait increased the orientation of the head towards and above the horizon and made that the environment was experienced as positive. Our findings suggest that the brain of healthy humans is using the interaction of the sensory input on their gait and the sensory input on the spatial context to regulate both the emotional experience of the environment and its exploration through head motion.

Visual adaptation enhances action sound discrimination

Prolonged exposure, or adaptation, to a stimulus in 1 modality can bias, but also enhance, perception of a subsequent stimulus presented within the same modality. However, recent research has also found that adaptation in 1 modality can bias perception in another modality. Here, we show a novel crossmodal adaptation effect, where adaptation to a visual stimulus enhances subsequent auditory perception. We found that when compared to no adaptation, prior adaptation to visual, auditory, or audiovisual hand actions enhanced discrimination between 2 subsequently presented hand action sounds. Discrimination was most enhanced when the visual action “matched” the auditory action. In addition, prior adaptation to a visual, auditory, or audiovisual action caused subsequent ambiguous action sounds to be perceived as less like the adaptor. In contrast, these crossmodal action aftereffects were not generated by adaptation to the names of actions. Enhanced crossmodal discrimination and crossmodal perceptual aftereffects may result from separate mechanisms operating in audiovisual action sensitive neurons within perceptual systems. Adaptation-induced crossmodal enhancements cannot be explained by postperceptual responses or decisions. More generally, these results together indicate that adaptation is a ubiquitous mechanism for optimizing perceptual processing of multisensory stimuli.

Faster but Less Careful Prehension in Presence of High, Rather than Low, Social Status Attendees

Ample evidence attests that social intention, elicited through gestures explicitly signaling a request of communicative intention, affects the patterning of hand movement kinematics. The current study goes beyond the effect of social intention and addresses whether the same action of reaching to grasp an object for placing it in an end target position within or without a monitoring attendee's peripersonal space, can be moulded by pure social factors in general, and by social facilitation in particular. A motion tracking system (Optotrak Certus) was used to record motor acts. We carefully avoided the usage of communicative intention by keeping constant both the visual information and the positional uncertainty of the end target position, while we systematically varied the social status of the attendee (a high, or a low social status) in separated blocks. Only thirty acts performed in the presence of a different social status attendee, revealed a significant change of kinematic parameterization of hand movement, independently of the attendee's distance. The amplitude of peak velocity reached by the hand during the reach-to-grasp and the lift-to-place phase of the movement was larger in the high rather than in the low social status condition. By contrast, the deceleration time of the reach-to-grasp phase and the maximum grasp aperture was smaller in the high rather than in the low social status condition. These results indicated that the hand movement was faster but less carefully shaped in presence of a high, but not of a low social status attendee. This kinematic patterning suggests that being monitored by a high rather than a low social status attendee might lead participants to experience evaluation apprehension that informs the control of motor execution. Motor execution would rely more on feedforward motor control in the presence of a high social status human attendee, vs. feedback motor control, in the presence of a low social status attendee.

Visual adaptation dominates bimodal visual-motor action adaptation

A long standing debate revolves around the question whether visual action recognition primarily relies on visual or motor action information. Previous studies mainly examined the contribution of either visual or motor information to action recognition. Yet, the interaction of visual and motor action information is particularly important for understanding action recognition in social interactions, where humans often observe and execute actions at the same time. Here, we behaviourally examined the interaction of visual and motor action recognition processes when participants simultaneously observe and execute actions. We took advantage of behavioural action adaptation effects to investigate behavioural correlates of neural action recognition mechanisms. In line with previous results, we find that prolonged visual exposure (visual adaptation) and prolonged execution of the same action with closed eyes (non-visual motor adaptation) influence action recognition. However, when participants simultaneously adapted visually and motorically - akin to simultaneous execution and observation of actions in social interactions - adaptation effects were only modulated by visual but not motor adaptation. Action recognition, therefore, relies primarily on vision-based action recognition mechanisms in situations that require simultaneous action observation and execution, such as social interactions. The results suggest caution when associating social behaviour in social interactions with motor based information.

Action Recognition and Movement Direction Discrimination Tasks Are Associated with Different Adaptation Patterns

The ability to discriminate between different actions is essential for action recognition and social interactions. Surprisingly previous research has often probed action recognition mechanisms with tasks that did not require participants to discriminate between actions, e.g., left-right direction discrimination tasks. It is not known to what degree visual processes in direction discrimination tasks are also involved in the discrimination of actions, e.g., when telling apart a handshake from a high-five. Here, we examined whether action discrimination is influenced by movement direction and whether direction discrimination depends on the type of action. We used an action adaptation paradigm to target action and direction discrimination specific visual processes. In separate conditions participants visually adapted to forward and backward moving handshake and high-five actions. Participants subsequently categorized either the action or the movement direction of an ambiguous action. The results showed that direction discrimination adaptation effects were modulated by the type of action but action discrimination adaptation effects were unaffected by movement direction. These results suggest that action discrimination and direction categorization rely on partly different visual information. We propose that action discrimination tasks should be considered for the exploration of visual action recognition mechanisms.

Intact recognition, but attenuated adaptation, for biological motion in youth with autism spectrum disorder

Given the ecological importance of biological motion and its relevance to social cognition, considerable effort has been devoted over the past decade to studying biological motion perception in autism. However, previous studies have asked observers to detect or recognize briefly presented human actions placed in isolation, without spatial or temporal context. Research on typical populations has shown the influence of temporal context in biological motion perception: prolonged exposure to one action gives rise to an aftereffect that biases perception of a subsequently displayed action. Whether people with autism spectrum disorders (ASD) show such adaptation effects for biological motion stimuli remains unknown. To address this question, this study examined how well youth with ASD recognize ambiguous actions and adapt to recently-observed actions. Compared to typically-developing (TD) controls, youth with ASD showed no differences in perceptual boundaries between actions categories, indicating intact ability in recognizing actions. However, children with ASD showed weakened adaptation to biological motion. It is unlikely that the reduced action adaptability in autism was due to delayed developmental trajectory, as older children with ASD showed weaker adaptation to actions than younger children with ASD. Our results further suggest that high-level (i.e., action) processing weakens with age for children with ASD, but this change may be accompanied by a potentially compensatory mechanism based on more involvement of low-level (i.e., motion) processing. Autism Res 2016, 9: 1103-1113. © 2016 International Society for Autism Research, Wiley Periodicals, Inc.

One step ahead: The perceived kinematics of others' actions are biased toward expected goals

Action observation is often conceptualized in a bottom-up manner, where sensory information activates conceptual (or motor) representations. In contrast, here we show that expectations about an actor's goal have a top-down predictive effect on action perception, biasing it toward these goals. In 3 experiments, participants observed hands reach for or withdraw from objects and judged whether a probe stimulus corresponded to the hand's final position. Before action onset, participants generated action expectations on the basis of either object types (safe or painful, Experiments 1 and 2) or abstract color cues (Experiment 3). Participants more readily mistook probes displaced in a predicted position (relative to unpredicted positions) for the hand's final position, and this predictive bias was larger when the movement and expectation were aligned. These effects were evident for low-level movement and high-level goal expectancies. Expectations bias action observation toward the predicted goals. These results challenge current bottom-up views and support recent predictive models of action observation.

Motor-visual neurons and action recognition in social interactions

Cook et al. suggest that motor-visual neurons originate from associative learning. This suggestion has interesting implications for the processing of socially relevant visual information in social interactions. Here, we discuss two aspects of the associative learning account that seem to have particular relevance for visual recognition of social information in social interactions - namely, context-specific and contingency based learning.