From observation to action simulation: the role of attention, eye-gaze, emotion, and body state

The acute effects of action observation training on upper extremity functions, cognitive processes and reaction times: a randomized controlled trial

Aim: To investigate the acute effects of action observation training on upper extremity functions, cognitive functions and response time in healthy, young adults. Materials & methods: A total of 60 participants were randomly divided into five groups: the self-action observation group, action observation group, action practice group, non-action observation group and control group. The Jebsen-Taylor hand function test (JTHFT), nine-hole peg test, serial reaction time task and d2 test of attention were applied to the participants before and after the interventions. Results: JTHFT performance with both non-dominant and dominant hands improved significantly compared with baseline in all groups (p < 0.001). JTHFT performance with non-dominant and dominant hands differed between the groups (p < 0.001). Conclusion: Action observation training seems to enhance the performance of upper extremity-related functions. Observing self-actions resulted in statistically significant positive changes in more variables compared with other methods. However, its clinical effectiveness over the other methods should be investigated in future long-term studies. Clinical Trial Registration: NCT04932057 (ClinicalTrials.gov).

The impact of action observation and motor imagery on the aesthetic preference of Chinese calligraphy

The creation of artwork requires motor activity. However, few empirical studies have directly explored the relationship between the experience of an artist's action and aesthetic experience. This study aimed to examine the effect of observing and imagery of the artist's action on the participants' aesthetic preferences. In Experiment 1 and 2, we took hard-pen and brush-pen Chinese calligraphy images as the stimulus, respectively, to explore the influence of action observation on the aesthetic preference by manipulating the artists' actions. The results of both Experiment 1 and 2 show that when participants observed the artists' actions, they tended to report a higher preference for calligraphy images compared with the control condition. In Experiments 3 and 4, we used instructions to manipulate the motor imagery tasks and investigated the effect of imaging the artist's action on the participants' aesthetic preferences. The results showed that both kinesthetic imagery and visual imagery increased the participants' preference. In general, our study shows that both action observation and motor imagery contribute to participants' aesthetic preferences. The results are discussed in terms of how artists' actions possibly influence the aesthetic preference of Chinese calligraphy.

The effects of physical training versus combined action observation and motor imagery in conjunction with physical training on upper-extremity performance

INTRODUCTION

Combined action observation and motor imagery training (AO+MI training), which involves motor imagery during action observation and physical training, has been attracting attention as an effective strategy for learning motor skills. However, little has been reported on the effects of AO+MI training. In the present study, we compared the effects of AO+MI training to the effects of physical training on upper-extremity performance.

MATERIALS AND METHODS

Ninety-six healthy participants were randomly assigned to either the control group or the experimental group. Sport stacking, which is often used to evaluate upper-extremity performance, was adopted for the task. The experiment was scheduled for three days. The training was 20 min per day. The control group performed only physical training, while the experimental group performed four 5-min AO+MI training sessions. Time taken to complete a sport stacking try (task completion time) was defined as the index of speed of upper-extremity performance and number of fallen cups as the index of its accuracy. The outcomes within each group and between the two groups were compared.

RESULTS

Both AO+MI training and physical training showed reduced task completion time and increased number of fallen cups. There were no significant differences in the degree of changes between the groups.

CONCLUSION

Results from the present study showed that AO+MI training and physical training had almost the same influence on upper-extremity performance in the early stages of learning sport stacking. This result suggests that AO+MI training may be an effective and low-burden training method for participants.

The extended present: an informational context for perception

Several previous authors have proposed a kind of specious or subjective present moment that covers a few seconds of recent information. This article proposes a new hypothesis about the subjective present, renamed the extended present, defined not in terms of time covered but as a thematically connected information structure held in working memory and in transiently accessible form in long-term memory. The three key features of the extended present are that information in it is thematically connected, both internally and to current attended perceptual input, it is organised in a hierarchical structure, and all information in it is marked with temporal information, specifically ordinal and duration information. Temporal boundaries to the information structure are determined by hierarchical structure processing and by limits on processing and storage capacity. Supporting evidence for the importance of hierarchical structure analysis is found in the domains of music perception, speech and language processing, perception and production of goal-directed action, and exact arithmetical calculation. Temporal information marking is also discussed and a possible mechanism for representing ordinal and duration information on the time scale of the extended present is proposed. It is hypothesised that the extended present functions primarily as an informational context for making sense of current perceptual input, and as an enabler for perception and generation of complex structures and operations in language, action, music, exact calculation, and other domains.

Visual-Electrotactile Stimulation Feedback to Improve Immersive Brain-Computer Interface Based on Hand Motor Imagery

In the aging society, the number of people suffering from vascular disorders is rapidly increasing and has become a social problem. The death rate due to stroke, which is the second leading cause of global mortality, has increased by 40% in the last two decades. Stroke can also cause paralysis. Of late, brain-computer interfaces (BCIs) have been garnering attention in the rehabilitation field as assistive technology. A BCI for the motor rehabilitation of patients with paralysis promotes neural plasticity, when subjects perform motor imagery (MI). Feedback, such as visual and proprioceptive, influences brain rhythm modulation to contribute to MI learning and motor function restoration. Also, virtual reality (VR) can provide powerful graphical options to enhance feedback visualization. This work aimed to improve immersive VR-BCI based on hand MI, using visual-electrotactile stimulation feedback instead of visual feedback. The MI tasks include grasping, flexion/extension, and their random combination. Moreover, the subjects answered a system perception questionnaire after the experiments. The proposed system was evaluated with twenty able-bodied subjects. Visual-electrotactile feedback improved the mean classification accuracy for the grasping (93.00%  $\pm$  3.50%) and flexion/extension (95.00%  $\pm$  5.27%) MI tasks. Additionally, the subjects achieved an acceptable mean classification accuracy (maximum of 86.5%  $\pm$  5.80%) for the random MI task, which required more concentration. The proprioceptive feedback maintained lower mean power spectral density in all channels and higher attention levels than those of visual feedback during the test trials for the grasping and flexion/extension MI tasks. Also, this feedback generated greater relative power in the $\mu$ -band for the premotor cortex, which indicated better MI preparation. Thus, electrotactile stimulation along with visual feedback enhanced the immersive VR-BCI classification accuracy by 5.5% and 4.5% for the grasping and flexion/extension MI tasks, respectively, retained the subject’s attention, and eased MI better than visual feedback alone.

Gaze and body cues interplay during interactive requests

Although observing other's gaze and body movements provides a crucial source of information to successfully interact with other people, it remains unclear whether observers weigh differently these cues and whether the convergence of gaze and body's directions determines facilitation effects. Here we aim to shed more light on this issue by testing the reliance upon these cues from both a behavioral and a neurophysiological perspective in a social interactive context. In Experiment 1, we manipulated the convergence between the direction of an actor's upper limb movement and gaze direction while he attempts to socially interact with the participants observing the scene. We determined the direction of gaze as well as the duration of participants' ocular fixations during the observation of the scene. In Experiment 2, we measured and correlated the effect of the body/gaze manipulation on corticospinal excitability and on the readiness to interact-a disposition to engage in social situations. Eye-tracking data revealed that participants fixated chiefly the actor's head when his hand and gaze directions were divergent. Possibly a strategy to disambiguate the scene. Whereas participants mainly fixated the actor's hand when he performed an interactive request toward the participants. From a neurophysiological point of view, the more participants felt involved in the interaction, the lower was motor preparation in the muscle potentially needed to fulfill the actor's request. We contend that social contexts are more likely to elicit motor preparation compared to non-social ones, and that muscular inhibition is a necessary mechanism in order to prevent unwanted overt reactions during action observation tasks.

A Central Component of the N1 Event-Related Brain Potential Could Index the Early and Automatic Inhibition of the Actions Systematically Activated by Objects

Stimuli of the environment, like objects, systematically activate the actions they are associated to. These activations occur extremely fast. Nevertheless, behavioral data reveal that, in most cases, these activations are then automatically inhibited, around 100 ms after the occurrence of the stimulus. We thus tested whether this early inhibition could be indexed by a central component of the N1 event-related brain potential (ERP). To achieve that goal, we looked at whether this ERP component is larger in tasks that could increase the inhibition and in trials where reaction times (RTs) happen to be long. The illumination of a real space bar of a keyboard out of the dark was used as a stimulus. To maximize the modulation of the inhibition, the task participants had to perform was manipulated across blocks. A look-only task and a count task were used to increase inhibition and an immediate press task was used to decrease it. ERPs of the two block-conditions where presses had to be prevented and where the largest central N1s were predicted were compared to those elicited in the press task, differentiating the ERPs to the third of the trials where presses were the slowest from the ERPs to the third of the trials with the fastest presses. Despite larger negativities due to lateralized readiness potentials (LRPs) and despite greater attention likely in immediate press-trials, central N1s were found to be minimal for the fastest presses, intermediate for the slowest ones and maximal for the two no-press conditions. These results thus provide a strong support for the idea that the central N1 indexes an early and short lasting automatic inhibition of the actions systematically activated by objects. They also confirm that the strength of this automatic inhibition spontaneously fluctuates across trials and tasks. On the other hand, just before N1s, parietal P1s were found larger for fastest presses. They might thus index the initial activation of these actions. Finally, consistent with the idea that N300s index late inhibition processes, that occur preferentially when the task requires them, these ERPs were quasi absent for fast presses trials and much larger in the three other conditions.

Enhancement of motor-imagery ability via combined action observation and motor-imagery training with proprioceptive neurofeedback

Varied individual ability to control the sensory-motor rhythms may limit the potential use of motor-imagery (MI) in neurorehabilitation and neuroprosthetics. We employed neurofeedback training of MI under action observation (AO: AOMI) with proprioceptive feedback and examined whether it could enhance MI-induced event-related desynchronization (ERD). Twenty-eight healthy young adults participated in the neurofeedback training. They performed MI while watching a video of hand-squeezing motion from a first-person perspective. Eleven participants received correct proprioceptive feedback of the same hand motion with the video, via an exoskeleton robot attached to their hand, upon their successful generation of ERD. Another nine participants received random feedback. The training lasted for approximately 20 min per day and continued for 6 days within an interval of 2 weeks. MI-ERD power was evaluated separately, without AO, on each experimental day. The MI-ERD power of the participants receiving correct feedback, as opposed to random feedback, was significantly increased after training. An additional experiment in which the remaining eight participants were trained with auditory instead of proprioceptive feedback failed to show statistically significant increase in MI-ERD power. The significant training effect obtained in shorter training time relative to previously proposed methods suggests the superiority of AOMI training and physiologically-congruent proprioceptive feedback to enhance the MI-ERD power. The proposed neurofeedback training could help patients with motor deficits to attain better use of brain-machine interfaces for rehabilitation and/or prosthesis.

Influence of time-of-day on joint Navon effect

The shared attention theory suggests that people devote greater cognitive resources to those features co-attended simultaneously with others, determining better performance in several types of tasks. When co-actors performed a go/no-go Navon task attending different features of target letters, the performance was impaired, reflecting a joint Navon effect (the representation of a co-actor's attentional focus made it more difficult to select and apply one's own focus of attention), probably due to asynchronous co-attention with a decrease in cognitive resources involved. Researches in chronobiology and chronopsychology demonstrated that not only selective attention (involved in a Navon task), but also cognitive resources have a daily fluctuations, mainly paralleling the circadian rhythm of body temperature (i.e. increasing values from the morning to evening with a subsequent decline in the night). The study was conducted to assess whether the presence of joint attention, as measured by the joint Navon effect, was influenced by the time-of-day. Sixteen pairs of participants sitting next to each other were required to respond to the identity letters in a go/no-go Navon task twice: in the morning (09:00-10:00) and early afternoon (13:00-14:00). The results showed a joint Navon effect in the morning session only, suggesting that joint attention was affected by the time-of-day effect on cognitive resources.

Gaze and Arrows: The Effect of Element Orientation on Apparent Motion is Modulated by Attention

In two experiments we investigated whether stimuli that elicit automatic orienting of attention (i.e., arrow or averted gaze) could drive apparent motion perception in one of two possible directions, modulating the effect of a low-level property (the orientation of elements along the motion direction). To this end, the competing motion paradigm was used, in which at time 1, a stimulus appears in the center of the display, and at time 2, two other stimuli appear in different spatial locations. Three kinds of stimuli with eight possible orientations were used in separate blocks; (1) a line; (2) an arrow; and, (3) an averted gaze. First, since the three stimuli present in the display at time 2 should be perceived to be located at the same distance (i.e., equidistant), the threshold for perceived equidistance was calculated for each participant and then used as the customized inter-stimulus distance. Participants were asked to press the button corresponding to the direction of the perceived motion. Results show a preference for collinear motion (motions between elements oriented along the motion direction), with a higher percentage of responses for gaze and arrow stimuli. In Experiment 1, a difference between gaze- and arrow-stimuli was observed. Apparent motion was seen towards the collinear position more often for gaze than for arrow when the stimulus was pointing to the vertical directions, while the opposite was true when the stimulus was pointing to the horizontal directions. In Experiment 2, where the lightness contrast between the gaze and the background was reduced, no difference between gaze- and arrow-stimuli emerged. We interpret our results as due to the social and biological value of gaze, which solved a possible ambiguity between gaze direction and the directions conveyed by the figural properties of the contrasted background in Experiment 1. These findings are consistent with the idea that stimuli known to automatically orient visual attention modulate motion perception.

Visual exploration patterns of human figures in action: an eye tracker study with art paintings

Art exploration is a complex process conditioned by factors at different levels and includes both basic visual principles and complex cognitive factors. The human figure is considered a critical factor attracting the attention in art painting. Using an eye-tracking methodology, the goal of this study was to explore different elements of the human figure performing an action (face and body parts in action) in complex social scenes characterized by different levels of social interaction between agents depicted in scenes (individual vs. social). The sample included 44 laypersons, and the stimuli consisted of 10 fine art paintings representing the figurative style of classical art. The results revealed different scanning patterns of the human figure elements related to the level of social interaction of agents depicted in the scene. The agents’ face attracted eye movements in social interaction scenes while the agents’ body parts attracted eye movements only when the agents were involved in individual actions. These processes were confirmed specifically in participants with high empathic abilities who became immediately fixated on faces to develop a mimetic engagement with other agents. Future studies integrating other measures would help confirm the results obtained and strengthen their implication for embodiment processes.

Dance experience sculpts aesthetic perception and related brain circuits

Previous research on aesthetic preferences demonstrates that people are more likely to judge a stimulus as pleasing if it is familiar. Although general familiarity and liking are related, it is less clear how motor familiarity, or embodiment, relates to a viewer's aesthetic appraisal. This study directly compared how learning to embody an action impacts the neural response when watching and aesthetically evaluating the same action. Twenty-two participants trained for 4 days on dance sequences. Each day they physically rehearsed one set of sequences, passively watched a second set, listened to the music of a third set, and a fourth set remained untrained. Functional MRI was obtained prior to and immediately following the training period, as were affective and physical ability ratings for each dance sequence. This approach enabled precise comparison of self-report methods of embodiment with nonbiased, empirical measures of action performance. Results suggest that after experience, participants most enjoy watching those dance sequences they danced or observed. Moreover, brain regions involved in mediating the aesthetic response shift from subcortical regions associated with dopaminergic reward processing to posterior temporal regions involved in processing multisensory integration, emotion, and biological motion.

Impairment of gaze-directed spatial coding in recent-onset schizophrenia

Patients with schizophrenia show deficits in core cognitive functions as well as in social cognition. The aim of the present study was to test whether deficits in social cognition influence nonsocial, “cold”, cognition. Thirty-five patients with recent-onset schizophrenia (SC) and 30 healthy controls (HC) performed a Simon task with social and simple geometric stimuli. We investigated whether the Simon effect, the slowing of reaction times produced by stimulus incongruities in the task-irrelevant spatial domain, differs between patients and healthy participants as a function of the social nature of the cues. The Simon effect was generated by a schematic drawing of human eyes (social cues) or rectangles (nonsocial cues). Overall, patients had longer reaction times than HC. In the eye-like condition, the Simon effect was significantly stronger for HC than for SC. In HC the Simon effect was significantly stronger in the eye-like than in the rectangle condition. In patients, the Simon effect did not differ significantly between both conditions. Thus, the influence of social cues was greatly reduced in the patient group. Current psychopathology or antipsychotic treatment did not influence results. The present study supports earlier findings of altered processing of schematic social cues in patients with schizophrenia, especially when gaze is task-irrelevant.

Attention Allocation and Task Representation during Joint Action Planning

We investigated whether people take into account an interaction partner's attentional focus and whether they represent in advance their partner's part of the task when planning to engage in a synchronous joint action. The experiment involved two participants planning and performing joint actions (i.e., synchronously lifting and clinking glasses), unimanual individual actions (i.e., lifting and moving a glass as if clinking with another person), and bimanual individual actions. EEG was recorded from one of the participants. We employed a choice reaction paradigm where a visual cue indicated the type of action to be planned, followed 1.5 sec later by a visual go stimulus, prompting the participants to act. We studied attention allocation processes by examining two lateralized EEG components, namely the anterior directing attention negativity and the late directing attention positivity. Action planning processes were examined using the late contingent negative variation and the movement-related potential. The results show that early stages of joint action planning involve dividing attention between locations in space relevant for one's own part of the joint action and locations relevant for one's partner's part of the joint action. At later stages of joint action planning, participants represented in advance their partner's upcoming action in addition to their own action, although not at an effector-specific level. Our study provides electrophysiological evidence supporting the operation of attention sharing processes and predictive self/other action representation during the planning phase of a synchronous joint task.

Visuo-olfactory integration during action observation and execution of reach-to-grasp movements

Observing the actions of others prompts the motor system to perform a similar action. However, visual cues are not the only source of sensory information for the motor system, which is affected by stimuli presented in all modalities even when they are irrelevant for action completion. The current experiment explored whether (and how) olfactory stimuli can influence the performance of a reach-to-grasp movement to visual objects differing in size (small and large) in the context of an automatic imitation task. Odours could match-or not- the size of a to-be-grasped visual target, or be nonexistent. Movement duration, an integral index of motor control, was significantly shorter when participants previously observed the same action. Addition of the odour component suggested that when the odour matched the size of a small target, a facilitation effect was found. Results are discussed in terms of olfactory-visual integration mechanisms and how they relate to embodied cognition.

Priorities for selection and representation in natural tasks

Selecting and remembering visual information is an active and competitive process. In natural environments, representations are tightly coupled to task. Objects that are task-relevant are remembered better due to a combination of increased selection for fixation and strategic control of encoding and/or retaining viewed information. However, it is not understood how physically manipulating objects when performing a natural task influences priorities for selection and memory. In this study, we compare priorities for selection and memory when actively engaged in a natural task with first-person observation of the same object manipulations. Results suggest that active manipulation of a task-relevant object results in a specific prioritization for object position information compared with other properties and compared with action observation of the same manipulations. Experiment 2 confirms that this spatial prioritization is likely to arise from manipulation rather than differences in spatial representation in real environments and the movies used for action observation. Thus, our findings imply that physical manipulation of task relevant objects results in a specific prioritization of spatial information about task-relevant objects, possibly coupled with strategic de-prioritization of colour memory for irrelevant objects.

Action co-representation and social exclusion

Humans are thought to be able to form shared representations, considered a keystone of social cognition. However, whether and to what extent such representations are social in nature is still open for debate. In the present study, we address the question of whether action co-representation can be modulated by social attitudes, such as judgments about one's own social status. Two groups of participants performed an Interactive Simon task after the experimental induction of a feeling of social inclusion or exclusion (ostracism) by means of a virtual ball tossing game. Results showed a compatibility effect in included, but not in excluded participants. This indicates that judgments about one's own social status modulate action co-representation. We suggest that this modulation may occur by way of a redirection of one's attentional focus away from others when one experiences social exclusion. This is a far-reaching issue given the increasing need for integration in modern society. Indeed, if integration fails, modern society fails also.

Observation of another's action but not eye gaze triggers allocentric visual perspective

In the present paper, we investigated whether observation of bodily cues—that is, hand action and eye gaze—can modulate the onlooker's visual perspective taking. Participants were presented with scenes of an actor gazing at an object (or straight ahead) and grasping an object (or not) in a 2 × 2 factorial design and a control condition with no actor in the scene. In Experiment 1, two groups of subjects were explicitly required to judge the left/right location of the target from their own (egocentric group) or the actor's (allocentric group) point of view, whereas in Experiment 2 participants did not receive any instruction on the point of view to assume. In both experiments, allocentric coding (i.e., the actor's point of view) was triggered when the actor grasped the target, but not when he gazed towards it, or when he adopted a neutral posture. In Experiment 3, we demonstrate that the actor's gaze but not action affected participants' attention orienting. The different effects of others' grasping and eye gaze on observers' behaviour demonstrated that specific bodily cues convey distinctive information about other people's intentions.

Eye gaze cannot be ignored (but neither can arrows)

Recent studies have tried to shed light on the automaticity of attentional shifts triggered by gaze and arrows with mixed results. In the present research, we aimed at testing a strong definition of resistance to suppression for orienting of attention elicited by these two cues. In five experiments, participants were informed with 100% certainty about the future location of a target they had to react to by presentation of either a direction word at the beginning of each trial or instructions at the beginning of each block. Gaze and arrows were presented before the target as uninformative distractors irrelevant for the task. The results showed similar patterns for gaze and arrows—namely, an interference effect when the distractors were incongruent with the upcoming target location. This suggests that the orienting of attention mediated by gaze and arrows can be considered as strongly automatic.

Predicting others' actions via grasp and gaze: evidence for distinct brain networks

During social interactions, how do we predict what other people are going to do next? One view is that we use our own motor experience to simulate and predict other people's actions. For example, when we see Sally look at a coffee cup or grasp a hammer, our own motor system provides a signal that anticipates her next action. Previous research has typically examined such gaze and grasp-based simulation processes separately, and it is not known whether similar cognitive and brain systems underpin the perception of object-directed gaze and grasp. Here we use functional magnetic resonance imaging to examine to what extent gaze- and grasp-perception rely on common or distinct brain networks. Using a 'peeping window' protocol, we controlled what an observed actor could see and grasp. The actor could peep through one window to see if an object was present and reach through a different window to grasp the object. However, the actor could not peep and grasp at the same time. We compared gaze and grasp conditions where an object was present with matched conditions where the object was absent. When participants observed another person gaze at an object, left anterior inferior parietal lobule (aIPL) and parietal operculum showed a greater response than when the object was absent. In contrast, when participants observed the actor grasp an object, premotor, posterior parietal, fusiform and middle occipital brain regions showed a greater response than when the object was absent. These results point towards a division in the neural substrates for different types of motor simulation. We suggest that left aIPL and parietal operculum are involved in a predictive process that signals a future hand interaction with an object based on another person's eye gaze, whereas a broader set of brain areas, including parts of the action observation network, are engaged during observation of an ongoing object-directed hand action.