Ventral encoding of functional affordances: A neural pathway for identifying errors in action

Mechanical and semantic knowledge mediate the implicit understanding of the physical world

Most recent accounts highlight the importance of two aspects of cognition in the implicit understanding of the physical world: semantic knowledge (the ability to recognize, categorize, and relate concepts) and mechanical knowledge (the capability to comprehend how things mechanically work). However, how the human brain may integrate these cognitive processes remains largely unexplored. Here, we use functional magnetic resonance imaging to investigate this integration employing a novel free-viewing task. Participants viewed images depicting object-tool pairs that were either mechanically consistent (e.g., nail - steel hammer) or mechanically inconsistent (e.g., scarf - steel hammer). These pairs were situated on a metal plate atop a table, with a stripped electrical cable in contact with the plate that could be plugged in or out from the electrical line, rendering the tools either electrified or not. Task-based functional connectivity revealed an interplay among specific left-brain regions - the middle temporal (MTG), inferior frontal (IFG), and supramarginal (SMG) gyri - during the processing of mechanical actions and physics principles, associating the activity of these areas with mechanical knowledge (SMG) and object-related semantic knowledge (MTG). Notably, the IFG was active during both types of processing, suggesting a critical role of this region in multi-modal information integration. These findings support the most recent integrated neurocognitive models of physical understanding, deepening our comprehension of how we make sense of the physical world.

On the functional brain networks involved in tool-related action understanding

Tool-use skills represent a significant cognitive leap in human evolution, playing a crucial role in the emergence of complex technologies. Yet, the neural mechanisms underlying such capabilities are still debated. Here we explore with fMRI the functional brain networks involved in tool-related action understanding. Participants viewed images depicting action-consistent (e.g., nail-hammer) and action-inconsistent (e.g., scarf-hammer) object-tool pairs, under three conditions: semantic (recognizing the tools previously seen in the pairs), mechanical (assessing the usability of the pairs), and control (looking at the pairs without explicit tasks). During the observation of the pairs, task-based left-brain functional connectivity differed within conditions. Compared to the control, both the semantic and mechanical conditions exhibited co-activations in dorsal (precuneus) and ventro-dorsal (inferior frontal gyrus) regions. However, the semantic condition recruited medial and posterior temporal areas, whereas the mechanical condition engaged inferior parietal and posterior temporal regions. Also, when distinguishing action-consistent from action-inconsistent pairs, an extensive frontotemporal neural circuit was activated. These findings support recent accounts that view tool-related action understanding as the combined product of semantic and mechanical knowledge. Furthermore, they emphasize how the left inferior parietal and anterior temporal lobes might be considered as hubs for the cross-modal integration of physical and conceptual knowledge, respectively.

Error observation as a window on performance monitoring in social contexts? A systematic review

Living in a social world requires social monitoring, i.e., the ability to keep track of others' actions and mistakes. Here, we demonstrate the good reliability of the behavioral and neurophysiological indexes ascribed to social monitoring. We also show that no consensus exists on the cognitive bases of this phenomenology and discuss three alternative hypotheses: (i) the direct matching hypothesis, postulating that observed errors are processed through automatic simulation; (ii) the attentional hypothesis, considering errors as unexpected events that take resources away from task processing; and (iii) the goal representation hypothesis, which weighs social error monitoring depending on how relevant the other's task is to the observer's goals. To date, evidence on the role played by factors that could help to disentangle these hypotheses (e.g., the human vs. non-human nature of the actor, the error rate, and the reward context) is insufficient, although the goal representation hypothesis seems to receive more support. Theory-driven experimental designs are needed to enlighten this debate and clarify the role of error monitoring during interactive exchanges.

Action goals and the praxis network: an fMRI study

The praxis representation network (PRN) of the left cerebral hemisphere is typically linked to the control of functional interactions with familiar tools. Surprisingly, little is known about the PRN engagement in planning and execution of tool-directed actions motivated by non-functional but purposeful action goals. Here we used functional neuroimaging to perform both univariate and multi-voxel pattern analyses (MVPA) in 20 right-handed participants who planned and later executed, with their dominant and non-dominant hands, disparate grasps of tools for different goals, including: (1) planning simple vs. demanding functional grasps of conveniently vs. inconveniently oriented tools with an intention to immediately use them, (2) planning simple—but non-functional—grasps of inconveniently oriented tools with a goal to pass them to a different person, (3) planning reaching movements directed at such tools with an intention to move/push them with the back of the hand, and (4) pantomimed execution of the earlier planned tasks. While PRN contributed to the studied interactions with tools, the engagement of its critical nodes, and/or complementary right hemisphere processing, was differently modulated by task type. E.g., planning non-functional/structural grasp-to-pass movements of inconveniently oriented tools, regardless of the hand, invoked the left parietal and prefrontal nodes significantly more than simple, non-demanding functional grasps. MVPA corroborated decoding capabilities of critical PRN areas and some of their right hemisphere counterparts. Our findings shed new lights on how performance of disparate action goals influences the extraction of object affordances, and how or to what extent it modulates the neural activity within the parieto-frontal brain networks.

Tool use and function knowledge shape visual object processing

Perceiving the environment automatically informs how we can interact with it through affordance mechanisms. However, it remains unknown how our knowledge about the environment shapes how it is perceived. In this training study, we evaluated whether motor and function knowledge about novel objects affects visual object processing. Forty-three participants associated a usage or function to a novel object in interactive virtual reality while their EEG was recorded. Both usage and function influenced the mu-band (8-12 Hz) rhythms, suggesting that motor and function object information influence motor processing during object recognition. Learning the usage also prevented the reduction of the theta-band (4-8 Hz) rhythms recorded over the posterior cortical areas, suggesting a predominant top-down influence of tool use information on visuo-motor pathways. The modulation being specifically induced by learning an object usage, the results support further the embodied cognition approach rather than the reasoning-based approach of object processing.

Hand-Selective Visual Regions Represent How to Grasp 3D Tools: Brain Decoding during Real Actions

Most neuroimaging experiments that investigate how tools and their actions are represented in the brain use visual paradigms where tools or hands are displayed as 2D images and no real movements are performed. These studies discovered selective visual responses in occipitotemporal and parietal cortices for viewing pictures of hands or tools, which are assumed to reflect action processing, but this has rarely been directly investigated. Here, we examined the responses of independently visually defined category-selective brain areas when participants grasped 3D tools (N = 20; 9 females). Using real-action fMRI and multivoxel pattern analysis, we found that grasp typicality representations (i.e., whether a tool is grasped appropriately for use) were decodable from hand-selective areas in occipitotemporal and parietal cortices, but not from tool-, object-, or body-selective areas, even if partially overlapping. Importantly, these effects were exclusive for actions with tools, but not for biomechanically matched actions with control nontools. In addition, grasp typicality decoding was significantly higher in hand than tool-selective parietal regions. Notably, grasp typicality representations were automatically evoked even when there was no requirement for tool use and participants were naive to object category (tool vs nontools). Finding a specificity for typical tool grasping in hand-selective, rather than tool-selective, regions challenges the long-standing assumption that activation for viewing tool images reflects sensorimotor processing linked to tool manipulation. Instead, our results show that typicality representations for tool grasping are automatically evoked in visual regions specialized for representing the human hand, the primary tool of the brain for interacting with the world.

Influence of cognitive-motor expertise on brain dynamics of anticipatory-based outcome processing

Motor experience plays an important role in the ability to anticipate action outcomes, but little is known about the brain processes through which it modulates the preparation for unexpected events. To address this issue, EEG was employed while table tennis players and novices observed videos of serves in order to predict the expected ball direction based on the kinematics of a model's movement. Furthermore, we manipulated the congruency between the model's body kinematics and the subsequent ball trajectory while assessing the cerebral cortical activity of novices and experts to understand how experts respond to unexpected outcomes. Experts were more accurate in predicting the ball trajectories than novices and were further differentiated from novices in the cortical dynamics just prior to ball contact and during the period of observation of the ball trajectories. Consistent with the predicted response-outcome model, experts exhibited elevated theta oscillations during the incongruent relative to the congruent trajectories, while no such differences were observed in the novices. Source estimation for theta activity revealed stronger activation in the middle frontal gyrus for the experts in response to the incongruent trajectories. Collectively, the observed differences in cortical dynamics between the groups suggest that motor experience promotes central neural system adaptations that facilitate preparation for anticipated outcomes and contributes to adaptive cognitive-motor responses in the face of uncertainty.

Are Sensory-Motor Relationships Encoded ad hoc or by Default?: An ERP Study

In this event-related potentials study we tested whether sensory-motor relations between concrete words are encoded by default or only under explicit ad hoc instructions. In Exp. 1, participants were explicitly asked to encode sensory-motor relations (e.g., “do the following objects fit in a pencil-cup?”), while other possible semantic relations remained implicit. In Exp. 2, using the same materials other group of participants were explicitly asked to encode semantic relations (e.g., “are the following objects related to a pencil-cup?”), and the possible sensory-motor relations remained implicit. The N400 component was sensitive to semantic relations (e.g., “desk” related to “pencil-cup”) both under implicit (Exp. 1) and explicit instructions (Exp. 2). By contrast, most sensory-motor relations (e.g., “pea” fitting in “pencil-cup”) were encoded ad hoc under explicit instructions (Exp. 1). Interestingly some sensory-motor relations were also encoded implicitly, but only when they corresponded to “functional” actions associated with high-related objects (e.g., “eraser” fitting in “pencil-cup”) and occurring at a late time window (500–650 ms; Exp. 2), suggesting that this type of sensory-motor relations were encoding by default.

Decoding Brain States for Planning Functional Grasps of Tools: A Functional Magnetic Resonance Imaging Multivoxel Pattern Analysis Study

OBJECTIVES

We used multivoxel pattern analysis (MVPA) to investigate neural selectivity for grasp planning within the left-lateralized temporo-parieto-frontal network of areas (praxis representation network, PRN) typically associated with tool-related actions, as studied with traditional neuroimaging contrasts.

METHODS

We used data from 20 participants whose task was to plan functional grasps of tools, with either right or left hands. Region of interest and whole-brain searchlight analyses were performed to show task-related neural patterns.

RESULTS

MVPA revealed significant contributions to functional grasp planning from the anterior intraparietal sulcus (aIPS) and its immediate vicinities, supplemented by inputs from posterior subdivisions of IPS, and the ventral lateral occipital complex (vLOC). Moreover, greater local selectivity was demonstrated in areas near the superior parieto-occipital cortex and dorsal premotor cortex, putatively forming the dorso-dorsal stream.

CONCLUSIONS

A contribution from aIPS, consistent with its role in prospective grasp formation and/or encoding of relevant tool properties (e.g., potential graspable parts), is likely to accompany the retrieval of manipulation and/or mechanical knowledge subserved by the supramarginal gyrus for achieving action goals. An involvement of vLOC indicates that MVPA is particularly sensitive to coding of object properties, their identities and even functions, for a support of grip formation. Finally, the engagement of the superior parieto-frontal regions as revealed by MVPA is consistent with their selectivity for transient features of tools (i.e., variable affordances) for anticipatory hand postures. These outcomes support the notion that, compared to traditional approaches, MVPA can reveal more fine-grained patterns of neural activity. (JINS, 2018, 24, 1013-1025).

The Role of Attention and Saccades on Parietofrontal Encoding of Contextual and Grasp-specific Affordances of Tools: An ERP Study

The ability to recognize a tool's affordances (how a spoon should be appropriately grasped and used), is vital for daily life. Prior research has identified parietofrontal circuits, including mirror neurons, to be critical in understanding affordances. However, parietofrontal action-encoding regions receive extensive visual input and are adjacent to parietofrontal attention control networks. It is unclear how eye movements and attention modulate parietofrontal encoding of affordances. To address this issue, scenes depicting tools in different use-contexts and grasp-postures were presented to healthy subjects across two experiments, with stimuli durations of 100 ms or 500 ms. The 100-ms experiment automatically restricted saccades and required covert attention, while the 500-ms experiment allowed overt attention. The two experiments elicited similar behavioral decisions on tool-use correctness and isolated the influence of attention on parietofrontal activity. Parietofrontal ERPs (P600) distinguishing tool-use contexts (e.g., spoon-yogurt vs. spoon-ball) were similar in both experiments. Conversely, parietofrontal ERPs distinguishing tool-grasps were characterized by posterior to frontal N130-N200 ERPs in the 100-ms experiment and by saccade-perturbed N130-N200 ERPs, frontal N400 and parietal P500 in the 500-ms experiment. Particularly, only overt gaze toward the hand-tool interaction engaged mirror neurons (frontal N400) when discerning grasps that manipulate but not functionally use a tool - (grasp bowl rather than stem of spoon). Results here detail the first human electrophysiological evidence on how attention selectively modulates multiple parietofrontal grasp-perception circuits, especially the mirror neuron system, while unaffecting parietofrontal encoding of tool-use contexts. These results are pertinent to neurophysiological models of affordances that typically neglect the role of attention in action perception.

Planning Functional Grasps of Simple Tools Invokes the Hand-independent Praxis Representation Network: An fMRI Study

OBJECTIVES

Neuropsychological and neuroimaging evidence indicates that tool use knowledge and abilities are represented in the praxis representation network (PRN) of the left cerebral hemisphere. We investigated whether PRN would also underlie the planning of function-appropriate grasps of tools, even though such an assumption is inconsistent with some neuropsychological evidence for independent representations of tool grasping and skilled tool use.

METHODS

Twenty right-handed participants were tested in an event-related functional magnetic resonance imaging (fMRI) study wherein they planned functionally appropriate grasps of tools versus grasps of non-tools matched for size and/or complexity, and later executed the pantomimed grasps of these objects. The dominant right, and non-dominant left hands were used in two different sessions counterbalanced across participants. The tool and non-tool stimuli were presented at three different orientations, some requiring uncomfortable hand rotations for effective grips, with the difficulty matched for both hands.

RESULTS

Planning functional grasps of tools (vs. non-tools) was associated with significant asymmetrical increases of activity in the temporo/occipital-parieto-frontal networks. The greater involvement of the left hemisphere PRN was particularly evident when hand movement kinematics (including wrist rotations) for grasping tools and non-tools were matched. The networks engaged in the task for the dominant and non-dominant hand were virtually identical. The differences in neural activity for the two object categories disappeared during grasp execution.

CONCLUSIONS

The greater hand-independent engagement of the left-hemisphere praxis representation network for planning functional grasps reveals a genuine effect of an early affordance/function-based visual processing of tools. (JINS, 2017, 23, 108-120).

Mechanisms underlying selecting objects for action

We assessed the factors which affect the selection of objects for action, focusing on the role of action knowledge and its modulation by distracters. Fourteen neuropsychological patients and 10 healthy aged-matched controls selected pairs of objects commonly used together among distracters in two contexts: with real objects and with pictures of the same objects presented sequentially on a computer screen. Across both tasks, semantically related distracters led to slower responses and more errors than unrelated distracters and the object actively used for action was selected prior to the object that would be passively held during the action. We identified a sub-group of patients (N = 6) whose accuracy was 2SDs below the controls performances in the real object task. Interestingly, these impaired patients were more affected by the presence of unrelated distracters during both tasks than intact patients and healthy controls. Note that the impaired patients had lesions to left parietal, right anterior temporal and bilateral pre-motor regions. We conclude that: (1) motor procedures guide object selection for action, (2) semantic knowledge affects action-based selection, (3) impaired action decision making is associated with the inability to ignore distracting information and (4) lesions to either the dorsal or ventral visual stream can lead to deficits in making action decisions. Overall, the data indicate that impairments in everyday tasks can be evaluated using a simulated computer task. The implications for rehabilitation are discussed.

I give you a cup, I get a cup: a kinematic study on social intention

While affordances have been intensively studied, the mechanisms according to how their activation is modulated by context are poorly understood. We investigated how the Agent׳s reach-to-grasp movement towards a target-object (e.g. a can) is influenced by the other׳s interaction with a second object (manipulative/functional) and by his/her eye-gaze communication. To manipulate physical context we showed participants two objects that could be linked by a spatial relation (e.g. can-knife, typically found in the same context), or by different functional relations. The functional relations could imply an action to perform with another person (functional-cooperative: e.g. can-glass), or on our own (functional-individual: e.g. can-straw). When objects were not related (e.g. can-toothbrush) participants had to refrain from responding. In order to respond, in the giving condition participants had to move the target object towards the other person, in the getting condition towards their own body. When participants (Agents) performed a reach-to-grasp movement to give the target object, in presence of eye-gaze communication they reached the wrist׳s acceleration peak faster if the Other previously interacted with the second object in accordance with its conventional use. Consistently participants reached faster the MFA when the objects were related by a functional-individual than a functional-cooperative relation. The Agent׳s getting response strongly affected the grasping component of the movement: in case of eye-gaze sharing, MFA was greater when the other previously performed a manipulative than a functional grip. Results reveal that humans have developed a sophisticated capability in detecting information from hand posture and eye-gaze, which are informative as to the Agent׳s intention.

Tool use as distributed cognition: how tools help, hinder and define manual skill

Our thesis in this paper is that, in order to appreciate the interplay between cognitive (goal-directed) and physical performance in tool use, it is necessary to determine the role that representations play in the use of tools. We argue that rather being solely a matter of internal (mental) representation, tool use makes use of the external representations that define the human–environment–tool–object system. This requires the notion of Distributed Cognition to encompass not simply the manner in which artifacts represent concepts but also how they represent praxis. Our argument is that this can be extended to include how artifacts-in-context afford use and how this response to affordances constitutes a particular form of skilled performance. By artifacts-in-context, we do not mean solely the affordances offered by the physical dimensions of a tool but also the interaction between the tool and the object that it is being used on. From this, “affordance” does not simply relate to the physical appearance of the tool but anticipates subsequent actions by the user directed towards the goal of changing the state of the object and this is best understood in terms of the “complimentarity” in the system. This assertion raises two challenges which are explored in this paper. The first is to distinguish “affordance” from the adaptation that one might expect to see in descriptions of motor control; when we speak of “affordance” as a form of anticipation, don’t we just mean the ability to adjust movements in response to physical demands? The second is to distinguish “affordance” from a schema of the tool; when we talk about anticipation, don’t we just mean the ability to call on a schema representing a “recipe” for using that tool for that task? This question of representation, specifically what knowledge needs to be represented in tool use, is central to this paper.

Differential mechanisms of action understanding in left and right handed subjects: the role of perspective and handedness

The ability to comprehend outcomes of skilled action is important for understanding the world around us. Prior studies have evaluated the perspective an action is performed in, but few have evaluated how handedness of the actor and the observer interact with action perspective. Understanding handedness affords the opportunity to identify the role of mirroring and matched limb action encoding, which may display unique strategies of action understanding. Right and left-handed subjects were presented with images of tools from egocentric or allocentric perspectives performing movements by either a left or right hand. Subjects had to judge the outcome of the task, and accuracy and latency were evaluated. Our hypothesis was that both left and right-handed subjects would predict action best from an egocentric perspective. In allocentric perspectives, identification of action outcomes would occur best in the mirror-matched dominant limb for all subjects. Results showed there was a significant effect on accuracy and latency with respect to perspective for both right and left-handed subjects. The highest accuracies and fastest latencies were found in the egocentric perspective. Handedness of subject also showed an effect on accuracy, where right-handed subjects were significantly more accurate in the task than left-handed subjects. An interaction effect revealed that left-handed subjects were less accurate at judging images from an allocentric viewpoint compared to all other conditions. These findings suggest that action outcomes are best facilitated in an internal perspective, regardless of the hand being used. The decreased accuracy for left-handed subjects on allocentric images could be due to asymmetrical lateralization of encoding action and motoric dominance, which may interfere with translating allocentric limb action outcomes. Further neurophysiological studies will help us understand the specific processes of how left and right-handed subjects may encode actions.