The visual encoding of tool–object affordances

On the functional brain networks involved in tool-related action understanding

Tool-use skills represent a significant cognitive leap in human evolution, playing a crucial role in the emergence of complex technologies. Yet, the neural mechanisms underlying such capabilities are still debated. Here we explore with fMRI the functional brain networks involved in tool-related action understanding. Participants viewed images depicting action-consistent (e.g., nail-hammer) and action-inconsistent (e.g., scarf-hammer) object-tool pairs, under three conditions: semantic (recognizing the tools previously seen in the pairs), mechanical (assessing the usability of the pairs), and control (looking at the pairs without explicit tasks). During the observation of the pairs, task-based left-brain functional connectivity differed within conditions. Compared to the control, both the semantic and mechanical conditions exhibited co-activations in dorsal (precuneus) and ventro-dorsal (inferior frontal gyrus) regions. However, the semantic condition recruited medial and posterior temporal areas, whereas the mechanical condition engaged inferior parietal and posterior temporal regions. Also, when distinguishing action-consistent from action-inconsistent pairs, an extensive frontotemporal neural circuit was activated. These findings support recent accounts that view tool-related action understanding as the combined product of semantic and mechanical knowledge. Furthermore, they emphasize how the left inferior parietal and anterior temporal lobes might be considered as hubs for the cross-modal integration of physical and conceptual knowledge, respectively.

Investigating the cognitive processing of tools: Effects of dangerousness and directionality on attentional biases

Research has found a tool-based head-superiority effect (responses are faster to target stimuli that appear at the head than at the handle position). However, these studies did not consider the effects of directionality and dangerousness. This study investigated how directionality and dangerousness influence the head-superiority effect. Subjects were required to respond to the target location in all experiments. Experiment 1 manipulated the directionality, dangerousness and target location. Experiment 2 matched the sharpness of the tool tip in the directed conditions. Experiment 3 shortened the presentation time of the cue stimuli from 800 ms to 200 ms. Experiment 4 selected four tools with different functions to rule out an alternative explanation caused by functional repetition. The results indicate that a head-superiority effect emerges in the directed condition, and that dangerousness modulates the magnitude of this effect during the 800 ms condition. However, the modulation effect of dangerousness diminishes during the 200 ms condition.

The visual encoding of graspable unfamiliar objects

We explored by eye-tracking the visual encoding modalities of participants (N = 20) involved in a free-observation task in which three repetitions of ten unfamiliar graspable objects were administered. Then, we analysed the temporal allocation (t = 1500 ms) of visual-spatial attention to objects' manipulation (i.e., the part aimed at grasping the object) and functional (i.e., the part aimed at recognizing the function and identity of the object) areas. Within the first 750 ms, participants tended to shift their gaze on the functional areas while decreasing their attention on the manipulation areas. Then, participants reversed this trend, decreasing their visual-spatial attention to the functional areas while fixing the manipulation areas relatively more. Crucially, the global amount of visual-spatial attention for objects' functional areas significantly decreased as an effect of stimuli repetition while remaining stable for the manipulation areas, thus indicating stimulus familiarity effects. These findings support the action reappraisal theoretical approach, which considers object/tool processing as abilities emerging from semantic, technical/mechanical, and sensorimotor knowledge integration.

Grounding human-object interaction to affordance behavior in multimodal datasets

While affordance detection and Human-Object interaction (HOI) detection tasks are related, the theoretical foundation of affordances makes it clear that the two are distinct. In particular, researchers in affordances make distinctions between J. J. Gibson's traditional definition of an affordance, "the action possibilities" of the object within the environment, and the definition of a telic affordance, or one defined by conventionalized purpose or use. We augment the HICO-DET dataset with annotations for Gibsonian and telic affordances and a subset of the dataset with annotations for the orientation of the humans and objects involved. We then train an adapted Human-Object Interaction (HOI) model and evaluate a pre-trained viewpoint estimation system on this augmented dataset. Our model, AffordanceUPT, is based on a two-stage adaptation of the Unary-Pairwise Transformer (UPT), which we modularize to make affordance detection independent of object detection. Our approach exhibits generalization to new objects and actions, can effectively make the Gibsonian/telic distinction, and shows that this distinction is correlated with features in the data that are not captured by the HOI annotations of the HICO-DET dataset.

Visual Attention and Cognitive Archaeology: An Eye-Tracking Study of Palaeolithic Stone Tools

The study of lithic technology can provide information on human cultural evolution. This article aims to analyse visual behaviour associated with the exploration of ancient stone artefacts and how this relates to perceptual mechanisms in humans. In Experiment 1, we used eye tracking to record patterns of eye fixations while participants viewed images of stone tools, including examples of worked pebbles and handaxes. The results showed that the focus of gaze was directed more towards the upper regions of worked pebbles and on the basal areas for handaxes. Knapped surfaces also attracted more fixation than natural cortex for both tool types. Fixation distribution was different to that predicted by models that calculate visual salience. Experiment 2 was an online study using a mouse-click attention tracking technique and included images of unworked pebbles and 'mixed' images combining the handaxe's outline with the pebble's unworked texture. The pattern of clicks corresponded to that revealed using eye tracking and there were differences between tools and other images. Overall, the findings suggest that visual exploration is directed towards functional aspects of tools. Studies of visual attention and exploration can supply useful information to inform understanding of human cognitive evolution and tool use.

From Affordances to Abstract Words: The Flexibility of Sensorimotor Grounding

The sensorimotor system plays a critical role in several cognitive processes. Here, we review recent studies documenting this interplay at different levels. First, we concentrate on studies that have shown how the sensorimotor system is flexibly involved in interactions with objects. We report evidence demonstrating how social context and situations influence affordance activation, and then focus on tactile and kinesthetic components in body-object interactions. Then, we turn to word use, and review studies that have shown that not only concrete words, but also abstract words are grounded in the sensorimotor system. We report evidence that abstract concepts activate the mouth effector more than concrete concepts, and discuss this effect in light of studies on adults, children, and infants. Finally, we pinpoint possible sensorimotor mechanisms at play in the acquisition and use of abstract concepts. Overall, we show that the involvement of the sensorimotor system is flexibly modulated by context, and that its role can be integrated and flanked by that of other systems such as the linguistic system. We suggest that to unravel the role of the sensorimotor system in cognition, future research should fully explore the complexity of this intricate, and sometimes slippery, relation.

Flexible constraint hierarchy during the visual encoding of tool-object interactions

Tools and objects are associated with numerous action possibilities that are reduced depending on the task-related internal and external constraints presented to the observer. Action hierarchies propose that goals represent higher levels of the hierarchy while kinematic patterns represent lower levels of the hierarchy. Prior work suggests that tool-object perception is heavily influenced by grasp and action context. The current study sought to evaluate whether the presence of action hierarchy can be perceptually identified using eye tracking during tool-object observation. We hypothesize that gaze patterns will reveal a perceptual hierarchy based on the observed task context and grasp constraints. Participants viewed tool-objects scenes with two types of constraints: task-context and grasp constraints. Task-context constraints consisted of correct (e.g., frying pan-spatula) and incorrect tool-object pairings (e.g., stapler-spatula). Grasp constraints involved modified tool orientations, which requires participants to understand how initially awkward grasp postures can help achieve the task. The visual scene contained three areas of interests (AOIs): the object, the functional tool-end (e.g., spoon handle) and the manipulative tool-end (e.g., spoon bowl). Results revealed two distinct processes based on stimuli constraints. Goal-oriented encoding, the attentional bias towards the object and manipulative tool-end, was demonstrated when grasp did not lead to meaningful tool-use. In images where grasp postures were critical to action performance, attentional bias was primarily between the object and functional tool-end, which suggests means-related encoding of the graspable properties of the object. This study expands from previous work and demonstrates a flexible constraint hierarchy depending on the observed task constraints.

Tool heads prime saccades

Tools are wielded by their handles, but a lot of information about their function comes from their heads (the action-ends). Here we investigated whether eye saccadic movements are primed by tool handles, or whether they are primed by tool heads. We measured human saccadic reaction times while subjects were performing an attentional task. We found that saccades were executed quicker when performed to the side congruent with the tool head, even though "toolness" was irrelevant for the task. Our results show that heads are automatically processed by the visual system to orient eye movements, indicating that eyes are attracted by functional parts of manipulable objects and by the characteristic information these parts convey.

Semantic congruency effects of prime words on tool visual exploration

Most recent research on human tool use highlighted how people might integrate multiple sources of information through different neurocognitive systems to exploit the environment for action. This mechanism of integration is known as "action reappraisal". In the present eye-tracking study, we further tested the action reappraisal idea by devising a word-priming paradigm to investigate how semantically congruent (e.g., "nail") vs. semantically incongruent words (e.g., "jacket") that preceded the vision of tools (e.g., a hammer) may affect participants' visual exploration of them. We found an implicit modulation of participants' temporal allocation of visuospatial attention as a function of the object-word consistency. Indeed, participants tended to increase over time their fixations on tools' manipulation areas under semantically congruent conditions. Conversely, participants tended to concentrate their visual-spatial attention on tools' functional areas when inconsistent object-word pairs were presented. These results support and extend the information-integrated perspective of the action reappraisal approach. Also, these findings provide further evidence about how higher-level semantic information may influence tools' visual exploration.

Consumer Decision-Making Creativity and Its Relation to Exploitation-Exploration Activities: Eye-Tracking Approach

Modern consumers face a dramatic rise in web-based technological advancements and have trouble making rational and proper decisions when they shop online. When they try to make decisions about products and services, they also feel pressured against time when sorting among all of the unnecessary items in the flood of information available on the web. In this sense, they need to use consumer decision-making creativity (CDMC) to make rational decisions. However, unexplored research questions on this subject remain. First, in what ways do task difficulty and time constraints affect visual attention on exploitative and exploratory activities differently? Second, how does the location of the reference (i.e., hints) influence the level of visual attention to exploitative and exploratory activities depending on affordance theory? Third, how do exploratory and exploitative activities affect CDMC? Eye-tracking experiments were conducted with 70 participants to obtain relevant metrics such as total fixation duration (TFD), fixation count (FC), and visit count (VC) to answer these research questions. Our findings suggest that task difficulty influences exploitative activity, whereas time constraint is related to the exploratory activity. The result of the location of hints aligns with the affordance theory for the exploitative activity. Besides, exploratory activity positively affected CDMC, but exploitative activity did not show any effect.

Hazardous tools: the emergence of reasoning in human tool use

Humans are unique in the way they understand the causal relationships between the use of tools and achieving a goal. The idea at the core of the present research is that tool use can be considered as an instance of problem-solving situations supported by technical reasoning. In an eye-tracking study, we investigated the fixation patterns of participants (N = 32) looking at 3D images of thematically consistent (e.g., nail-steel hammer) and thematically inconsistent (e.g., scarf-steel hammer) object-tool pairs that could be either "hazardous" (accidentally electrified) or not. Results showed that under thematically consistent conditions, participants focused on the tool's manipulation area (e.g., the handle of a steel hammer). However, when electrified tools were present or when the visual scene was not action-prompting, regardless of the presence of electricity, the tools' functional/identity areas (e.g., the head of a steel hammer) were fixated longer than the tools' manipulation areas. These results support an integrated and reasoning-based approach to human tool use and document, for the first time, the crucial role of mechanical/semantic knowledge in tool visual exploration.

Tool and object affordances: An ecological eye-tracking study

In the present eye-tracking study, we analysed the visuo-spatial attentional patterns of participants looking at 3D images depicting single tools and object-tool pairs. The object-tool pairs could be thematically consistent, thematically inconsistent or spatially inconsistent. During the first 500 ms of visual exploration, tools were fixated longer on their functional area in all experimental conditions. However, extending the time-window of analysis to 1750 ms, the visual scene was encoded in a faster and more suited-for-action way in the thematically consistent condition (e.g., hammer-nail). Most important, the visual exploration of the thematically consistent pairs focused on the manipulation area of the tool (e.g., the handle of the hammer) more than on its functional area (e.g., the head of the hammer). Finally, when single tools were shown and the entire time-window of analysis was considered (1750 ms), fixation focused on the tool's manipulation area. These results are discussed within the reasoning-based framework of tool use. They highlight the relative role of the visuo-perceptual context in affordance perception and suggest a novel interpretation of the cognitive mechanisms underlying the processing of tools and object-tool pairs in terms of action reappraisal (i.e., a re-functionalization process when the action possibility is mined by the visuo-perceptual context).

The Role of Attention and Saccades on Parietofrontal Encoding of Contextual and Grasp-specific Affordances of Tools: An ERP Study

The ability to recognize a tool's affordances (how a spoon should be appropriately grasped and used), is vital for daily life. Prior research has identified parietofrontal circuits, including mirror neurons, to be critical in understanding affordances. However, parietofrontal action-encoding regions receive extensive visual input and are adjacent to parietofrontal attention control networks. It is unclear how eye movements and attention modulate parietofrontal encoding of affordances. To address this issue, scenes depicting tools in different use-contexts and grasp-postures were presented to healthy subjects across two experiments, with stimuli durations of 100 ms or 500 ms. The 100-ms experiment automatically restricted saccades and required covert attention, while the 500-ms experiment allowed overt attention. The two experiments elicited similar behavioral decisions on tool-use correctness and isolated the influence of attention on parietofrontal activity. Parietofrontal ERPs (P600) distinguishing tool-use contexts (e.g., spoon-yogurt vs. spoon-ball) were similar in both experiments. Conversely, parietofrontal ERPs distinguishing tool-grasps were characterized by posterior to frontal N130-N200 ERPs in the 100-ms experiment and by saccade-perturbed N130-N200 ERPs, frontal N400 and parietal P500 in the 500-ms experiment. Particularly, only overt gaze toward the hand-tool interaction engaged mirror neurons (frontal N400) when discerning grasps that manipulate but not functionally use a tool - (grasp bowl rather than stem of spoon). Results here detail the first human electrophysiological evidence on how attention selectively modulates multiple parietofrontal grasp-perception circuits, especially the mirror neuron system, while unaffecting parietofrontal encoding of tool-use contexts. These results are pertinent to neurophysiological models of affordances that typically neglect the role of attention in action perception.