I can see what you are doing: Action familiarity and affordance promote recovery from extinction

Investigating the cognitive processing of tools: Effects of dangerousness and directionality on attentional biases

Research has found a tool-based head-superiority effect (responses are faster to target stimuli that appear at the head than at the handle position). However, these studies did not consider the effects of directionality and dangerousness. This study investigated how directionality and dangerousness influence the head-superiority effect. Subjects were required to respond to the target location in all experiments. Experiment 1 manipulated the directionality, dangerousness and target location. Experiment 2 matched the sharpness of the tool tip in the directed conditions. Experiment 3 shortened the presentation time of the cue stimuli from 800 ms to 200 ms. Experiment 4 selected four tools with different functions to rule out an alternative explanation caused by functional repetition. The results indicate that a head-superiority effect emerges in the directed condition, and that dangerousness modulates the magnitude of this effect during the 800 ms condition. However, the modulation effect of dangerousness diminishes during the 200 ms condition.

The causal future: The influence of shape features caused by external transformation on visual attention

Previous studies have validated that participants can distinguish different origins of objects’ shape features, teasing apart features caused by transformation (causal history) from those of the original shape. Considering bite as a transformation example, two experiments were designed to investigate the effect of causal history on the allocation of visual attention. Participants were presented with regular and familiar complete or bitten shapes in Experiment 1 and unfamiliar and irregular complete or bitten shapes in Experiment 2 over a range of stimulus onset asynchronies (SOAs). The task was to identify different probes (i.e., punctuation marks) that equally appeared at four positions around these shapes. The results showed that complete regular shapes had no impact on participants’ reaction times to identify probes that appeared at the four different positions (Experiment 1), whereas complete irregular shapes would facilitate participants’ responses to the probes that appeared at the positions around the “head” of the irregular shape (Experiment 2) regardless of SOAs. When presented with bitten shapes, in the earlier phase of visual processing, participants’ response patterns resembled those found when complete shapes were presented. However, with longer SOAs, participants were faster in identifying probes that appeared at those positions that were around the nontransformed region of the bitten shapes. The results revealed that information about shape features caused by causal history could be incorporated, albeit relatively later, into the allocation of visual attention. The role of causal history in the speculation about one object's future development is discussed.

Handgrip Based Action Information Modulates Attentional Selection: An ERP Study

Prior work shows that the possibility of action to an object (visual affordance) facilitates attentional deployment. We sought to investigate the neural mechanisms underlying this modulation of attention by examining ERPs to target objects that were either congruently or incongruently gripped for their use in the presence of a congruently or incongruently gripped distractor. Participants responded to the presence or absence of a target object matching a preceding action word with a distractor object presented in the opposite location. Participants were faster in responding to congruently gripped targets compared to incongruently gripped targets. There was a reduced N2pc potential when the target was congruently gripped, and the distractor was incongruently gripped compared to the conditions where targets were incongruently gripped or when the distractor, as well as target, was congruently gripped. The N2pc results indicate that target selection is easier when action information is congruent with an object’s use.

Looking to recognise: the pre-eminence of semantic over sensorimotor processing in human tool use

Alongside language and bipedal locomotion, tool use is a characterizing activity of human beings. Current theories in the field embrace two contrasting approaches: "manipulation-based" theories, which are anchored in the embodied-cognition view, explain tool use as deriving from past sensorimotor experiences, whereas "reasoning-based" theories suggest that people reason about object properties to solve everyday-life problems. Here, we present results from two eye-tracking experiments in which we manipulated the visuo-perceptual context (thematically consistent vs. inconsistent object-tool pairs) and the goal of the task (free observation or looking to recognise). We found that participants exhibited reversed tools' visual-exploration patterns, focusing on the tool's manipulation area under thematically consistent conditions and on its functional area under thematically inconsistent conditions. Crucially, looking at the tools with the aim of recognising them produced longer fixations on the tools' functional areas irrespective of thematic consistency. In addition, tools (but not objects) were recognised faster in the thematically consistent conditions. These results strongly support reasoning-based theories of tool use, as they indicate that people primarily process semantic rather than sensorimotor information to interact with the environment in an agent's consistent-with-goal way. Such a pre-eminence of semantic processing challenges the mainstream embodied-cognition view of human tool use.

Tool and object affordances: An ecological eye-tracking study

In the present eye-tracking study, we analysed the visuo-spatial attentional patterns of participants looking at 3D images depicting single tools and object-tool pairs. The object-tool pairs could be thematically consistent, thematically inconsistent or spatially inconsistent. During the first 500 ms of visual exploration, tools were fixated longer on their functional area in all experimental conditions. However, extending the time-window of analysis to 1750 ms, the visual scene was encoded in a faster and more suited-for-action way in the thematically consistent condition (e.g., hammer-nail). Most important, the visual exploration of the thematically consistent pairs focused on the manipulation area of the tool (e.g., the handle of the hammer) more than on its functional area (e.g., the head of the hammer). Finally, when single tools were shown and the entire time-window of analysis was considered (1750 ms), fixation focused on the tool's manipulation area. These results are discussed within the reasoning-based framework of tool use. They highlight the relative role of the visuo-perceptual context in affordance perception and suggest a novel interpretation of the cognitive mechanisms underlying the processing of tools and object-tool pairs in terms of action reappraisal (i.e., a re-functionalization process when the action possibility is mined by the visuo-perceptual context).

Dorsal and ventral stream contribution to the paired-object affordance effect

Visual extinction, a parietal syndrome in which patients exhibit perceptual impairments when two objects are simultaneously presented in the visual field, is reduced when objects are correctly positioned for action, indicating that action helps patients' visual attention. Similarly, healthy individuals make faster action decisions on object pairs that appear in left/right standard co-location for actions in comparison to object pairs that appear in a mirror location, a phenomenon called the paired-object affordance effect. However, the neural locus of such effect remains debated and may be related to the activity of ventral or dorsal brain regions. The present fMRI study aims at determining the neural substrates of the paired-object affordance effect. Fourteen right-handed participants made decisions about semantically related (i.e. thematically related and co-manipulated) and unrelated object pairs. Pairs were either positioned in a standard location for a right-handed action (with the active object - lid - in the right visual hemifield, and the passive object - pan - in the left visual hemifield), or in the reverse location. Behavioral results showed a suppression of the observed cost of correctly positioning related pairs for action when performing action decisions (deciding if the two objects are usually used together), but not when performing contextual decisions (deciding if the two objects are typically found in the kitchen). Anterior regions of the dorsal stream (e.g. supplementary motor area) responded to inadequate object co-positioning for action, but only when the perceptual task required action decisions. In the ventral cortex, the left lateral occipital complex showed increased activation for objects correctly positioned for action in all conditions except when neither task demands nor object relatedness was relevant for action. Thus, fMRI results demonstrated a joint contribution of ventral and dorsal cortical streams to the paired-affordance effect. They further suggest that this contribution may depend on contextual situations and task demands, in line with flexible views of affordance evocation.

Age effects on associative memory for novel picture pairings

Normal aging is usually accompanied by greater memory decline for associations than for single items. Though associative memory is generally supported by recollection, it has been suggested that familiarity can also contribute to associative memory when stimuli can be unitized and encoded as a single entity. Given that familiarity remains intact during healthy aging, this may be one route to reducing age-related associative deficits. The current study investigated age-related differences in associative memory under conditions that were expected to differentially promote unitization, in this case by manipulating the spatial arrangement of two semantically unrelated objects positioned relative to each other in either spatially implausible or plausible orientations. Event-related potential (ERP) correlates of item and associative memory were recorded whilst younger and older adults were required to discriminate between old, recombined and new pairs of objects. These ERP correlates of item and associative memory did not vary with plausibility, whereas behavioral measures revealed that both associative and item memory were greater for spatially plausible than implausible pair arrangements. Contrary to predictions, older adults were less able to take advantage of this memory benefit than younger participants. Potential reasons for this are considered, and these are informed by those lines of evidence which indicate older participants were less sensitive to the bottom-up spatial manipulation employed here. It is recommended that future strategies for redressing age-related associative deficits should take account of the aging brain's increasing reliance on pre-existing semantic associations.

The involvement of the dorsal stream in processing implied actions between paired objects: A TMS study

Perceiving and selecting the action possibilities (affordances) provided by objects is an important challenge to human vision, and is not limited to single-object scenarios. Xu et al. (2015) identified two effects of implied actions between paired objects on response selection: an inhibitory effect on responses aligned with the passive object in the pair (e.g. a bowl) and an advantage associated with responses aligned with the active objects (e.g. a spoon). The present study investigated the neurocognitive mechanisms behind these effects by examining the involvement of the ventral (vision for perception) and the dorsal (vision for action) visual streams, as defined in Goodale and Milner's (1992) two visual stream theory. Online repetitive transcranial magnetic stimulation (rTMS) applied to the left anterior intraparietal sulcus (aIPS) reduced both the inhibitory effect of implied actions on responses aligned with the passive objects and the advantage of those aligned with the active objects, but only when the active objects were contralateral to the stimulation. rTMS to the left lateral occipital areas (LO) did not significantly alter the influence of implied actions. The results reveal that the dorsal visual stream is crucial not only in single-object affordance processing, but also in responding to implied actions between objects.

Functional Fixedness in Creative Thinking Tasks Depends on Stimulus Modality

Pictorial examples during creative thinking tasks can lead participants to fixate on these examples and reproduce their elements even when yielding suboptimal creative products. Semantic memory research may illuminate the cognitive processes underlying this effect. Here, we examined whether pictures and words differentially influence access to semantic knowledge for object concepts depending on whether the task is close- or open-ended. Participants viewed either names or pictures of everyday objects, or a combination of the two, and generated common, secondary, or ad hoc uses for them. Stimulus modality effects were assessed quantitatively through reaction times and qualitatively through a novel coding system, which classifies creative output on a continuum from top-down-driven to bottom-up-driven responses. Both analyses revealed differences across tasks. Importantly, for ad hoc uses, participants exposed to pictures generated more top-down-driven responses than those exposed to object names. These findings have implications for accounts of functional fixedness in creative thinking, as well as theories of semantic memory for object concepts.

Effects of broken affordance on visual extinction

Previous studies have shown that visual extinction can be reduced if two objects are positioned to “afford” an action. Here we tested if this affordance effect was disrupted by “breaking” the affordance, i.e., if one of the objects actively used in the action had a broken handle. We assessed the effects of broken affordance on recovery from extinction in eight patients with right hemisphere lesions and left-sided extinction. Patients viewed object pairs that were or were not commonly used together and that were positioned for left- or right-hand actions. In the unrelated pair conditions, either two tools or two objects were presented. In line with previous research (e.g., Riddoch et al., 2006), extinction was reduced when action-related object pairs and when unrelated tool pairs were presented compared to unrelated object pairs. There was no significant difference in recovery rate between action-related (object-tool) and unrelated tool pairs. In addition, performance with action-related objects decreased when the tool appeared on the ipsilesional side compared to when it was on the contralesional side, but only when the tool handle was intact. There were minimal effects of breaking the handle of an object rather than a tool, and there was no effect of breaking the handle on either tools or objects on single item trials. The data suggest that breaking the handle of a tool lessens the degree to which it captures attention, with this attentional capture being strongest when the tool appears on the ipsilesional side. The capture of attention by the ipsilesional item then reduces the chance of detecting the contralesional stimulus. This attentional capture effect is mediated by the affordance to the intact tool.

Visual field asymmetries in object individuation

Repetition blindness (RB) is a failure to detect both instances of two identical stimuli presented in close temporal proximity. It is due to an inability to form separate episodic tokens for a repeated stimulus, resulting in a single conscious representation. In three experiments, participants identified two targets presented simultaneously in different spatial locations. These stimuli were either the same or different. In two experiments the targets occurred on either side of fixation, and in a third experiment both were in the same hemifield. In all experiments, RB was more pronounced for stimuli in the right hemifield. In addition, there was a left hemifield advantage for both repeated and non-repeated stimuli when the two stimuli occurred in opposite visual fields and, thus, were processed by different hemispheres. These findings suggest that the right hemisphere plays a dominant role in attentional selection and in creating conscious representations of visual events.

Attention is required for the perceptual integration of action object pairs

Previous studies have demonstrated that functionally related objects are perceptually grouped during visual identification if they are depicted as if interacting with each other (Green and Hummel in J Exp Psychol Hum Percept Perform 32(5):1107-1119, 2006). However, it is unclear whether this integration requires attention or occurs pre-attentively. Here, we used a divided-attention task with variable attentional load to address this question. Participants matched a word label to a target object that was immediately preceded by a briefly presented, task-irrelevant tool that was either functionally related or unrelated to the word label (e.g., axe | "log" or hammer | "log"). The tool was either positioned to interact with the target object or faced away from it. The amount of attention available to process the tool was manipulated by asking participants to make a concurrent perceptual discrimination of varying difficulty on a surrounding frame stimulus. The previously demonstrated advantage for the related-and-interacting condition was replicated under conditions of no or low attentional load. This benefit disappeared under high competing attentional load, indicating that attention is required to integrate functionally related objects together into a single perceptual unit.

Selecting object pairs for action: Is the active object always first?

Perception is linked to action via two routes: a direct route based on affordance information in the environment and an indirect route based on semantic knowledge about objects. The present study explored the factors modulating the recruitment of the two routes, in particular which factors affecting the selection of paired objects. In Experiment 1, we presented real objects among semantically related or unrelated distracters. Participants had to select two objects that can interact. The presence of distracters affected selection times, but not the semantic relations of the objects with the distracters. Furthermore, participants first selected the active object (e.g. teaspoon) with their right hand, followed by the passive object (e.g. mug), often with their left hand. In Experiment 2, we presented pictures of the same objects with no hand grip, congruent or incongruent hand grip. Participants had to decide whether the two objects can interact. Action decisions were faster when the presentation of the active object preceded the presentation of the passive object, and when the grip was congruent. Interestingly, participants were slower when the objects were semantically but not functionally related; this effect increased with congruently gripped objects. Our data showed that action decisions in the presence of strong affordance cues (real objects, pictures of congruently gripped objects) relied on sensory-motor representation, supporting the direct route from perception-to-action that bypasses semantic knowledge. However, in the case of weak affordance cues (pictures), semantic information interfered with action decisions, indicating that semantic knowledge impacts action decisions. The data support the dual-route account from perception-to-action.

Tool use as distributed cognition: how tools help, hinder and define manual skill

Our thesis in this paper is that, in order to appreciate the interplay between cognitive (goal-directed) and physical performance in tool use, it is necessary to determine the role that representations play in the use of tools. We argue that rather being solely a matter of internal (mental) representation, tool use makes use of the external representations that define the human–environment–tool–object system. This requires the notion of Distributed Cognition to encompass not simply the manner in which artifacts represent concepts but also how they represent praxis. Our argument is that this can be extended to include how artifacts-in-context afford use and how this response to affordances constitutes a particular form of skilled performance. By artifacts-in-context, we do not mean solely the affordances offered by the physical dimensions of a tool but also the interaction between the tool and the object that it is being used on. From this, “affordance” does not simply relate to the physical appearance of the tool but anticipates subsequent actions by the user directed towards the goal of changing the state of the object and this is best understood in terms of the “complimentarity” in the system. This assertion raises two challenges which are explored in this paper. The first is to distinguish “affordance” from the adaptation that one might expect to see in descriptions of motor control; when we speak of “affordance” as a form of anticipation, don’t we just mean the ability to adjust movements in response to physical demands? The second is to distinguish “affordance” from a schema of the tool; when we talk about anticipation, don’t we just mean the ability to call on a schema representing a “recipe” for using that tool for that task? This question of representation, specifically what knowledge needs to be represented in tool use, is central to this paper.

Impaired texture segregation but spared contour integration following damage to right posterior parietal cortex

We examined the relations between texture segregation and contour integration in patients with deficits in spatial attention leading to left or right hemisphere extinction. Patients and control participants were presented with texture and contour stimuli consisting of oriented elements. We induced regularity in the stimuli by manipulating the element orientations resulting in an implicit texture border or explicit contour. Participants had to discriminate curved from straight shapes without making eye movements, while the stimulus presentation time was varied using a QUEST procedure. The results showed that only patients with right hemisphere extinction had a spatial bias, needing a longer presentation time to determine the shape of the border or contour on the contralesional side, especially for borders defined by texture. These results indicate that texture segregation is modulated by attention-related brain areas in the right posterior parietal cortex.

The distributed representation of random and meaningful object pairs in human occipitotemporal cortex: the weighted average as a general rule

Natural scenes typically contain multiple visual objects, often in interaction, such as when a bottle is used to fill a glass. Previous studies disagree about the representation of multiple objects and the role of object position herein, nor did they pinpoint the effect of potential interactions between the objects. In an fMRI study, we presented four single objects in two different positions and object pairs consisting of all possible combinations of the single objects. Objects pairs could form either a meaningful action configuration in which they interact with each other or a non-meaningful configuration. We found that for single objects and object pairs both identity and position were represented in multi-voxel activity patterns in LOC. The response patterns of object pairs were best predicted by a weighted average of the response patterns of the constituent objects, with the strongest single-object response (the max response) weighted more than the min response. The difference in weight between the max and the min object was larger for familiar action pairs than for other pairs when participants attended to the configuration. A weighted average thus relates the response patterns of object pairs to the response patterns of single objects, even when the objects interact.

Where do objects become scenes?

Regions tuned to individual visual categories, such as faces and objects, have been discovered in the later stages of the ventral visual pathway in the cortex. But most visual experience is composed of scenes, where multiple objects are interacting. Such interactions are readily described by prepositions or verb forms, for example, a bird perched on a birdhouse. At what stage in the pathway does sensitivity to such interactions arise? Here we report that object pairs shown as interacting, compared with their side-by-side depiction (e.g., a bird besides a birdhouse), elicit greater activity in the lateral occipital complex, the earliest cortical region where shape is distinguished from texture. Novelty of the interactions magnified this gain, an effect that was absent in the side-by-side depictions. Scene-like relations are thus likely achieved simultaneously with the specification of object shape.

Functional relations trump implied motion in recovery from extinction: evidence from the effects of animacy on extinction

Patients with extinction show a characteristic impairment in the identification of objects when two items are presented simultaneously, typically reporting the ipsilesional item only. The effect is thought to be due to a spatial bias advantaging the ipsilesional item under conditions of competing concurrent stimulation. Action relations between objects can result in recovery from extinction as the object pair may be perceived as a single group rather than competing perceptual units. However, objects interacting together can also have implied motion. Here we test whether implied motion is necessary to generate recovery from extinction. We varied orthogonally whether animate and inanimate objects were paired together in positions related or unrelated to action. Implied motion was greater when an animate object was present than when both stimuli were inanimate. Despite this, recovery from extinction was greater when actions were shown between inanimate objects. We suggest that actions between inanimate objects are perceived more easily due to the surfaces of these stimuli being designed for functional goals (e.g., the flat surface of a hammer head is designed to hit the flattened head of a nail). Attention is sensitive to the fit between potential action and the functional properties of objects, and not just to implied motion between stimuli.

Action-related objects influence the distribution of visuospatial attention

Previous studies have shown that attention is drawn to the location of manipulable objects and is distributed across pairs of objects that are positioned for action. Here, we investigate whether central, action-related objects can cue attention to peripheral targets. Experiment 1 compared the effect of uninformative arrow and object cues on a letter discrimination task. Arrow cues led to spatial-cueing benefits across a range of stimulus onset asynchronies (SOAs: 0 ms, 120 ms, 400 ms), but object-cueing benefits were slow to build and were only significant at the 400-ms SOA. Similar results were found in Experiment 2, in which the targets were objects that could be either congruent or incongruent with the cue (e.g., screwdriver and screw versus screwdriver and glass). Cueing benefits were not influenced by the congruence between the cue and target, suggesting that the cueing effects reflected the action implied by the central object, not the interaction between the objects. For Experiment 3 participants decided whether the cue and target objects were related. Here, the interaction between congruent (but not incongruent) targets led to significant cueing/positioning benefits at all three SOAs. Reduced cueing benefits were obtained in all three experiments when the object cue did not portray a legitimate action (e.g., a bottle pointing towards an upper location, since a bottle cannot pour upwards), suggesting that it is the perceived action that is critical, rather than the structural properties of individual objects. The data suggest that affordance for action modulates the allocation of visual attention.

Action relationships concatenate representations of separate objects in the ventral visual system

Objects in the real world are encountered in contexts where they interact together. Though it is known that neurons in the ventral visual stream mediate the recognition of individual objects, we have minimal knowledge of how multiple objects are processed at a neural level. We examined the neural response to pairs of objects using functional magnetic resonance imaging (fMRI). Objects positioned to interact together activated bilateral lateral occipital complex (LOC) and fusiform gyrus. This occurred irrespective of whether the objects were attended. In LOC, the effect of positioning objects for action was found regardless of whether the objects formed a familiar or unfamiliar action pair. In the fusiform gyrus activation was found when objects formed a familiar action pair. No changes were apparent in visuomotor (premotor and parietal) regions which might reflect a motor-based response to objects. These results show that ventral-stream regions respond to the interaction between objects, as well as to the sensory and functional properties of individual objects.

Sentence comprehension and simulation of object temporary, canonical and stable affordances

Two experiments investigate the effects of language comprehension on affordances. Participants read a sentence composed by either an observation or an action verb (Look at/Grasp) followed by an object name. They had to decide whether the visual object following the sentence was the same as the one mentioned in the sentence. Objects graspable with either a precision or a power grip were presented in an orientation affording action (canonical) or not. Action sentences were faster than observation sentences, and power grip objects were faster than precision grip objects. Moreover, faster RTs were obtained when orientation afforded action. Results indicate that the simulation activated during language comprehension leads to the formation of a "motor prototype" of the object. This motor prototype encodes information on temporary/canonical and stable affordances (e.g., orientation, size), which can be possibly referred to different cognitive and neural systems (dorsal, ventral systems).