The role of the sound of objects in object identification: evidence from picture naming

The label-feedback effect is influenced by target category in visual search

The label-feedback hypothesis states that language can modulate visual processing. In particular, hearing or reading aloud target names (labels) speeds up performance in visual search tasks by facilitating target detection and such advantage is often measured against a condition where the target name is shown visually (i.e. via the same modality as the search task). The current study conceptually complements and expands previous investigations. The effect of a multimodal label presentation (i.e., an audio+visual, AV, priming label) in a visual search task is compared to that of a multimodal (i.e. white noise+visual, NV, label) and two unimodal (i.e. audio, A, label or visual, V, label) control conditions. The name of a category (i.e. a label at the superordinate level) is used as a cue, instead of the more commonly used target name (a basic level label), with targets belonging to one of three categories: garments, improper weapons, and proper weapons. These categories vary for their structure, improper weapons being an ad hoc category (i.e. context-dependent), unlike proper weapons and garments. The preregistered analysis shows an overall facilitation of visual search performance in the AV condition compared to the NV condition, confirming that the label-feedback effect may not be explained away by the effects of multimodal stimulation only and that it extends to superordinate labels. Moreover, exploratory analyses show that such facilitation is driven by the garments and proper weapons categories, rather than improper weapons. Thus, the superordinate label-feedback effect is modulated by the structural properties of a category. These findings are consistent with the idea that the AV condition prompts an "up-regulation" of the label, a requirement for enhancing the label's beneficial effects, but not when the label refers to an ad hoc category. They also highlight the peculiar status of the category of improper weapons and set it apart from that of proper weapons.

Cutting Through the Noise: Auditory Scenes and Their Effects on Visual Object Processing

Despite the intuitive feeling that our visual experience is coherent and comprehensive, the world is full of ambiguous and indeterminate information. Here we explore how the visual system might take advantage of ambient sounds to resolve this ambiguity. Young adults (ns = 20-30) were tasked with identifying an object slowly fading in through visual noise while a task-irrelevant sound played. We found that participants demanded more visual information when the auditory object was incongruent with the visual object compared to when it was not. Auditory scenes, which are only probabilistically related to specific objects, produced similar facilitation even for unheard objects (e.g., a bench). Notably, these effects traverse categorical and specific auditory and visual-processing domains as participants performed across-category and within-category visual tasks, underscoring cross-modal integration across multiple levels of perceptual processing. To summarize, our study reveals the importance of audiovisual interactions to support meaningful perceptual experiences in naturalistic settings.

The Embodied Penman: Effector-Specific Motor-Language Integration During Handwriting

Several studies have illuminated how processing manual action verbs (MaVs) affects the programming or execution of concurrent hand movements. Here, to circumvent key confounds in extant designs, we conducted the first assessment of motor-language integration during handwriting-a task in which linguistic and motoric processes are co-substantiated. Participants copied MaVs, non-manual action verbs, and non-action verbs as we collected measures of motor programming and motor execution. Programming latencies were similar across conditions, but execution was faster for MaVs than for the other categories, regardless of whether word meanings were accessed implicitly or explicitly. In line with the Hand-Action-Network Dynamic Language Embodiment (HANDLE) model, such findings suggest that effector-congruent verbs can prime manual movements even during highly automatized tasks in which motoric and verbal processes are naturally intertwined. Our paradigm opens new avenues for fine-grained explorations of embodied language processes.

Compatibility between response position and either object typical size or semantic category: SNARC- and MARC-like effects in primary school children

In two experiments, we administered two binary classification tasks to third- to fifth-grade students and found that either the conceptual size (Experiment 1) or semantic category (Experiment 2) of the target picture interacted with response position: Participants emitted faster and/or more accurate right- and left-side responses to stimuli referring to typically large and small objects (i.e., a SNARC [spatial-numerical association of response codes]-like effect) or to living and nonliving objects (i.e., a MARC [markedness association of response codes]-like effect), respectively. The effect of either stimulus dimension was present only when this dimension was the task-relevant one. These findings may be accounted for by the polarity compatibility principle. In binary classification tasks, one of the two relevant stimulus values (i.e., the dominant one) and one of the two response values would be coded as having a positive polarity, whereas the other would be coded as having a negative polarity. Response selection would be faster and/or more accurate when stimulus and response polarities are compatible. According to a less parsimonious hypothesis, the polarity principle would only explain the effect of the stimulus semantic category in Experiment 2, whereas the effect of stimulus conceptual size in Experiment 1 might be traced back to the fact that children spatially code the typical size of stimulus referents (even if only when the task requires accessing this stimulus dimension). Small and large objects would be represented relatively on the left and right, respectively, which would result in faster and/or more accurate responses when response position is compatible with the stimulus referent's position in this spatial representation.

Does Perceptual Simulation Explain Spatial Effects in Word Categorization?

In three experiments we investigated the origin of the effects of the compatibility between the typical location of entities denoted by written words (e.g., “up” for eagle and “down” for carpet) and either the actual position of the words on the screen (e.g., upper vs. lower part of the screen), or the response position (e.g., upper- vs. lower- key presses) in binary categorization tasks. Contrary to predictions of the perceptual simulation account (Barsalou, 1999), conceptual spatial compatibility effects observed in the present study (faster RTs when the typical position of the stimulus referent in the real word was compatible with either the stimulus or response physical position) seem to be independent of whether there was an overlap between simulated processes possibly triggered by the presented stimulus and sensory-motor processes actually required by the task. Rather, they appear to depend critically on whether the involved stimulus and/or response dimensions had binary, variable (vs. fixed) values. Notably, no stimulus–stimulus compatibility effect was observed in Experiment 3, when the stimulus physical position was presented in a blocked design (i.e., it was kept constant within each block of trials). In contrast, in all three experiments, a compatibility effect between response position and another (non-spatial) conceptual dimension of the stimulus (i.e., its semantic category) was observed (i.e., an effect analogous to the MARC [linguistic markedness of response codes] effect, which is usually observed in the number domain; Nuerk et al., 2004). This pattern of results is fully accounted for by the polarity principle, according to which these effects originate from the alignment of the polarities of either different stimulus dimensions or stimulus and response dimensions.

An Exception to Mental Simulation: No Evidence for Embodied Odor Language

Do we mentally simulate olfactory information? We investigated mental simulation of odors and sounds in two experiments. Participants retained a word while they smelled an odor or heard a sound, then rated odor/sound intensity and recalled the word. Later odor/sound recognition was also tested, and pleasantness and familiarity judgments were collected. Word recall was slower when the sound and sound‐word mismatched (e.g., bee sound with the word typhoon). Sound recognition was higher when sounds were paired with a match or near‐match word (e.g., bee sound with bee or buzzer). This indicates sound‐words are mentally simulated. However, using the same paradigm no memory effects were observed for odor. Instead it appears odor‐words only affect lexical‐semantic representations, demonstrated by higher ratings of odor intensity and pleasantness when an odor was paired with a match or near‐match word (e.g., peach odor with peach or mango). These results suggest fundamental differences in how odor and sound‐words are represented.