1
|
Wegner-Clemens K, Malcolm GL, Shomstein S. Predicting attentional allocation in real-world environments: The need to investigate crossmodal semantic guidance. WILEY INTERDISCIPLINARY REVIEWS. COGNITIVE SCIENCE 2024; 15:e1675. [PMID: 38243393 DOI: 10.1002/wcs.1675] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Revised: 12/01/2023] [Accepted: 12/07/2023] [Indexed: 01/21/2024]
Abstract
Real-world environments are multisensory, meaningful, and highly complex. To parse these environments in a highly efficient manner, a subset of this information must be selected both within and across modalities. However, the bulk of attention research has been conducted within sensory modalities, with a particular focus on vision. Visual attention research has made great strides, with over a century of research methodically identifying the underlying mechanisms that allow us to select critical visual information. Spatial attention, attention to features, and object-based attention have all been studied extensively. More recently, research has established semantics (meaning) as a key component to allocating attention in real-world scenes, with the meaning of an item or environment affecting visual attentional selection. However, a full understanding of how semantic information modulates real-world attention requires studying more than vision in isolation. The world provides semantic information across all senses, but with this extra information comes greater complexity. Here, we summarize visual attention (including semantic-based visual attention), crossmodal attention, and argue for the importance of studying crossmodal semantic guidance of attention. This article is categorized under: Psychology > Attention Psychology > Perception and Psychophysics.
Collapse
Affiliation(s)
- Kira Wegner-Clemens
- Psychological and Brain Sciences, George Washington University, Washington, DC, USA
| | | | - Sarah Shomstein
- Psychological and Brain Sciences, George Washington University, Washington, DC, USA
| |
Collapse
|
2
|
Zhao S, Zhou Y, Ma F, Xie J, Feng C, Feng W. The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink. Front Neurosci 2023; 17:1295010. [PMID: 38161792 PMCID: PMC10755906 DOI: 10.3389/fnins.2023.1295010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2023] [Accepted: 11/29/2023] [Indexed: 01/03/2024] Open
Abstract
Introduction Recent studies have found that the sound-induced alleviation of visual attentional blink, a well-known phenomenon exemplifying the beneficial influence of multisensory integration on time-based attention, was larger when that sound was semantically congruent relative to incongruent with the second visual target (T2). Although such an audiovisual congruency effect has been attributed mainly to the semantic conflict carried by the incongruent sound restraining that sound from facilitating T2 processing, it is still unclear whether the integrated semantic information carried by the congruent sound benefits T2 processing. Methods To dissociate the congruence-induced benefit and incongruence-induced reduction in the alleviation of visual attentional blink at the behavioral and neural levels, the present study combined behavioral measures and event-related potential (ERP) recordings in a visual attentional blink task wherein the T2-accompanying sound, when delivered, could be semantically neutral in addition to congruent or incongruent with respect to T2. Results The behavioral data clearly showed that compared to the neutral sound, the congruent sound improved T2 discrimination during the blink to a higher degree while the incongruent sound improved it to a lesser degree. The T2-locked ERP data revealed that the early occipital cross-modal N195 component (192-228 ms after T2 onset) was uniquely larger in the congruent-sound condition than in the neutral-sound and incongruent-sound conditions, whereas the late parietal cross-modal N440 component (400-500 ms) was prominent only in the incongruent-sound condition. Discussion These findings provide strong evidence that the modulating effect of audiovisual semantic congruency on the sound-induced alleviation of visual attentional blink contains not only a late incongruence-induced cost but also an early congruence-induced benefit, thereby demonstrating for the first time an unequivocal congruent-sound-induced benefit in alleviating the limitation of time-based visual attention.
Collapse
Affiliation(s)
- Song Zhao
- Department of Psychology, School of Education, Soochow University, Suzhou, China
| | - Yuxin Zhou
- Department of Psychology, School of Education, Soochow University, Suzhou, China
| | - Fangfang Ma
- Department of Psychology, School of Education, Soochow University, Suzhou, China
| | - Jimei Xie
- Department of Psychology, School of Education, Soochow University, Suzhou, China
| | - Chengzhi Feng
- Department of Psychology, School of Education, Soochow University, Suzhou, China
| | - Wenfeng Feng
- Department of Psychology, School of Education, Soochow University, Suzhou, China
- Research Center for Psychology and Behavioral Sciences, Soochow University, Suzhou, China
| |
Collapse
|
3
|
Zhao S, Wang C, Chen M, Zhai M, Leng X, Zhao F, Feng C, Feng W. Cross-modal enhancement of spatially unpredictable visual target discrimination during the attentional blink. Atten Percept Psychophys 2023; 85:2178-2195. [PMID: 37312000 DOI: 10.3758/s13414-023-02739-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/20/2023] [Indexed: 06/15/2023]
Abstract
The attentional blink can be substantially reduced by delivering a task-irrelevant sound synchronously with the second target (T2) embedded in a rapid serial visual presentation stream, which is further modulated by the semantic congruency between the sound and T2. The present study extended the cross-modal boost during attentional blink and the modulation of audiovisual semantic congruency in the spatial domain by showing that a spatially uninformative, semantically congruent (but not incongruent) sound could even improve the discrimination of spatially unpredictable T2 during attentional blink. T2-locked event-related potential (ERP) data yielded that the early cross-modal P195 difference component (184-234 ms) over the occipital scalp contralateral to the T2 location was larger preceding accurate than inaccurate discriminations of semantically congruent, but not incongruent, audiovisual T2s. Interestingly, the N2pc component (194-244 ms) associated with visual-spatial attentional allocation was enlarged for incongruent audiovisual T2s relative to congruent audiovisual and unisensory visual T2s only when they were accurately discriminated. These ERP findings suggest that the spatially extended cross-modal boost during attentional blink involves an early cross-modal interaction strengthening the perceptual processing of T2, without any sound-induced enhancement of visual-spatial attentional allocation toward T2. In contrast, the absence of an accuracy decrease in response to semantically incongruent audiovisual T2s may originate from the semantic mismatch capturing extra visual-spatial attentional resources toward T2.
Collapse
Affiliation(s)
- Song Zhao
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China
| | - Chongzhi Wang
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China
| | - Minran Chen
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China
| | - Mengdie Zhai
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China
| | - Xuechen Leng
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China
| | - Fan Zhao
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China
| | - Chengzhi Feng
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China.
| | - Wenfeng Feng
- Department of Psychology, School of Education, Soochow University, Suzhou, 215123, Jiangsu, China.
- Research Center for Psychology and Behavioral Sciences, Soochow University, Suzhou, 215123, Jiangsu, China.
| |
Collapse
|
4
|
Jiang Y, Qiao R, Shi Y, Tang Y, Hou Z, Tian Y. The effects of attention in auditory-visual integration revealed by time-varying networks. Front Neurosci 2023; 17:1235480. [PMID: 37600005 PMCID: PMC10434229 DOI: 10.3389/fnins.2023.1235480] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Accepted: 07/17/2023] [Indexed: 08/22/2023] Open
Abstract
Attention and audiovisual integration are crucial subjects in the field of brain information processing. A large number of previous studies have sought to determine the relationship between them through specific experiments, but failed to reach a unified conclusion. The reported studies explored the relationship through the frameworks of early, late, and parallel integration, though network analysis has been employed sparingly. In this study, we employed time-varying network analysis, which offers a comprehensive and dynamic insight into cognitive processing, to explore the relationship between attention and auditory-visual integration. The combination of high spatial resolution functional magnetic resonance imaging (fMRI) and high temporal resolution electroencephalography (EEG) was used. Firstly, a generalized linear model (GLM) was employed to find the task-related fMRI activations, which was selected as regions of interesting (ROIs) for nodes of time-varying network. Then the electrical activity of the auditory-visual cortex was estimated via the normalized minimum norm estimation (MNE) source localization method. Finally, the time-varying network was constructed using the adaptive directed transfer function (ADTF) technology. Notably, Task-related fMRI activations were mainly observed in the bilateral temporoparietal junction (TPJ), superior temporal gyrus (STG), primary visual and auditory areas. And the time-varying network analysis revealed that V1/A1↔STG occurred before TPJ↔STG. Therefore, the results supported the theory that auditory-visual integration occurred before attention, aligning with the early integration framework.
Collapse
Affiliation(s)
- Yuhao Jiang
- Institute for Advanced Sciences, Chongqing University of Posts and Telecommunications, Chongqing, China
- Guangyang Bay Laboratory, Chongqing Institute for Brain and Intelligence, Chongqing, China
- Central Nervous System Drug Key Laboratory of Sichuan Province, Luzhou, China
| | - Rui Qiao
- Institute for Advanced Sciences, Chongqing University of Posts and Telecommunications, Chongqing, China
- Guangyang Bay Laboratory, Chongqing Institute for Brain and Intelligence, Chongqing, China
| | - Yupan Shi
- Institute for Advanced Sciences, Chongqing University of Posts and Telecommunications, Chongqing, China
- Guangyang Bay Laboratory, Chongqing Institute for Brain and Intelligence, Chongqing, China
| | - Yi Tang
- Institute for Advanced Sciences, Chongqing University of Posts and Telecommunications, Chongqing, China
- Guangyang Bay Laboratory, Chongqing Institute for Brain and Intelligence, Chongqing, China
| | - Zhengjun Hou
- Institute for Advanced Sciences, Chongqing University of Posts and Telecommunications, Chongqing, China
- Guangyang Bay Laboratory, Chongqing Institute for Brain and Intelligence, Chongqing, China
| | - Yin Tian
- Institute for Advanced Sciences, Chongqing University of Posts and Telecommunications, Chongqing, China
- Guangyang Bay Laboratory, Chongqing Institute for Brain and Intelligence, Chongqing, China
| |
Collapse
|
5
|
Nambiar K, Bhargava P. An Exploration of the Effects of Cross-Modal Tasks on Selective Attention. Behav Sci (Basel) 2023; 13:bs13010051. [PMID: 36661623 PMCID: PMC9854760 DOI: 10.3390/bs13010051] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2022] [Revised: 12/23/2022] [Accepted: 12/31/2022] [Indexed: 01/09/2023] Open
Abstract
Successful performance of a task relies on selectively attending to the target, while ignoring distractions. Studies on perceptual load theory (PLT), conducted involving independent tasks with visual and auditory modalities, have shown that if a task is low-load, distractors and the target are both processed. If the task is high-load, distractions are not processed. The current study expands these findings by considering the effect of cross-modality (target and distractor from separate modalities) and congruency (similarity of target and distractor) on selective attention, using a word-identification task. Parameters were analysed, including response time, accuracy rates, congruency of distractions, and subjective report of load. In contrast to past studies on PLT, the results of the current study show that modality (congruency of the distractors) had a significant effect and load had no effect on selective attention. This study demonstrates that subjective measurement of load is important when studying perceptual load and selective attention.
Collapse
|
6
|
Yuan Y, He X, Yue Z. Working memory load modulates the processing of audiovisual distractors: A behavioral and event-related potentials study. Front Integr Neurosci 2023; 17:1120668. [PMID: 36908504 PMCID: PMC9995450 DOI: 10.3389/fnint.2023.1120668] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2022] [Accepted: 01/30/2023] [Indexed: 02/25/2023] Open
Abstract
The interplay between different modalities can help to perceive stimuli more effectively. However, very few studies have focused on how multisensory distractors affect task performance. By adopting behavioral and event-related potentials (ERPs) techniques, the present study examined whether multisensory audiovisual distractors could attract attention more effectively than unisensory distractors. Moreover, we explored whether such a process was modulated by working memory load. Across three experiments, n-back tasks (1-back and 2-back) were adopted with peripheral auditory, visual, or audiovisual distractors. Visual and auditory distractors were white discs and pure tones (Experiments 1 and 2), pictures and sounds of animals (Experiment 3), respectively. Behavioral results in Experiment 1 showed a significant interference effect under high working memory load but not under low load condition. The responses to central letters with audiovisual distractors were significantly slower than those to letters without distractors, while no significant difference was found between unisensory distractor and without distractor conditions. Similarly, ERP results in Experiments 2 and 3 showed that there existed an integration only under high load condition. That is, an early integration for simple audiovisual distractors (240-340 ms) and a late integration for complex audiovisual distractors (440-600 ms). These findings suggest that multisensory distractors can be integrated and effectively attract attention away from the main task, i.e., interference effect. Moreover, this effect is pronounced only under high working memory load condition.
Collapse
Affiliation(s)
- Yichen Yuan
- Department of Psychology, Sun Yat-sen University, Guangzhou, China
| | - Xiang He
- Department of Psychology, Sun Yat-sen University, Guangzhou, China
| | - Zhenzhu Yue
- Department of Psychology, Sun Yat-sen University, Guangzhou, China
| |
Collapse
|
7
|
Zimmer U, Wendt M, Pacharra M. Enhancing allocation of visual attention with emotional cues presented in two sensory modalities. BEHAVIORAL AND BRAIN FUNCTIONS : BBF 2022; 18:10. [PMID: 36138461 PMCID: PMC9494825 DOI: 10.1186/s12993-022-00195-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Accepted: 08/27/2022] [Indexed: 11/10/2022]
Abstract
Background Responses to a visual target stimulus in an exogenous spatial cueing paradigm are usually faster if cue and target occur in the same rather than in different locations (i.e., valid vs. invalid), although perceptual conditions for cue and target processing are otherwise equivalent. This cueing validity effect can be increased by adding emotional (task-unrelated) content to the cue. In contrast, adding a secondary non-emotional sensory modality to the cue (bimodal), has not consistently yielded increased cueing effects in previous studies. Here, we examined the interplay of bimodally presented cue content (i.e., emotional vs. neutral), by using combined visual-auditory cues. Specifically, the current ERP-study investigated whether bimodal presentation of fear-related content amplifies deployment of spatial attention to the cued location. Results A behavioral cueing validity effect occurred selectively in trials in which both aspects of the cue (i.e., face and voice) were related to fear. Likewise, the posterior contra-ipsilateral P1-activity in valid trials was significantly larger when both cues were fear-related than in all other cue conditions. Although the P3a component appeared uniformly increased in invalidly cued trials, regardless of cue content, a positive LPC deflection, starting about 450 ms after target onset, was, again, maximal for the validity contrast in trials associated with bimodal presentation of fear-related cues. Conclusions Simultaneous presentation of fear-related stimulus information in the visual and auditory modality appears to increase sustained visual attention (impairing disengagement of attention from the cued location) and to affect relatively late stages of target processing.
Collapse
Affiliation(s)
- Ulrike Zimmer
- Faculty of Human Sciences, Department of Psychology, MSH Medical School Hamburg, Hamburg, Germany. .,ICAN Insitute of Cognitive and Affective Neuroscience, MSH Medical School Hamburg, Hamburg, Germany.
| | - Mike Wendt
- Faculty of Human Sciences, Department of Psychology, MSH Medical School Hamburg, Hamburg, Germany.,ICAN Insitute of Cognitive and Affective Neuroscience, MSH Medical School Hamburg, Hamburg, Germany
| | - Marlene Pacharra
- Faculty of Psychology, Department of Biopsychology, Institute of Cognitive Neuroscience, Ruhr-University Bochum, Bochum, Germany
| |
Collapse
|
8
|
Semantically congruent audiovisual integration with modal-based attention accelerates auditory short-term memory retrieval. Atten Percept Psychophys 2022; 84:1625-1634. [PMID: 35641858 DOI: 10.3758/s13414-021-02437-4] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/28/2021] [Indexed: 11/08/2022]
Abstract
Evidence has shown that multisensory integration benefits to unisensory perception performance are asymmetric and that auditory perception performance can receive more multisensory benefits, especially when the attention focus is directed toward a task-irrelevant visual stimulus. At present, whether the benefits of semantically (in)congruent multisensory integration with modal-based attention for subsequent unisensory short-term memory (STM) retrieval are also asymmetric remains unclear. Using a delayed matching-to-sample paradigm, the present study investigated this issue by manipulating the attention focus during multisensory memory encoding. The results revealed that both visual and auditory STM retrieval reaction times were faster under semantically congruent multisensory conditions than under unisensory memory encoding conditions. We suggest that coherent multisensory representation formation might be optimized by restricted multisensory encoding and can be rapidly triggered by subsequent unisensory memory retrieval demands. Crucially, auditory STM retrieval is exclusively accelerated by semantically congruent multisensory memory encoding, indicating that the less effective sensory modality of memory retrieval relies more on the coherent prior formation of a multisensory representation optimized by modal-based attention.
Collapse
|
9
|
How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness. Atten Percept Psychophys 2022; 84:1317-1327. [PMID: 35449432 DOI: 10.3758/s13414-022-02488-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/27/2022] [Indexed: 11/08/2022]
Abstract
Semantic information about objects, events, and scenes influences how humans perceive, interact with, and navigate the world. The semantic information about any object or event can be highly complex and frequently draws on multiple sensory modalities, which makes it difficult to quantify. Past studies have primarily relied on either a simplified binary classification of semantic relatedness based on category or on algorithmic values based on text corpora rather than human perceptual experience and judgement. With the aim to further accelerate research into multisensory semantics, we created a constrained audiovisual stimulus set and derived similarity ratings between items within three categories (animals, instruments, household items). A set of 140 participants provided similarity judgments between sounds and images. Participants either heard a sound (e.g., a meow) and judged which of two pictures of objects (e.g., a picture of a dog and a duck) it was more similar to, or saw a picture (e.g., a picture of a duck) and selected which of two sounds it was more similar to (e.g., a bark or a meow). Judgements were then used to calculate similarity values of any given cross-modal pair. An additional 140 participants provided word judgement to calculate similarity of word-word pairs. The derived and reported similarity judgements reflect a range of semantic similarities across three categories and items, and highlight similarities and differences among similarity judgments between modalities. We make the derived similarity values available in a database format to the research community to be used as a measure of semantic relatedness in cognitive psychology experiments, enabling more robust studies of semantics in audiovisual environments.
Collapse
|
10
|
Almadori E, Mastroberardino S, Botta F, Brunetti R, Lupiáñez J, Spence C, Santangelo V. Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance Short-Term Memory Performance. Brain Sci 2021; 11:brainsci11091206. [PMID: 34573227 PMCID: PMC8467083 DOI: 10.3390/brainsci11091206] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2021] [Revised: 08/30/2021] [Accepted: 09/09/2021] [Indexed: 11/17/2022] Open
Abstract
Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene.
Collapse
Affiliation(s)
- Erika Almadori
- Neuroimaging Laboratory, IRCCS Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, Italy;
| | - Serena Mastroberardino
- Department of Psychology, School of Medicine & Psychology, Sapienza University of Rome, Via dei Marsi 78, 00185 Rome, Italy;
| | - Fabiano Botta
- Department of Experimental Psychology and Mind, Brain, and Behavior Research Center (CIMCYC), University of Granada, 18071 Granada, Spain; (F.B.); (J.L.)
| | - Riccardo Brunetti
- Cognitive and Clinical Psychology Laboratory, Department of Human Sciences, Università Europea di Roma, 00163 Roma, Italy;
| | - Juan Lupiáñez
- Department of Experimental Psychology and Mind, Brain, and Behavior Research Center (CIMCYC), University of Granada, 18071 Granada, Spain; (F.B.); (J.L.)
| | - Charles Spence
- Department of Experimental Psychology, Oxford University, Oxford OX2 6GG, UK;
| | - Valerio Santangelo
- Neuroimaging Laboratory, IRCCS Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, Italy;
- Department of Philosophy, Social Sciences & Education, University of Perugia, Piazza G. Ermini, 1, 06123 Perugia, Italy
- Correspondence:
| |
Collapse
|
11
|
A nonspatial sound modulates processing of visual distractors in a flanker task. Atten Percept Psychophys 2020; 83:800-809. [PMID: 33083988 PMCID: PMC8577790 DOI: 10.3758/s13414-020-02161-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/17/2020] [Indexed: 11/21/2022]
Abstract
Successful navigation of information-rich, multimodal environments involves processing of both auditory and visual information. The extent to which information within each modality is processed varies due to many factors, but the influence of auditory stimuli on the processing of visual stimuli in these multimodal environments is not well understood. Previous research has shown that a preceding sound leads to decreased reaction times in visual tasks (Bertelson, 1967). The current study examines if a non-spatial, task-irrelevant sound additionally alters processing of visual distractors that flank a central target. We utilized a version of a flanker task in which participants responded to a central letter surrounded by two irrelevant flanker letters. When these flankers are associated with a conflicting response, a congruency effect occurs such that reaction time to the target is slowed (Eriksen & Eriksen, 1974). In two experiments using this task, results showed that a preceding tone caused general speeding of reaction time across flanker types, consistent with alerting. The tone also caused decreased variation in response time. Critically, the tone modulated the congruency effect, with a greater speeding for congruent flankers than for incongruent flankers. This suggests that the influence of flanker identity was more intense after tone presentation, consistent with a nonspatial sound increasing perceptual and/or response-association processing of flanking stimuli.
Collapse
|
12
|
Pomper U, Schmid R, Ansorge U. Continuous, Lateralized Auditory Stimulation Biases Visual Spatial Processing. Front Psychol 2020; 11:1183. [PMID: 32655440 PMCID: PMC7325992 DOI: 10.3389/fpsyg.2020.01183] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2020] [Accepted: 05/07/2020] [Indexed: 11/25/2022] Open
Abstract
Sounds in our environment can easily capture human visual attention. Previous studies have investigated the impact of spatially localized, brief sounds on concurrent visuospatial attention. However, little is known on how the presence of a continuous, lateralized auditory stimulus (e.g., a person talking next to you while driving a car) impacts visual spatial attention (e.g., detection of critical events in traffic). In two experiments, we investigated whether a continuous auditory stream presented from one side biases visual spatial attention toward that side. Participants had to either passively or actively listen to sounds of various semantic complexities (tone pips, spoken digits, and a spoken story) while performing a visual target discrimination task. During both passive and active listening, we observed faster response times to visual targets presented spatially close to the relevant auditory stream. Additionally, we found that higher levels of semantic complexity of the presented sounds led to reduced visual discrimination sensitivity, but only during active listening to the sounds. We provide important novel results by showing that the presence of a continuous, ongoing auditory stimulus can impact visual processing, even when the sounds are not endogenously attended to. Together, our findings demonstrate the implications of ongoing sounds on visual processing in everyday scenarios such as moving about in traffic.
Collapse
Affiliation(s)
- Ulrich Pomper
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
| | - Rebecca Schmid
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
| | - Ulrich Ansorge
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria.,Cognitive Science Hub, University of Vienna, Vienna, Austria
| |
Collapse
|
13
|
Allan PG, Briggs RG, Conner AK, O'Neal CM, Bonney PA, Maxwell BD, Baker CM, Burks JD, Sali G, Glenn CA, Sughrue ME. Parcellation-based tractographic modeling of the ventral attention network. J Neurol Sci 2020; 408:116548. [DOI: 10.1016/j.jns.2019.116548] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2019] [Accepted: 10/20/2019] [Indexed: 11/25/2022]
|
14
|
Kvasova D, Garcia-Vernet L, Soto-Faraco S. Characteristic Sounds Facilitate Object Search in Real-Life Scenes. Front Psychol 2019; 10:2511. [PMID: 31749751 PMCID: PMC6848886 DOI: 10.3389/fpsyg.2019.02511] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2019] [Accepted: 10/23/2019] [Indexed: 12/02/2022] Open
Abstract
Real-world events do not only provide temporally and spatially correlated information across the senses, but also semantic correspondences about object identity. Prior research has shown that object sounds can enhance detection, identification, and search performance of semantically consistent visual targets. However, these effects are always demonstrated in simple and stereotyped displays that lack ecological validity. In order to address identity-based cross-modal relationships in real-world scenarios, we designed a visual search task using complex, dynamic scenes. Participants searched for objects in video clips recorded from real-life scenes. Auditory cues, embedded in the background sounds, could be target-consistent, distracter-consistent, neutral, or just absent. We found that, in these naturalistic scenes, characteristic sounds improve visual search for task-relevant objects but fail to increase the salience of irrelevant distracters. Our findings generalize previous results on object-based cross-modal interactions with simple stimuli and shed light upon how audio-visual semantically congruent relationships play out in real-life contexts.
Collapse
Affiliation(s)
- Daria Kvasova
- Center for Brain and Cognition, Universitat Pompeu Fabra, Barcelona, Spain
| | - Laia Garcia-Vernet
- Center for Brain and Cognition, Universitat Pompeu Fabra, Barcelona, Spain
| | - Salvador Soto-Faraco
- Center for Brain and Cognition, Universitat Pompeu Fabra, Barcelona, Spain
- ICREA – Catalan Institution for Research and Advanced Studies, Barcelona, Spain
| |
Collapse
|
15
|
Dugué L, Merriam EP, Heeger DJ, Carrasco M. Specific Visual Subregions of TPJ Mediate Reorienting of Spatial Attention. Cereb Cortex 2019; 28:2375-2390. [PMID: 28981585 DOI: 10.1093/cercor/bhx140] [Citation(s) in RCA: 49] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2016] [Accepted: 05/18/2017] [Indexed: 02/07/2023] Open
Abstract
The temporo-parietal junction (TPJ) has been associated with various cognitive and social functions, and is critical for attentional reorienting. Attention affects early visual processing. Neuroimaging studies dealing with such processes have thus far concentrated on striate and extrastriate areas. Here, we investigated whether attention orienting or reorienting modulate activity in visually driven TPJ subregions. For each observer we identified 3 visually responsive subregions within TPJ: 2 bilateral (vTPJant and vTPJpost) and 1 right lateralized (vTPJcent). Cortical activity in these subregions was measured using fMRI while observers performed a 2-alternative forced-choice orientation discrimination task. Covert spatial endogenous (voluntary) or exogenous (involuntary) attention was manipulated using either a central or a peripheral cue with task, stimuli and observers constant. Both endogenous and exogenous attention increased activity for invalidly cued trials in right vTPJpost; only endogenous attention increased activity for invalidly cued trials in left vTPJpost and in right vTPJcent; and neither type of attention modulated either right or left vTPJant. These results demonstrate that vTPJpost and vTPJcent mediate the reorientation of covert attention to task relevant stimuli, thus playing a critical role in visual attention. These findings reveal a differential reorienting cortical response after observers' attention has been oriented to a given location voluntarily or involuntarily.
Collapse
Affiliation(s)
- Laura Dugué
- Department of Psychology and Center for Neural Science, New York University, New York, NY, USA
| | - Elisha P Merriam
- Department of Psychology and Center for Neural Science, New York University, New York, NY, USA
| | - David J Heeger
- Department of Psychology and Center for Neural Science, New York University, New York, NY, USA
| | - Marisa Carrasco
- Department of Psychology and Center for Neural Science, New York University, New York, NY, USA
| |
Collapse
|
16
|
Spilcke-Liss J, Zhu J, Gluth S, Spezio M, Gläscher J. Semantic Incongruency Interferes With Endogenous Attention in Cross-Modal Integration of Semantically Congruent Objects. Front Integr Neurosci 2019; 13:53. [PMID: 31572138 PMCID: PMC6749080 DOI: 10.3389/fnint.2019.00053] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2019] [Accepted: 08/29/2019] [Indexed: 12/04/2022] Open
Abstract
Efficient multisensory integration is often influenced by other cognitive processes including, but not limited to, semantic congruency and focused endogenous attention. Semantic congruency can re-allocate processing resources to the location of a congruent stimulus, while attention can prioritize the integration of multi-sensory stimuli under focus. Here, we explore the robustness of this phenomenon in the context of three stimuli, two of which are in the focus of endogenous attention. Participants completed an endogenous attention task with a stimulus compound consisting of 3 different objects: (1) a visual object (V) in the foreground, (2) an auditory object (A), and (3) a visual background scene object (B). Three groups of participants focused their attention on either the visual object and auditory sound (Group VA, n = 30), the visual object and the background (VB, n = 27), or the auditory sound and the background (AB, n = 30), and judged the semantic congruency of the objects under focus. Congruency varied systematically across all 3 stimuli: All stimuli could be semantically incongruent (e.g., V, ambulance; A, church bell; and B, swimming-pool) or all could be congruent (e.g., V, lion; A, roar; and B, savannah), or two objects could be congruent with the remaining one incongruent to the other two (e.g., V, duck; A, quack; and B, phone booth). Participants exhibited a distinct pattern of errors: when participants attended two congruent objects (e.g., group VA: V, lion; A, roar), in the presence of an unattended, incongruent third object (e.g., B, bath room) they tended to make more errors than in any other stimulus combination. Drift diffusion modeling of the behavioral data revealed a significantly smaller drift rate in two-congruent-attended condition, indicating slower evidence accumulation, which was likely due to interference from the unattended, incongruent object. Interference with evidence accumulation occurred independently of which pair of objects was in the focus of attention, which suggests that the vulnerability of congruency judgments to incongruent unattended distractors is not affected by sensory modalities. A control analysis ruled out the simple explanation of a negative response bias. These findings implicate that our perceptual system is highly sensitive to semantic incongruencies even when they are not endogenously attended.
Collapse
Affiliation(s)
- Julia Spilcke-Liss
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Jun Zhu
- Department of Computer Science, Tsinghua University, Beijing, China
| | - Sebastian Gluth
- Department of Psychology, University of Basel, Basel, Switzerland
| | - Michael Spezio
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.,Psychology and Neuroscience, Scripps College, Claremont, CA, United States
| | - Jan Gläscher
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| |
Collapse
|
17
|
McCormick K, Lacey S, Stilla R, Nygaard LC, Sathian K. Neural basis of the crossmodal correspondence between auditory pitch and visuospatial elevation. Neuropsychologia 2018; 112:19-30. [PMID: 29501792 DOI: 10.1016/j.neuropsychologia.2018.02.029] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2017] [Revised: 02/22/2018] [Accepted: 02/26/2018] [Indexed: 01/09/2023]
Abstract
Crossmodal correspondences refer to associations between otherwise unrelated stimulus features in different sensory modalities. For example, high and low auditory pitches are associated with high and low visuospatial elevation, respectively. The neural mechanisms underlying crossmodal correspondences are currently unknown. Here, we used functional magnetic resonance imaging (fMRI) to investigate the neural basis of the pitch-elevation correspondence. Pitch-elevation congruency effects were observed bilaterally in the inferior frontal and insular cortex, the right frontal eye field and right inferior parietal cortex. Independent functional localizers failed to provide strong evidence for any of three proposed mechanisms for crossmodal correspondences: semantic mediation, magnitude estimation, and multisensory integration. Instead, pitch-elevation congruency effects overlapped with areas selective for visually presented non-word strings relative to sentences, and with regions sensitive to audiovisual asynchrony. Taken together with the prior literature, the observed congruency effects are most consistent with mediation by multisensory attention.
Collapse
Affiliation(s)
- Kelly McCormick
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA; Department of Psychology, Emory University, Atlanta, GA 30322, USA
| | - Simon Lacey
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA
| | - Randall Stilla
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA
| | - Lynne C Nygaard
- Department of Psychology, Emory University, Atlanta, GA 30322, USA
| | - K Sathian
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA; Department of Rehabilitation Medicine, Emory University, Atlanta, GA 30322, USA; Department of Psychology, Emory University, Atlanta, GA 30322, USA; Center for Visual and Neurocognitive Rehabilitation, Atlanta VAMC, Decatur, GA 30033, USA.
| |
Collapse
|
18
|
Zhou Y, Ho HN, Watanabe J. Perceptual-Semantic Congruency Facilitates Semantic Discrimination of Thermal Qualities. Front Psychol 2017; 8:2113. [PMID: 29270142 PMCID: PMC5724365 DOI: 10.3389/fpsyg.2017.02113] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2017] [Accepted: 11/20/2017] [Indexed: 01/23/2023] Open
Abstract
The ability to sense temperature is vital to our life. It signals the environmental condition, reflects the physiological conditions of our own body, and generates feelings of pleasantness or unpleasantness. Moreover, recent studies have demonstrated implicit associations between physical temperature and social/emotional concepts, suggesting the processing of temperature may even influence cognition. In this work, we examined the effect of physical warmth and coldness on semantic cognition. Participants performed speeded target categorization for thermal descriptors in the form of semantic words or illustrative figures representing the thermal qualities “warm” or “cold” while physical thermal stimulation was presented. We compared the average reaction time (RT) for the congruent and incongruent conditions managed by response key assignments. In the congruent condition, the response key for the symbol associated with warmth (coldness) was assigned to the hand with warm (cold) thermal stimulation, and in the incongruent condition the key assignment was reversed. Our results demonstrate that the average RT in the congruent condition was faster than in the incongruent one for both forms of thermal descriptors, suggesting that the experience of physical temperature facilitates the internal processing of the meaning of thermal quality.
Collapse
Affiliation(s)
- Yizhen Zhou
- Department of Information and Communications Engineering, Tokyo Institute of Technology, Tokyo, Japan
| | - Hsin-Ni Ho
- NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Kanagawa, Japan
| | - Junji Watanabe
- Department of Information and Communications Engineering, Tokyo Institute of Technology, Tokyo, Japan.,NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Kanagawa, Japan
| |
Collapse
|
19
|
Steady-state and dynamic network modes for perceptual expectation. Sci Rep 2017; 7:40626. [PMID: 28079163 PMCID: PMC5228187 DOI: 10.1038/srep40626] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2016] [Accepted: 12/08/2016] [Indexed: 12/05/2022] Open
Abstract
Perceptual expectation can attenuate repetition suppression, the stimulus-induced neuronal response generated by repeated stimulation, suggesting that repetition suppression is a top-down modulatory phenomenon. However, it is still unclear which high-level brain areas are involved and how they interact with low-level brain areas. Further, the temporal range over which perceptual expectation can effectively attenuate repetition suppression effects remains unclear. To elucidate the details of this top-down modulatory process, we used two short and long inter-stimulus intervals for a perceptual expectation paradigm of paired stimulation. We found that top-down modulation enhanced the response to the unexpected stimulus when repetition suppression was weak and that the effect disappeared at 1,000 ms prior to stimulus exposure. The high-level areas involved in this process included the left inferior frontal gyrus (IFG_L) and left parietal lobule (IPL_L). We also found two systems providing modulatory input to the right fusiform face area (FFA_R): one from IFG_L and the other from IPL_L. Most importantly, we identified two states of networks through which perceptual expectation modulates sensory responses: one is a dynamic state and the other is a steady state. Our results provide the first functional magnetic resonance imaging (fMRI) evidence of temporally nested networks in brain processing.
Collapse
|
20
|
ten Oever S, Romei V, van Atteveldt N, Soto-Faraco S, Murray MM, Matusz PJ. The COGs (context, object, and goals) in multisensory processing. Exp Brain Res 2016; 234:1307-23. [PMID: 26931340 DOI: 10.1007/s00221-016-4590-z] [Citation(s) in RCA: 32] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2015] [Accepted: 01/30/2016] [Indexed: 12/20/2022]
Abstract
Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and "top-down" control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer's goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications.
Collapse
Affiliation(s)
- Sanne ten Oever
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | - Vincenzo Romei
- Department of Psychology, Centre for Brain Science, University of Essex, Colchester, UK
| | - Nienke van Atteveldt
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands.,Department of Educational Neuroscience, Faculty of Psychology and Education and Institute LEARN!, VU University Amsterdam, Amsterdam, The Netherlands
| | - Salvador Soto-Faraco
- Multisensory Research Group, Center for Brain and Cognition, Universitat Pompeu Fabra, Barcelona, Spain.,Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain
| | - Micah M Murray
- The Laboratory for Investigative Neurophysiology (The LINE), Neuropsychology and Neurorehabilitation Service and Department of Radiology, Centre Hospitalier Universitaire Vaudois (CHUV), University Hospital Center and University of Lausanne, BH7.081, rue du Bugnon 46, 1011, Lausanne, Switzerland.,EEG Brain Mapping Core, Center for Biomedical Imaging (CIBM) of Lausanne and Geneva, Lausanne, Switzerland.,Department of Ophthalmology, Jules-Gonin Eye Hospital, University of Lausanne, Lausanne, Switzerland
| | - Pawel J Matusz
- The Laboratory for Investigative Neurophysiology (The LINE), Neuropsychology and Neurorehabilitation Service and Department of Radiology, Centre Hospitalier Universitaire Vaudois (CHUV), University Hospital Center and University of Lausanne, BH7.081, rue du Bugnon 46, 1011, Lausanne, Switzerland. .,Attention, Brain, and Cognitive Development Group, Department of Experimental Psychology, University of Oxford, Oxford, UK.
| |
Collapse
|