Predicting attentional allocation in real-world environments: The need to investigate crossmodal semantic guidance

Real-world environments are multisensory, meaningful, and highly complex. To parse these environments in a highly efficient manner, a subset of this information must be selected both within and across modalities. However, the bulk of attention research has been conducted within sensory modalities, with a particular focus on vision. Visual attention research has made great strides, with over a century of research methodically identifying the underlying mechanisms that allow us to select critical visual information. Spatial attention, attention to features, and object-based attention have all been studied extensively. More recently, research has established semantics (meaning) as a key component to allocating attention in real-world scenes, with the meaning of an item or environment affecting visual attentional selection. However, a full understanding of how semantic information modulates real-world attention requires studying more than vision in isolation. The world provides semantic information across all senses, but with this extra information comes greater complexity. Here, we summarize visual attention (including semantic-based visual attention), crossmodal attention, and argue for the importance of studying crossmodal semantic guidance of attention. This article is categorized under: Psychology > Attention Psychology > Perception and Psychophysics.

Object-based inhibition of return in three-dimensional space: From simple drawings to real objects

Cued to an object in space, inhibition of the attended location can spread to the entire object. Although object-based inhibition of return (IOR) studies in a two-dimensional plane have been documented, the IOR has not been explored when objects cross depth in three-dimensional (3D) space. In the present study, we used a virtual reality technique to adapt the double-rectangle paradigm to a 3D space, and manipulated the cue validity and target location to examine the difference in object-based IOR between far and near spaces under different object representations. The study showed that the object-based IOR of simple drawings existed only in near space, whereas object-based IOR of real objects existed only in far space at first, and as the object similarity decreases, it appeared in both far and near spaces.

The effect of context congruency on fMRI repetition suppression for objects

The recognition of objects is strongly facilitated when they are presented in the context of other objects (Biederman, 1972). Such contexts facilitate perception and induce expectations of context-congruent objects (Trapp and Bar, 2015). The neural mechanisms underlying these facilitatory effects of context on object processing, however, are not yet fully understood. In the present study, we investigate how context-induced expectations affect subsequent object processing. We used functional magnetic resonance imaging and measured repetition suppression as a proxy for prediction error processing. Participants viewed pairs of alternating or repeated object images which were preceded by context-congruent, context-incongruent or neutral cues. We found a stronger repetition suppression in congruent as compared to incongruent or neutral cues in the object sensitive lateral occipital cortex. Interestingly, this stronger effect was driven by enhanced responses to alternating stimulus pairs in the congruent contexts, rather than by suppressed responses to repeated stimulus pairs, which emphasizes the contribution of surprise-related response enhancement for the context modulation on RS when expectations are violated. In addition, in the congruent condition, we discovered significant functional connectivity between object-responsive and frontal cortical regions, as well as between object-responsive regions and the fusiform gyrus. Our findings indicate that prediction errors, reflected in enhanced brain responses to violated contextual expectations, underlie the facilitating effect of context during object perception.

Representations of Temporal Community Structure in Hippocampus and Precuneus Predict Inductive Reasoning Decisions

Our understanding of the world is shaped by inferences about underlying structure. For example, at the gym, you might notice that the same people tend to arrive around the same time and infer that they are friends that work out together. Consistent with this idea, after participants are presented with a temporal sequence of objects that follows an underlying community structure, they are biased to infer that objects from the same community share the same properties. Here, we used fMRI to measure neural representations of objects after temporal community structure learning and examine how these representations support inference about object relationships. We found that community structure learning affected inferred object similarity: When asked to spatially group items based on their experience, participants tended to group together objects from the same community. Neural representations in perirhinal cortex predicted individual differences in object grouping, suggesting that high-level object representations are affected by temporal community learning. Furthermore, participants were biased to infer that objects from the same community would share the same properties. Using computational modeling of temporal learning and inference decisions, we found that inductive reasoning is influenced by both detailed knowledge of temporal statistics and abstract knowledge of the temporal communities. The fidelity of temporal community representations in hippocampus and precuneus predicted the degree to which temporal community membership biased reasoning decisions. Our results suggest that temporal knowledge is represented at multiple levels of abstraction, and that perirhinal cortex, hippocampus, and precuneus may support inference based on this knowledge.

Task-Irrelevant Semantic Properties of Objects Impinge on Sensory Representations within the Early Visual Cortex

Objects can be described in terms of low-level (e.g., boundaries) and high-level properties (e.g., object semantics). While recent behavioral findings suggest that the influence of semantic relatedness between objects on attentional allocation can be independent of task-relevance, the underlying neural substrate of semantic influences on attention remains ill-defined. Here, we employ behavioral and functional magnetic resonance imaging measures to uncover the mechanism by which semantic information increases visual processing efficiency. We demonstrate that the strength of the semantic relatedness signal decoded from the left inferior frontal gyrus: 1) influences attention, producing behavioral semantic benefits; 2) biases spatial attention maps in the intraparietal sulcus, subsequently modulating early visual cortex activity; and 3) directly predicts the magnitude of behavioral semantic benefit. Altogether, these results identify a specific mechanism driving task-independent semantic influences on attention.

Mind Meets Machine: Towards a Cognitive Science of Human-Machine Interactions

As robots advance from the pages and screens of science fiction into our homes, hospitals, and schools, they are poised to take on increasingly social roles. Consequently, the need to understand the mechanisms supporting human-machine interactions is becoming increasingly pressing. We introduce a framework for studying the cognitive and brain mechanisms that support human-machine interactions, leveraging advances made in cognitive neuroscience to link different levels of description with relevant theory and methods. We highlight unique features that make this endeavour particularly challenging (and rewarding) for brain and behavioural scientists. Overall, the framework offers a way to study the cognitive science of human-machine interactions that respects the diversity of social machines, individuals' expectations and experiences, and the structure and function of multiple cognitive and brain systems.

Real-world expectations and their affective value modulate object processing

It is well known that expectations influence how we perceive the world. Yet the neural mechanisms underlying this process remain unclear. Studies about the effects of prior expectations have focused so far on artificial contingencies between simple neutral cues and events. Real-world expectations are however often generated from complex associations between contexts and objects learned over a lifetime. Additionally, these expectations may contain some affective value and recent proposals present conflicting hypotheses about the mechanisms underlying affect in predictions. In this study, we used fMRI to investigate how object processing is influenced by realistic context-based expectations, and how affect impacts these expectations. First, we show that the precuneus, the inferotemporal cortex and the frontal cortex are more active during object recognition when expectations have been elicited a priori, irrespectively of their validity or their affective intensity. This result supports previous hypotheses according to which these brain areas integrate contextual expectations with object sensory information. Notably, these brain areas are different from those responsible for simultaneous context-object interactions, dissociating the two processes. Then, we show that early visual areas, on the contrary, are more active during object recognition when no prior expectation has been elicited by a context. Lastly, BOLD activity was shown to be enhanced in early visual areas when objects are less expected, but only when contexts are neutral; the reverse effect is observed when contexts are affective. This result supports the proposal that affect modulates the weighting of sensory information during predictions. Together, our results help elucidate the neural mechanisms of real-world expectations.

Event Representations and Predictive Processing: The Role of the Midline Default Network Core

The human brain is tightly coupled to the world through its sensory‐motor systems—but it also spends a lot of its metabolism talking to itself. One important function of this intrinsic activity is the establishment and updating of event models—representations of the current situation that can predictively guide perception, learning, and action control. Here, we propose that event models largely depend on the default network (DN) midline core that includes the posterior cingulate and anterior medial prefrontal cortex. An increasing body of data indeed suggests that this subnetwork can facilitate stimuli processing during both naturalistic event comprehension and cognitive tasks in which mental representations of prior situations, trials, and task rules can predictively guide attention and performance. This midline core involvement in supporting predictions through event models can make sense of an otherwise complex and conflicting pattern of results regarding the possible cognitive functions subserved by the DN.

Stawarczyk, Bezdek, and Zacks offer neuroscience evidence for a midline default network core, which appears to coordinate internal, top‐down mentation with externally‐triggered, bottom‐up attention in a push‐pull relationship. The network may enable the flexible pursuance of thoughts tuned into or detached from the current environment.

What does the functional organization of cortico-hippocampal networks tell us about the functional organization of memory?

Historically, research on the cognitive processes that support human memory proceeded, to a large extent, independently of research on the neural basis of memory. Accumulating evidence from neuroimaging, however, has enabled the field to develop a broader and more integrative perspective. Here, we briefly outline how advances in cognitive neuroscience can potentially shed light on concepts and controversies in human memory research. We argue that research on the functional properties of cortico-hippocampal networks informs us about how memories might be organized in the brain, which, in turn, helps to reconcile seemingly disparate perspectives in cognitive psychology. Finally, we discuss several open questions and directions for future research.

5-HT2a receptor in mPFC influences context-guided reconsolidation of object memory in perirhinal cortex

Context-dependent memories may guide adaptive behavior relaying in previous experience while updating stored information through reconsolidation. Retrieval can be triggered by partial and shared cues. When the cue is presented, the most relevant memory should be updated. In a contextual version of the object recognition task, we examined the effect of medial PFC (mPFC) serotonin 2a receptor (5-HT2aR) blockade during retrieval in reconsolidation of competing objects memories. We found that mPFC 5-HT2aR controls retrieval and reconsolidation of object memories in the perirhinal cortex (PRH), but not in the dorsal hippocampus in rats. Also, reconsolidation of objects memories in PRH required a functional interaction between the ventral hippocampus and the mPFC. Our results indicate that in the presence of conflicting information at retrieval, mPFC 5-HT2aR may facilitate top-down context-guided control over PRH to control the behavioral response and object memory reconsolidation.

Intrusive effects of semantic information on visual selective attention

Every object is represented by semantic information in extension to its low-level properties. It is well documented that such information biases attention when it is necessary for an ongoing task. However, whether semantic relationships influence attentional selection when they are irrelevant to the ongoing task remains an open question. The ubiquitous nature of semantic information suggests that it could bias attention even when these properties are irrelevant. In the present study, three objects appeared on screen, two of which were semantically related. After a varying time interval, a target or distractor appeared on top of each object. The objects’ semantic relationships never predicted the target location. Despite this, a semantic bias on attentional allocation was observed, with an initial, transient bias to semantically related objects. Further experiments demonstrated that this effect was contingent on the objects being attended: if an object never contained the target, it no longer exerted a semantic influence. In a final set of experiments, we demonstrated that the semantic bias is robust and appears even in the presence of more predictive cues (spatial probability). These results suggest that as long as an object is attended, its semantic properties bias attention, even if it is irrelevant to an ongoing task and if more predictive factors are available.

Two Distinct Scene-Processing Networks Connecting Vision and Memory

A number of regions in the human brain are known to be involved in processing natural scenes, but the field has lacked a unifying framework for understanding how these different regions are organized and interact. We provide evidence from functional connectivity and meta-analyses for a new organizational principle, in which scene processing relies upon two distinct networks that split the classically defined parahippocampal place area (PPA). The first network of strongly connected regions consists of the occipital place area/transverse occipital sulcus and posterior PPA, which contain retinotopic maps and are not strongly coupled to the hippocampus at rest. The second network consists of the caudal inferior parietal lobule, retrosplenial complex, and anterior PPA, which connect to the hippocampus (especially anterior hippocampus), and are implicated in both visual and nonvisual tasks, including episodic memory and navigation. We propose that these two distinct networks capture the primary functional division among scene-processing regions, between those that process visual features from the current view of a scene and those that connect information from a current scene view with a much broader temporal and spatial context. This new framework for understanding the neural substrates of scene-processing bridges results from many lines of research, and makes specific functional predictions.