Object-based attention occurs regardless of object awareness

Objects guide human gaze behavior in dynamic real-world scenes

The complexity of natural scenes makes it challenging to experimentally study the mechanisms behind human gaze behavior when viewing dynamic environments. Historically, eye movements were believed to be driven primarily by space-based attention towards locations with salient features. Increasing evidence suggests, however, that visual attention does not select locations with high saliency but operates on attentional units given by the objects in the scene. We present a new computational framework to investigate the importance of objects for attentional guidance. This framework is designed to simulate realistic scanpaths for dynamic real-world scenes, including saccade timing and smooth pursuit behavior. Individual model components are based on psychophysically uncovered mechanisms of visual attention and saccadic decision-making. All mechanisms are implemented in a modular fashion with a small number of well-interpretable parameters. To systematically analyze the importance of objects in guiding gaze behavior, we implemented five different models within this framework: two purely spatial models, where one is based on low-level saliency and one on high-level saliency, two object-based models, with one incorporating low-level saliency for each object and the other one not using any saliency information, and a mixed model with object-based attention and selection but space-based inhibition of return. We optimized each model's parameters to reproduce the saccade amplitude and fixation duration distributions of human scanpaths using evolutionary algorithms. We compared model performance with respect to spatial and temporal fixation behavior, including the proportion of fixations exploring the background, as well as detecting, inspecting, and returning to objects. A model with object-based attention and inhibition, which uses saliency information to prioritize between objects for saccadic selection, leads to scanpath statistics with the highest similarity to the human data. This demonstrates that scanpath models benefit from object-based attention and selection, suggesting that object-level attentional units play an important role in guiding attentional processing.

The Architecture of Object-Based Attention

The allocation of attention to objects raises several intriguing questions: What are objects, how does attention access them, what anatomical regions are involved? Here, we review recent progress in the field to determine the mechanisms underlying object-based attention. First, findings from unconscious priming and cueing suggest that the preattentive targets of object-based attention can be fully developed object representations that have reached the level of identity. Next, the control of object-based attention appears to come from ventral visual areas specialized in object analysis that project downward to early visual areas. How feedback from object areas can accurately target the object's specific locations and features is unknown but recent work in autoencoding has made this plausible. Finally, we suggest that the three classic modes of attention may not be as independent as is commonly considered, and instead could all rely on object-based attention. Specifically, studies show that attention can be allocated to the separated members of a group-without affecting the space between them-matching the defining property of feature-based attention. At the same time, object-based attention directed to a single small item has the properties of space-based attention. We outline the architecture of object-based attention, the novel predictions it brings, and discuss how it works in parallel with other attention pathways.

Awareness-independent gradual spread of object-based attention

Although attention can be directed at certain objects, how object-based attention spreads within an object and whether this spread interacts with awareness remain unclear. Using a modified spatial cuing paradigm with backward masking, we addressed these issues with either visible or invisible displays presenting the real (Experiment 1) and illusory (Experiment 2) U-shaped objects (UOs), whose ends and middles, the possible locations of the cue and target, have iso-eccentric distances from the fixation. These equidistant ends and middles of UOs offered us a unique opportunity to examine whether attention gradually spreads within a given object, i.e., within an UO, attention spreads from its cued-end to uncued-end via the uncued-middle. Despite the visibility (visible or invisible) of UOs, both experiments supported this gradual spread manner by showing a faster response of human participants (male and female) to the target in the uncued-middle than that in the uncued-end. Our results thus indicate a gradual spread of object-based attention and further reveal that this gradual spread is independent of both the “visual objectness” (whether the object is defined as the real or illusory boundaries) and conscious access to objects.

Can faces affect object-based attention? Evidence from online experiments

This study tested how human faces affect object-based attention (OBA) through two online experiments in a modified double-rectangle paradigm. The results of Experiment 1 revealed that faces did not elicit the OBA effect as non-face objects, which was caused by a longer response time (RT) when attention is focused on faces relative to non-face objects. In addition, by observing faster RTs when attention was engaged horizontally rather than vertically, we found a significant horizontal attention bias, which might override the OBA effect if vertical rectangles were the only items presented; these results were replicated in Experiment 2 (using only vertical rectangles) after directly measuring horizontal bias and excluding its influence on the OBA effect. This study suggested that faces cannot elicit the same-object advantage in the double-rectangle paradigm and provided a method to measure the OBA effect free from horizontal bias.

Attention can operate on object representations in visual sensory memory

Numerous studies have shown that attention can be allocated to various types of objects, such as low-level objects developed by perceptual organization and high-level objects developed by semantic associations. However, little is known about whether attention can also be affected solely by object representations in the brain, after the disappearance of physical objects. Here, we used a modified double-rectangle paradigm to investigate how attention is affected by object representation in visual sensory memory when the physical objects disappear for a short period of time before the target onset. By manipulating the interstimulus interval (ISI) between the offset of the objects and the onset of the target, an object-based attention effect, with shorter reaction times (RTs) for within-object relative to between-object conditions, was observed in the short-ISI (within 500 ms in Experiments 1a, 1b, 2, and 3) conditions while disappearing in the long-ISI (800 ms in Experiment 4) conditions. This result demonstrated that the mere presence of object representation in visual sensory memory, or the sensory memory-maintained object, can serve as an object unit that attention can operate on. This provides evidence for the relationship between object-based attention and visual sensory memory: object representation in visual sensory memory could affect attentional allocation, or attention can operate on a sensory memory-maintained object.

Perceptual rivalry across animal species

This review in memoriam of Jack Pettigrew provides an overview of past and current research into the phenomenon of multistable perception across multiple animal species. Multistable perception is characterized by two or more perceptual interpretations spontaneously alternating, or rivaling, when animals are exposed to stimuli with inherent sensory ambiguity. There is a wide array of ambiguous stimuli across sensory modalities, ranging from the configural changes observed in simple line drawings, such as the famous Necker cube, to the alternating perception of entire visual scenes that can be instigated by interocular conflict. The latter phenomenon, called binocular rivalry, in particular caught the attention of the late Jack Pettigrew, who combined his interest in the neuronal basis of perception with a unique comparative biological approach that considered ambiguous sensation as a fundamental problem of sensory systems that has shaped the brain throughout evolution. Here, we examine the research findings on visual perceptual alternation and suppression in a wide variety of species including insects, fish, reptiles, and primates. We highlight several interesting commonalities across species and behavioral indicators of perceptual alternation. In addition, we show how the comparative approach provides new avenues for understanding how the brain suppresses opposing sensory signals and generates alternations in perceptual dominance.

Dynamic face mask enhances continuous flash suppression

In continuous flash suppression (CFS), an image presented to one eye is suppressed from awareness by a dynamic image masker presented to the other eye. Previous studies report that face stimuli break out of CFS more readily when they are oriented upright and contain ecologically relevant information such as facial expressions or direct eye gaze, potentially implicating face processing in the mechanisms of interocular competition. It is unknown, however, whether face content helps to drive interocular suppression when incorporated into the dynamic masker itself, either by engaging higher-level visual mechanisms that underlie face detection or due to lower-level image features that the faces happen to contain. To investigate this, we devised a dynamic mask composed of upright faces and tested how well it suppressed detection of face or grating targets presented to the other eye. Relative contributions of higher-level and lower-level features were compared by manipulating the image properties of the mask. Results show that the dynamic face mask is strikingly effective at suppressing sensory input presented to the opposing eye, but its effectiveness is largely attributable to image texture, which can be quantified in terms of image entropy and edge density. This is because strong suppression was still observed following phase-scrambling or spatial inversion of the face elements, and while a target-selective effect was observed for the face mask, inverting the face elements to interfere with configural processing did not significantly diminish this effect. Thus, visual properties of faces, such as their image entropy and complex phase structure, predominate in driving interocular suppression rather than face detection per se.

Seeing objects improves our hearing of the sounds they make

It has been established that lip reading improves the perception of auditory speech. But does seeing objects themselves help us hear better the sounds they make? Here we report a series of psychophysical experiments in humans showing that the visual enhancement of auditory sensitivity is not confined to speech. We further show that the crossmodal enhancement was associated with the conscious visualization of the stimulus: we can better hear the sounds an object makes when we are conscious of seeing that object. Our work extends an intriguing crossmodal effect, previously circumscribed to speech, to a wider domain of real-world objects, and suggests that consciousness contributes to this effect.

The Relationship between Biological Motion-Based Visual Consciousness and Attention: An Electroencephalograph Study

Understanding and predicting the intentions of others through limb movements are vital to social interaction. The processing of biological motion is unique from the processing of motion of inanimate objects. Presently, there is controversy over whether visual consciousness of biological motion is regulated by visual attention. In addition, the neural mechanisms involved in biological motion-related visual awareness are not known. In the current study, the relationship between visual awareness (aware vs unaware), represented by a point-light walker and biological-motion-based attention, manipulated by a difference in congruence (congruent, incongruent) between the direction of a pre-cue and that of biological motion was explored. The neural mechanisms involved in processing the stimuli were explored through electroencephalography. Both early (50-150 ms, 100-200 ms, and 174-226 ms after target presentation) and late (350-550 ms after target presentation) awareness-related neural processings were observed during a biological motion-based congruency task. Early processing was localized to occipital-parietal regions, such as the left postcentral gyrus, the left middle occipital gyrus, and the right precentral gyrus. In the 174-226-ms window, the activity in the occipital region was gradually replaced by activity in the parietal and frontal regions. Late processing was localized to frontal-parietal regions, such as the right dorsal superior frontal gyrus, the left medial superior frontal gyrus, and the occipito-temporal regions. Congruency-related processing occurred in the 246-260-ms window and was localized to the right superior occipital gyrus. In summary, due to its complexity, biological motion awareness has a unique neural basis.

Recent evidence that attention is necessary, but not sufficient, for conscious perception

Early descriptions of attention in the psychological literature highlighted its interdependence with conscious awareness. As the study of attention developed, consciousness and attention began to be considered separable phenomena, experimentally and theoretically. In recent years, an energetic debate has developed concerning the extent to which the two phenomena are related. One school of thought considers the two to be doubly dissociable, whereas the other considers them to be necessarily linked. In this review, we highlight experimental findings from the last 5 years that contribute to the leading consensus view: attention is necessary, but not sufficient, for conscious perception. We review studies that show attention operating in conjunction with unconscious information, and other evidence linking attention necessarily to conscious perception. By drawing upon evidence that attention comprises many cognitive and neural processes, we argue that by studying how different forms of attention are related to conscious perception, it is possible to gain new insights about the neural states or processes that are necessary for conscious perception to occur.

Attentional selection of multiple objects in the human visual system

Classic theories of object-based attention assume a single object of selection but real-world tasks, such as driving a car, often require attending to multiple objects simultaneously. However, whether object-based attention can operate on more than one object at a time remains unexplored. Here, we used functional magnetic resonance imaging (fMRI) to address this question as human participants performed object-based attention tasks that required simultaneous attention to two objects differing in either their features or locations. Simultaneous attention to two objects differing in features (face and house) did not show significantly different responses in the fusiform face area (FFA) or parahippocampal place area (PPA), respectively, compared to attending a single object (face or house), but did enhance the response in the inferior frontal gyrus (IFG). Simultaneous attention to two circular arcs differing in locations did not show significantly different responses in the primary visual cortex (V1) compared to attending a single circular arc, but did enhance the response in the intraparietal sulcus (IPS). These results suggest that object-based attention can simultaneously select at least two objects differing in their features or locations, processes mediated by the frontal and parietal cortex, respectively.

The availability of attentional resources modulates the inhibitory strength related to weakly activated priming

The current study investigated the role of attention in inhibitory processes (the inhibitory processes described in the current study refer only to those associated with masked or flanked priming) using a mixed paradigm involving the negative compatibility effect (NCE) and object-based attention. Accumulating evidence suggests that attention can be spread more easily within the same object, which increases the availability of attentional resources, than across different objects. Accordingly, we manipulated distractor location (with primes presented in the same object versus presented in different objects) together with prime/target compatibility (compatible versus incompatible) and prime-distractor stimulus onset asynchrony (SOA, 23 ms vs 70 ms). The aim was to investigate whether inhibitory processes related to weakly activated priming, which have been previously assumed to be automatic, depend on the availability of attentional resources. The results of Experiment 1 showed a significant NCE for the 70-ms SOA when the prime and distractor were presented in the same object (greater attentional resource availability); however, reversed NCEs were obtained for all other conditions. Experiment 2 was designed to disentangle whether the results of Experiment 1 were affected by the prime position, and the results indicated that the prime position did not modulate the NCE in Experiment 1. Together, these results are consistent with the claim that the availability of attentional resources modulates the inhibitory strength related to weakly activated priming. Specifically, if attentional resources are assigned to the distractor when it is presented in the same object as the prime, the strength of the inhibition elicited by the distractor may increase and reverse the activation elicited by the prime, which could lead to a significant NCE.

Interocular grouping without awareness

Interocular grouping occurs when different parts of an image presented to each eye bound into a coherent whole. Previous studies anticipated that these parts are visible to both eyes simultaneously (i.e., the images altered back and forth). Although this view is consistent with the general consensus of binocular rivalry (BR) that suppressed stimuli receive no processing beyond rudimentary level (i.e., adaptation), it is actually inconsistent with studies that use continuous flash suppression (CFS). CFS is a form of interocular suppression that is more stable and causes stronger suppression of stimuli than BR. In the present study, we examined whether or not interocular grouping needs to occur at a conscious level as prior studies suggested. The modified double-rectangle paradigm used by Egly, Driver, and Rafal (1994) was adopted, and object-based attention was directed for successful grouping. To induce interocular grouping, we presented complementary parts of two rectangles dichoptically for possible interocular grouping and a dynamic Mondrian in front of one eye (i.e., CFS). Two concurrent targets were presented after one of the visible parts of the rectangles was cued. Participants were asked to judge which target appeared first. We found that the target showed on the cued rectangle after interocular grouping was reported to appear first more frequently than the target on the uncued rectangle. This result was based on the majority of trials where the suppressed parts of the objects remained invisible, which indicates that interocular grouping can occur without all the to-be-grouped parts being visible and without awareness.

The effects of visual search efficiency on object-based attention

The attentional prioritization hypothesis of object-based attention (Shomstein & Yantis in Perception & Psychophysics, 64, 41-51, 2002) suggests a two-stage selection process comprising an automatic spatial gradient and flexible strategic (prioritization) selection. The combined attentional priorities of these two stages of object-based selection determine the order in which participants will search the display for the presence of a target. The strategic process has often been likened to a prioritized visual search. By modifying the double-rectangle cueing paradigm (Egly, Driver, & Rafal in Journal of Experimental Psychology: General, 123, 161-177, 1994) and placing it in the context of a larger-scale visual search, we examined how the prioritization search is affected by search efficiency. By probing both targets located on the cued object and targets external to the cued object, we found that the attentional priority surrounding a selected object is strongly modulated by search mode. However, the ordering of the prioritization search is unaffected by search mode. The data also provide evidence that standard spatial visual search and object-based prioritization search may rely on distinct mechanisms. These results provide insight into the interactions between the mode of visual search and object-based selection, and help define the modulatory consequences of search efficiency for object-based attention.

To see or not to see--thalamo-cortical networks during blindsight and perceptual suppression

Even during moments when we fail to be fully aware of our environment, our brains never go silent. Instead, it appears that the brain can also operate in an alternate, unconscious mode. Delineating unconscious from conscious neural processes is a promising first step toward investigating how awareness emerges from brain activity. Here we focus on recent insights into the neuronal processes that contribute to visual function in the absence of a conscious visual percept. Drawing on insights from findings on the phenomenon of blindsight that results from injury to primary visual cortex and the results of experimentally induced perceptual suppression, we describe what kind of visual information the visual system analyzes unconsciously and we discuss the neuronal routing and responses that accompany this process. We conclude that unconscious processing of certain visual stimulus attributes, such as the presence of visual motion or the emotional expression of a face can occur in a geniculo-cortical circuit that runs independent from and in parallel to the predominant route through primary visual cortex. We speculate that in contrast, bidirectional neuronal interactions between cortex and the thalamic pulvinar nucleus that support large-scale neuronal integration and visual awareness are impeded during blindsight and perceptual suppression.

Invisible collinear structures impair search

Visual attention and perceptual grouping both help us from being overloaded by the vast amount of information, and attentional search is delayed when a target overlaps with a snake-like collinear distractor (Jingling & Tseng, 2013). We assessed whether awareness of the collinear distractor is required for this modulation. We first identified that visible long (=9 elements), but not short (=3 elements) collinear distractor slowed observers' detection of an overlapping target. Then we masked part of a long distractor (=9 elements) with continuous flashing color patches (=6 elements) so that the combined dichoptic percept to observers' awareness was a short collinear distractor (=3 elements). We found that the invisible collinear parts, like visible ones, can form a continuous contour to impair search, suggesting that conscious awareness is not a pre-requisite for contour integration and its interaction with selective attention.

Auditory and tactile signals combine to influence vision during binocular rivalry

Resolution of perceptual ambiguity is one function of cross-modal interactions. Here we investigate whether auditory and tactile stimuli can influence binocular rivalry generated by interocular temporal conflict in human subjects. Using dichoptic visual stimuli modulating at different temporal frequencies, we added modulating sounds or vibrations congruent with one or the other visual temporal frequency. Auditory and tactile stimulation both interacted with binocular rivalry by promoting dominance of the congruent visual stimulus. This effect depended on the cross-modal modulation strength and was absent when modulation depth declined to 33%. However, when auditory and tactile stimuli that were too weak on their own to bias binocular rivalry were combined, their influence over vision was very strong, suggesting the auditory and tactile temporal signals combined to influence vision. Similarly, interleaving discrete pulses of auditory and tactile stimuli also promoted dominance of the visual stimulus congruent with the supramodal frequency. When auditory and tactile stimuli were presented at maximum strength, but in antiphase, they had no influence over vision for low temporal frequencies, a null effect again suggesting audio-tactile combination. We also found that the cross-modal interaction was frequency-sensitive at low temporal frequencies, when information about temporal phase alignment can be perceptually tracked. These results show that auditory and tactile temporal processing is functionally linked, suggesting a common neural substrate for the two sensory modalities and that at low temporal frequencies visual activity can be synchronized by a congruent cross-modal signal in a frequency-selective way, suggesting the existence of a supramodal temporal binding mechanism.

Unmasking the dichoptic mask by sound: spatial congruency matters

People tend to look toward where a sound occurs; however, the role of spatial congruency between sound and sight in the effect of sound facilitation on visual detection remains controversial. We propose that the role of spatial congruency depends on the reliability of the information provided by the facilitator; if it is relatively unreliable, adding spatially congruent information can help to unify different sensory inputs to compensate for this unreliability. To test this, we examine the influence of sound location on visual detection with a non-temporal task, presumably unfavorable for sound since it is better for temporal resolution, and predict that spatial congruency should matter in this situation. We used the continuous flash suppression paradigm that makes the visual stimuli invisible to keep the relationship of sound and sight opaque. The sound is on the same depth plane as the visual stimulus (the congruent condition) or on a different plane (the incongruent condition). The target was presented to one eye with luminance contrast gradually increased and continuously masked by flashed Mondrian masks presented to the other eye until the target was released from suppression. We found that sound facilitated visual detection (measured by released-from-suppression time) in the spatially congruent condition but not in the spatially incongruent condition. Together with previous findings in the literature, it is suggested that both task type and modality determine the reliability of the information for multisensory integration and thus determine whether spatial congruency is critical.

Language can boost otherwise unseen objects into visual awareness

Linguistic labels (e.g., "chair") seem to activate visual properties of the objects to which they refer. Here we investigated whether language-based activation of visual representations can affect the ability to simply detect the presence of an object. We used continuous flash suppression to suppress visual awareness of familiar objects while they were continuously presented to one eye. Participants made simple detection decisions, indicating whether they saw any image. Hearing a verbal label before the simple detection task changed performance relative to an uninformative cue baseline. Valid labels improved performance relative to no-label baseline trials. Invalid labels decreased performance. Labels affected both sensitivity (d') and response times. In addition, we found that the effectiveness of labels varied predictably as a function of the match between the shape of the stimulus and the shape denoted by the label. Together, the findings suggest that facilitated detection of invisible objects due to language occurs at a perceptual rather than semantic locus. We hypothesize that when information associated with verbal labels matches stimulus-driven activity, language can provide a boost to perception, propelling an otherwise invisible image into awareness.

Object-based attention guided by an invisible object

Evidence for object-based attention typically comes from studies using displays with visible objects, and little is known about whether object-based attention can occur with invisible objects. We investigated this issue with a modified double-rectangle cuing paradigm, which was originally developed by Egly et al. (J Exp Psychol Gen 123:161-177, 1994). In this study, low-contrast rectangles were presented very briefly, which rendered them invisible to subjects. With the invisible rectangles, we found a classical object-based attentional effect as indexed by the same-object effect. We also found the instantaneous object effect-object-based attention was dependent on the orientation of the rectangles presented with the target, providing evidence for the dynamic updating hypothesis (Ho and Yeh in Acta Psychol 132:31-39, 2009). These results suggest that object-based attention can be guided by an invisible object in an automatic way, with a minimal influence from high-level top-down control.