Searching for Category-Consistent Features: A Computational Approach to Understanding Visual Category Representation

Distractor similarity and category variability effects in search

Categorical search involves looking for objects based on category information from long-term memory. Previous research has shown that search efficiency in categorical search is influenced by target/distractor similarity and category variability (i.e., heterogeneity). However, the interaction between these factors and their impact on different subprocesses of search remains unclear. This study examined the effects of target/distractor similarity and category variability on processes of categorical search. Using multidimensional scaling, we manipulated target/distractor similarity and measured category variability for target categories that participants searched for. Eye-tracking data were collected to examine attentional guidance and target verification. The results demonstrated that the effect of category variability on response times (RTs) was dependent on the level of target/distractor similarity. Specifically, when distractors were highly similar to target categories, there was a negative relation between RTs and variability, with low variability categories producing longer RTs than higher variability categories. Surprisingly, this trend was only present in the eye-tracking measures of target verification but not attentional guidance. Our results suggest that searchers more effectively guide attention to low-variability categories compared to high-variability categories, regardless of the degree of similarity between targets and distractors. However, low category variability interferes with target match decisions when distractors are highly similar to the category, thus the advantage that low category variability provides to searchers is not equal across processes of search.

The attentive reconstruction of objects facilitates robust object recognition

Humans are extremely robust in our ability to perceive and recognize objects-we see faces in tea stains and can recognize friends on dark streets. Yet, neurocomputational models of primate object recognition have focused on the initial feed-forward pass of processing through the ventral stream and less on the top-down feedback that likely underlies robust object perception and recognition. Aligned with the generative approach, we propose that the visual system actively facilitates recognition by reconstructing the object hypothesized to be in the image. Top-down attention then uses this reconstruction as a template to bias feedforward processing to align with the most plausible object hypothesis. Building on auto-encoder neural networks, our model makes detailed hypotheses about the appearance and location of the candidate objects in the image by reconstructing a complete object representation from potentially incomplete visual input due to noise and occlusion. The model then leverages the best object reconstruction, measured by reconstruction error, to direct the bottom-up process of selectively routing low-level features, a top-down biasing that captures a core function of attention. We evaluated our model using the MNIST-C (handwritten digits under corruptions) and ImageNet-C (real-world objects under corruptions) datasets. Not only did our model achieve superior performance on these challenging tasks designed to approximate real-world noise and occlusion viewing conditions, but also better accounted for human behavioral reaction times and error patterns than a standard feedforward Convolutional Neural Network. Our model suggests that a complete understanding of object perception and recognition requires integrating top-down and attention feedback, which we propose is an object reconstruction.

The time course of category-based attentional template pre-activation depends on the category framework

When searching for a target defined by a set of objects, attention can be directed toward task-relevant objects by creating a category-based attentional template (CAT). Previous studies have found that CAT can be activated before the onset of the target. However, the time course of CAT pre-activation and whether the category framework (prototypical or semantic) can modulate it remain unclear. To explore the time course of CAT pre-activation, we employed a rapid serial probe presentation paradigm (RSPP) with event-related potentials (ERPs). To investigate the effect of the category framework on the time course of CAT pre-activation, the target category was defined as the prototypical category (Experiment 1) or the semantic category (Experiment 2). The results showed that the prototype-based CAT was pre-activated 300 ms prior to the target, whereas the semantics-based CAT was pre-activated 1500 ms before the onset of the target. The difference in the time course of pre-activation between the two CAT types indicates that the category framework can modulate the time course of CAT pre-activation. Additionally, during the attentional selection phase, an overall comparison of the target revealed that a larger N2pc was elicited by the prototype-based CAT than by the semantics-based CAT, suggesting that the prototype-based CAT could capture more attention than the semantics-based CAT. The findings on the difference between the two CAT frameworks in the preparatory and attentional selection phases provide more evidence for categorical information in visual search and extend our understanding of the mechanism of categorical attentional selection.

Explicit and implicit category learning in categorical visual search

Categorical search has been heavily investigated over the past decade, mostly using natural categories that leave the underlying category mental representation unknown. The categorization literature offers several theoretical accounts of category mental representations. One prominent account is that separate learning systems account for classification: an explicit learning system that relies on easily verbalized rules and an implicit learning system that relies on an associatively learned (nonverbalizable) information integration strategy. The current study assessed the contributions of these separate category learning systems in the context of categorical search using simple stimuli. Participants learned to classify sinusoidal grating stimuli according to explicit or implicit categorization strategies, followed by a categorical search task using these same stimulus categories. Computational modeling determined which participants used the appropriate classification strategy during training and search, and eye movements collected during categorical search were assessed. We found that the trained categorization strategies overwhelmingly transferred to the verification (classification response) phase of search. Implicit category learning led to faster search response and shorter target dwell times relative to explicit category learning, consistent with the notion that explicit rule classification relies on a more deliberative response strategy. Participants who transferred the correct category learning strategy to the search guidance phase produced stronger search guidance (defined as the proportion of trials on which the target was the first item fixated) with evidence of greater guidance in implicit-strategy learners. This demonstrates that both implicit and explicit categorization systems contribute to categorical search and produce dissociable patterns of data.

Generalisation of value-based attentional priority is category-specific

A large body of research suggests that previously reward-associated stimuli can capture attention. Recent evidence also suggests that value-driven attentional biases can occur for a particular category of objects. However, it is unclear how broadly these category-level attentional biases can generalise. In the present study, we examined whether value-driven attentional biases can generalise to new exemplars of a category or semantically related categories using a modified version of the value-driven attentional capture paradigm. In an initial training phase, participants searched for two categories of objects and were rewarded for correctly fixating members of one target category. In a subsequent test phase, participants searched for two new categories of objects. A new exemplar of one of the previous target categories or a member of a semantically related category could appear as a critical distractor in this phase. Participants were more likely to initially fixate the critical distractor and fixated the distractor longer when it was a new exemplar of the previously rewarded category. However, similar findings were not observed for members of semantically related categories. Together, these findings suggest that the generalisation of value-based attentional priority is category-specific.

Good-enough attentional guidance

Theories of attention posit that attentional guidance operates on information held in a target template within memory. The template is often thought to contain veridical target features, akin to a photograph, and to guide attention to objects that match the exact target features. However, recent evidence suggests that attentional guidance is highly flexible and often guided by non-veridical features, a subset of features, or only associated features. We integrate these findings and propose that attentional guidance maximizes search efficiency based on a 'good-enough' principle to rapidly localize candidate target objects. Candidates are then serially interrogated to make target-match decisions using more precise information. We suggest that good-enough guidance optimizes the speed-accuracy-effort trade-offs inherent in each stage of visual search.

Children's knowledge of superordinate words predicts subsequent inductive reasoning

Children's early language knowledge-typically assessed using standardized word comprehension tests or through parental reports-has been positively linked to a variety of later outcomes, from reasoning tests to academic performance to income and health. To better understand the mechanisms behind these links, we examined whether knowledge of certain "seed words"-words with high inductive potential-is positively associated with inductive reasoning. This hypothesis stems from prior work on the effects of language on categorization suggesting that certain words may be important for helping people to deploy categorical hypotheses. Using a longitudinal design, we assessed 36 2- to 4-year-old children's knowledge of 333 words of varying levels of generality (e.g., toy vs. pinwheel, number vs. five). We predicted that adjusting for overall vocabulary, knowledge of more general words (e.g., toy, number) would predict children's performance on inductive reasoning tasks administered 6 months later (i.e., a subset of the Stanford-Binet Intelligence Scales for Early Childhood-Fifth Edition [SB-5] and Woodcock-Johnson Tests of Cognitive Abilities [WJ] concept formation tasks). This prediction was confirmed for one of the measures of inductive reasoning (i.e., the SB-5 but not the WJ) and notably for the task considered to be less reliant on language. Although our experimental design demonstrates only a correlational relationship between seed word knowledge and inductive reasoning ability, our results are consistent with the possibility that early knowledge of certain seed words facilitates performance on putatively nonverbal reasoning tasks.

Negative cues minimize visual search specificity effects

Prior target knowledge (i.e., positive cues) improves visual search performance. However, there is considerable debate about whether distractor knowledge (i.e., negative cues) can guide search. Some studies suggest the active suppression of negatively cued search items, while others suggest the initial capture of attention by negatively cued items. Prior work has used pictorial or specific text cues but has not explicitly compared them. We build on that work by comparing positive and negative cues presented pictorially and as categorical text labels using photorealistic objects and eye movement measures. Search displays contained a target (cued on positive trials), a lure from the target category (cued on negative trials), and four categorically-unrelated distractors. Search performance with positive cues resulted in stronger attentional guidance and faster object recognition for pictorial relative to categorical cues (i.e., a pictorial advantage, suggesting specific visual details afforded by pictorial cues improved search). However, in most search performance metrics, negative cues mitigate the pictorial advantage. Given that the negatively cued items captured attention, generated target guidance but mitigated the pictorial advantage, these results are partly consistent with both existing theories. Specific visual details provided in positive cues produce a large pictorial advantage in all measures, whereas specific visual details in negative cues only produce a small pictorial advantage for object recognition but not for attentional guidance. This asymmetry in the pictorial advantage suggests that the down-weighting of specific negatively cued visual features is less efficient than the up-weighting of specific positively cued visual features.

Real-world object categories and scene contexts conjointly structure statistical learning for the guidance of visual search

We examined how object categories and scene contexts act in conjunction to structure the acquisition and use of statistical regularities to guide visual search. In an exposure session, participants viewed five object exemplars in each of two colors in each of 42 real-world categories. Objects were presented individually against scene context backgrounds. Exemplars within a category were presented with different contexts as a function of color (e.g., the five red staplers were presented with a classroom scene, and the five blue staplers with an office scene). Participants then completed a visual search task, in which they searched for novel exemplars matching a category label cue among arrays of eight objects superimposed over a scene background. In the context-match condition, the color of the target exemplar was consistent with the color associated with that combination of category and scene context from the exposure phase (e.g., a red stapler in a classroom scene). In the context-mismatch condition, the color of the target was not consistent with that association (e.g., a red stapler in an office scene). In two experiments, search response time was reliably lower in the context-match than in the context-mismatch condition, demonstrating that the learning of category-specific color regularities was itself structured by scene context. The results indicate that categorical templates retrieved from long-term memory are biased toward the properties of recent exemplars and that this learning is organized in a scene-specific manner.

Categorical cuing: Object categories structure the acquisition of statistical regularities to guide visual search

Recent statistical regularities have been demonstrated to influence visual search across a wide variety of learning mechanisms and search features. To function in the guidance of real-world search, however, such learning must be contingent on the context in which the search occurs and the object that is the target of search. The former has been studied extensively under the rubric of contextual cuing. Here, we examined, for the first time, categorical cuing: The role of object categories in structuring the acquisition of statistical regularities used to guide visual search. After an exposure session in which participants viewed six exemplars with the same general color in each of 40 different real-world categories, they completed a categorical search task, in which they searched for any member of a category based on a label cue. Targets that matched recent within-category regularities were found faster than targets that did not (Experiment 1). Such categorical cuing was also found to span multiple recent colors within a category (Experiment 2). It was observed to influence both the guidance of search to the target object (Experiment 3) and the basic operation of assigning single exemplars to categories (Experiment 4). Finally, the rapid acquisition of category-specific regularities was also quickly modified, with the benefit rapidly decreasing during the search session as participants were exposed equally to the two possible colors in each category. The results demonstrate that object categories organize the acquisition of perceptual regularities and that this learning exerts strong control over the instantiation of the category representation as a template for visual search. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Semantically predictable input streams impede gaze-orientation to surprising locations

When available, people use prior knowledge to predict dimensions of future events such as their location and semantic features. However, few studies have examined how multi-dimensional predictions are implemented, and mechanistic accounts are absent. Using eye tracking, we evaluated whether predictions of target-location and target-category interact during the earliest stages of orientation. We presented stochastic series so that across four conditions, participants could predict either the location of the next target-image, its semantic category, both dimensions, or neither. Participants observed images in absence of any task involving their semantic content. We modeled saccade latencies using ELATER, a rise-to-threshold model that accounts for accumulation rate (AR), variance of AR over trials, and variance of decision baseline. The main findings were: 1) AR scaled with the degree of surprise associated with a target's location; 2) predictability of semantic-category hindered saccade latencies, suggesting a bottleneck in implementing joint predictions; 3) saccades to targets that satisfied semantic expectations were associated with greater AR-variance than saccades to semantically-surprising images, consistent with a richer repertoire of early evaluative processes for semantically-expected images. Predictability of target-category also impacted gaze pre-positioning prior to target presentation. The results indicate a strong interaction between foreknowledge of object location and semantics during stimulus-guided saccades, and suggest statistical regularities in an input stream can also impact anticipatory, non-stimulus-guided processes.

Eye movements and the label feedback effect: Speaking modulates visual search via template integrity

The label-feedback hypothesis (Lupyan, 2012) proposes that language modulates low- and high-level visual processing, such as priming visual object perception. Lupyan and Swingley (2012) found that repeating target names facilitates visual search, resulting in shorter response times (RTs) and higher accuracy. In the present investigation, we conceptually replicated and extended their study, using additional control conditions and recording eye movements during search. Our goal was to evaluate whether self-directed speech influences target locating (i.e. attentional guidance) or object perception (i.e., distractor rejection and target appreciation). In three experiments, during object search, people spoke target names, nonwords, irrelevant (absent) object names, or irrelevant (present) object names (all within-participants). Experiments 1 and 2 examined search RTs and accuracy: Speaking target names improved performance, without differences among the remaining conditions. Experiment 3 incorporated eye-tracking: Gaze fixation patterns suggested that language does not affect attentional guidance, but instead affects both distractor rejection and target appreciation. When search trials were conditionalized according to distractor fixations, language effects became more orderly: Search was fastest while people spoke target names, followed in linear order by the nonword, distractor-absent, and distractor-present conditions. We suggest that language affects template maintenance during search, allowing fluent differentiation of targets and distractors. Materials, data, and analyses can be retrieved here: https://osf.io/z9ex2/.

Effects of Language on Visual Perception

Does language change what we perceive? Does speaking different languages cause us to perceive things differently? We review the behavioral and electrophysiological evidence for the influence of language on perception, with an emphasis on the visual modality. Effects of language on perception can be observed both in higher-level processes such as recognition and in lower-level processes such as discrimination and detection. A consistent finding is that language causes us to perceive in a more categorical way. Rather than being fringe or exotic, as they are sometimes portrayed, we discuss how effects of language on perception naturally arise from the interactive and predictive nature of perception.

Typicality guides attention during categorical search, but not universally so

The degree to which an item is rated as being a typical member of its category influences an observer's ability to find that item during word-cued search. However, there are conflicting accounts as to whether or not typicality affects attentional guidance to categorical items, or whether it affects some other aspect of the search process. In this study, we employed word-cued search and eye tracking to disentangle typicality effects on attentional guidance and target verification across differing category cue specificities (i.e., superordinate or basic-level cues), while also varying the degree of similarity between targets and non-targets. We found that typicality influenced attentional guidance when searchers were cued at the superordinate level (e.g., clothing). When cues were provided at the basic level (e.g., pants), typicality did not influence attentional guidance, and only affected target verification when there was featural similarity between targets and non-targets. When a searcher uses a target template comprising features cued at the basic level, therefore, target/non-target similarity produces interference that affects attentional guidance, but we did not find evidence that it also affects target verification.

Template-to-distractor distinctiveness regulates visual search efficiency

All models of attention include the concept of an attentional template (or a target or search template). The template is conceptualized as target information held in memory that is used for prioritizing sensory processing and determining if an object matches the target. It is frequently assumed that the template contains a veridical copy of the target. However, we review recent evidence showing that the template encodes a version of the target that is adapted to the current context (e.g. distractors, task, etc.); information held within the template may include only a subset of target features, real world knowledge, pre-existing perceptual biases, or even be a distorted version of the veridical target. We argue that the template contents are customized in order to maximize the ability to prioritize information that distinguishes targets from distractors. We refer to this as template-to-distractor distinctiveness and hypothesize that it contributes to visual search efficiency by exaggerating target-to-distractor dissimilarity.

Specifying the precision of guiding features for visual search

Visual search is the task of finding things with uncertain locations. Despite decades of research, the features that guide visual search remain poorly specified, especially in realistic contexts. This study tested the role of two features-shape and orientation-both in the presence and absence of hue information. We conducted five experiments to describe preview-target mismatch effects, decreases in performance caused by differences between the image of the target as it appears in the preview and as it appears in the actual search display. These mismatch effects provide direct measures of feature importance, with larger performance decrements expected for more important features. Contrary to previous conclusions, our data suggest that shape and orientation only guide visual search when color is not available. By varying the probability of mismatch in each feature dimension, we also show that these patterns of feature guidance do not change with the probability that the previewed feature will be invalid. We conclude that the target representations used to guide visual search are much less precise than previously believed, with participants encoding and using color and little else. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Whereof one cannot speak: How language and capture of visual attention interact

Our research addresses the important question whether language influences cognition by studying crosslinguistic differences in nonlinguistic visual search tasks. We investigated whether capture of visual attention is mediated by characteristics corresponding to concepts that are differently expressed across different languages. Korean grammatically distinguishes between tight- (kkita) and loose-fit (nehta) containment whereas German collapses them into a single semantic category (in). Although linguistic processing was neither instructed nor necessary to perform the visual search task, we found that Korean speakers showed attention capture by non-instructed but target-coincident (Experiment 1) or distractor-coincident (Experiments 4 and 5) spatial fitness of the stimuli, whereas German speakers were not sensitive to it. As the tight- versus loose-fit distinction is grammaticalized only in the Korean but not the German language, our results demonstrate that language influences which visual features capture attention even in non-linguistic tasks that do not require paying attention to these features. In separate control experiments (Experiments 2 and 3), we ruled out cultural or general cognitive group differences between Korean and German speaking participants as alternative explanations. We outline the mechanisms underlying these crosslinguistic differences in nonlinguistic visual search behaviors. This is the first study showing that linguistic spatial relational concepts held in long-term memory can affect attention capture in visual search tasks.

Contingent capture during search for alphanumerical characters: A case of feature-based capture or of conceptual category membership?

To distinguish if search for alphanumerical characters is based on features or on conceptual category membership, we conducted two experiments where we presented upright and inverted characters as cues in a contingent-capture protocol. Here, only cues matching the top-down search template (e.g., a letter cue when searching for target letters) capture attention and lead to validity effects: shorter search times and fewer errors for validly than invalidly cued targets. Top-down nonmatching cues (e.g., a number cue when searching for target letters) do not capture attention. To tell a feature-based explanation from one based on conceptual category membership, we used both upright (canonical) and inverted characters as cues. These cues share the same features, but inverted cues cannot be conceptually categorized as easily as upright cues. Thus, we expected no difference between upright and inverted cues when search is feature-based, whereas inverted cues would elicit no or at least considerably weaker validity effects if search relies on conceptual category membership. Altogether, the results of both experiments (with overlapping and with separate sets of characters for cues and targets) provide evidence for search based on feature representations, as among other things, significant validity effects were found with upright and inverted characters as cues. However, an influence of category membership was also evident, as validity effects of inverted characters were diminished.

Object Categorization in Finer Levels Relies More on Higher Spatial Frequencies and Takes Longer

The human visual system contains a hierarchical sequence of modules that take part in visual perception at different levels of abstraction, i.e., superordinate, basic, and subordinate levels. One important question is to identify the “entry” level at which the visual representation is commenced in the process of object recognition. For a long time, it was believed that the basic level had a temporal advantage over two others. This claim has been challenged recently. Here we used a series of psychophysics experiments, based on a rapid presentation paradigm, as well as two computational models, with bandpass filtered images of five object classes to study the processing order of the categorization levels. In these experiments, we investigated the type of visual information required for categorizing objects in each level by varying the spatial frequency bands of the input image. The results of our psychophysics experiments and computational models are consistent. They indicate that the different spatial frequency information had different effects on object categorization in each level. In the absence of high frequency information, subordinate and basic level categorization are performed less accurately, while the superordinate level is performed well. This means that low frequency information is sufficient for superordinate level, but not for the basic and subordinate levels. These finer levels rely more on high frequency information, which appears to take longer to be processed, leading to longer reaction times. Finally, to avoid the ceiling effect, we evaluated the robustness of the results by adding different amounts of noise to the input images and repeating the experiments. As expected, the categorization accuracy decreased and the reaction time increased significantly, but the trends were the same. This shows that our results are not due to a ceiling effect. The compatibility between our psychophysical and computational results suggests that the temporal advantage of the superordinate (resp. basic) level to basic (resp. subordinate) level is mainly due to the computational constraints (the visual system processes higher spatial frequencies more slowly, and categorization in finer levels depends more on these higher spatial frequencies).

Visual search for object categories is predicted by the representational architecture of high-level visual cortex

Visual search is a ubiquitous visual behavior, and efficient search is essential for survival. Different cognitive models have explained the speed and accuracy of search based either on the dynamics of attention or on similarity of item representations. Here, we examined the extent to which performance on a visual search task can be predicted from the stable representational architecture of the visual system, independent of attentional dynamics. Participants performed a visual search task with 28 conditions reflecting different pairs of categories (e.g., searching for a face among cars, body among hammers, etc.). The time it took participants to find the target item varied as a function of category combination. In a separate group of participants, we measured the neural responses to these object categories when items were presented in isolation. Using representational similarity analysis, we then examined whether the similarity of neural responses across different subdivisions of the visual system had the requisite structure needed to predict visual search performance. Overall, we found strong brain/behavior correlations across most of the higher-level visual system, including both the ventral and dorsal pathways when considering both macroscale sectors as well as smaller mesoscale regions. These results suggest that visual search for real-world object categories is well predicted by the stable, task-independent architecture of the visual system.

NEW & NOTEWORTHY

Here, we ask which neural regions have neural response patterns that correlate with behavioral performance in a visual processing task. We found that the representational structure across all of high-level visual cortex has the requisite structure to predict behavior. Furthermore, when directly comparing different neural regions, we found that they all had highly similar category-level representational structures. These results point to a ubiquitous and uniform representational structure in high-level visual cortex underlying visual object processing.