Visual crowding cannot be wholly explained by feature pooling

When periphery rules: Enhanced sampling weights of the visual periphery in crowding across dimensions

Crowding, our inability to identify a feature or object - the target - due to its proximity to adjacent features or objects - flankers - exhibits a notable inner-outer asymmetry. This asymmetry is characterized by the outer flanker - more peripheral - creating stronger interference than the inner one - closer to the fovea. But crowding is not uniform across different feature dimensions. For example, in the case of orientation, this asymmetry reflects misreport errors: observers are more likely to misidentify the outer flanker as the target than the inner one. However, for spatial frequency (SF), observers tend to average the features of the target and flankers (Yashar et al., 2019). Here, we investigated whether and how the inner-outer asymmetry manifests across various feature dimensions: Gabor orientation and SF, as well as T-shape tilt and color. We reanalyzed continuous estimation reports data published by Yashar et al. (2019), focusing on a previously unanalyzed factor: the relative position of each flanker (inner vs. outer). We fit probabilistic models that assign variable weights to each flanker. Our analysis revealed that observers predominantly misreport the outer flanker as the target with Gabor orientation and T-shape tilt stimuli, and slightly so with color stimuli, whereas with Gabor SF, observers perform a weighted average of all features but also with a bias towards the outer flanker over the inner one. These findings suggest that an increased weighting on the more peripheral items is a general characteristic of crowding in peripheral vision.

Interference across time: dissociating short from long temporal interference

Our ability to identify an object is often impaired by the presence of preceding and/or succeeding task-irrelevant items. Understanding this temporal interference is critical for any theoretical account of interference across time and for minimizing its detrimental effects. Therefore, we used the same sequences of 3 orientation items, orientation estimation task, and computational models, to examine temporal interference over both short (<150 ms; visual masking) and long (175-475 ms; temporal crowding) intervals. We further examined how inter-item similarity modifies these different instances of temporal interference. Qualitatively different results emerged for interference of different scales. Interference over long intervals mainly degraded the precision of the target encoding while interference over short intervals mainly affected the signal-to-noise ratio. Although both interference instances modulated substitution errors (reporting a wrong item) and were alleviated with dissimilar items, their characteristics were markedly disparate. These findings suggest that different mechanisms mediate temporal interference of different scales.

Foveal crowding for large and small Landolt Cs: Similarity and Attention

We compare the recognition of foveal crowded Landolt Cs of two sizes: brief (40 ms), large, low-contrast Cs and high-contrast (1 sec) tests at the resolution limit of the visual system. In different series, the test Landolt C was surrounded by two identical distractors located symmetrically along the horizontal or by a single distractor. The distractors were Landolt Cs or rings. At the resolution limit, the critical spacing was similar in the two series and did not depend on the type of distractor. The result supports the hypothesis that crowding at the resolution limit occurs when both the test and the distractors fall into the same smallest receptive field responsible for the target recognition. For large stimuli, at almost all separations distractors of the same shape caused greater impairment than did rings, and recognition errors were non-random. The critical spacing was equal to 0.5 test diameters only in the presence of one distracting Landolt C. This result suggests that attention is involved: When one distractor is added, involuntary attention, which is directed to the centre of gravity of the stimulus, can lead to confusion of features that are present in both tests and distractors and thus to non-random errors.

Large depth differences between target and flankers can increase crowding: Evidence from a multi-depth plane display

Crowding occurs when the presence of nearby features causes highly visible objects to become unrecognizable. Although crowding has implications for many everyday tasks and the tremendous amounts of research reflect its importance, surprisingly little is known about how depth affects crowding. Most available studies show that stereoscopic disparity reduces crowding, indicating that crowding may be relatively unimportant in three-dimensional environments. However, most previous studies tested only small stereoscopic differences in depth in which disparity, defocus blur, and accommodation are inconsistent with the real world. Using a novel multi-depth plane display, this study investigated how large (0.54-2.25 diopters), real differences in target-flanker depth, representative of those experienced between many objects in the real world, affect crowding. Our findings show that large differences in target-flanker depth increased crowding in the majority of observers, contrary to previous work showing reduced crowding in the presence of small depth differences. Furthermore, when the target was at fixation depth, crowding was generally more pronounced when the flankers were behind the target as opposed to in front of it. However, when the flankers were at fixation depth, crowding was generally more pronounced when the target was behind the flankers. These findings suggest that crowding from clutter outside the limits of binocular fusion can still have a significant impact on object recognition and visual perception in the peripheral field.

The impact of training on the inner-outer asymmetry in crowding

Inner-outer asymmetry, where the outer flanker induces stronger crowding than the inner flanker, is a hallmark property of visual crowding. It is unclear the contribution of inner-outer asymmetry to the pattern of crowding errors (biased predominantly toward the flanker identities) and the role of training on crowding errors. In a typical radial crowding display, 20 observers were asked to report the orientation of a target Gabor (7.5° eccentricity) flanked by either an inner or outer Gabor along the horizontal meridian. The results showed that outer flanker conditions induced stronger crowding, accompanied by assimilative errors to the outer flanker for similar target/flanker elements. In contrast, the inner flanker condition exhibited weaker crowding, with no significant patterns of crowding errors. A population coding model showed that the flanker weights in the outer flanker condition were significantly higher than those in the inner flanker condition. Nine observers continued to train the outer flanker condition for four sessions. Training reduced inner-outer asymmetry and reduced flanker weights to the outer flanker. The learning effects were retained over 4 to 6 months. Individual differences in the appearance of crowding errors, the strength of inner-outer asymmetry, and the training effects were evident. Nevertheless, our findings indicate that different crowding mechanisms may be responsible for the asymmetric crowding effects induced by inner and outer flankers, with the outer flankers dominating the appearance more than the inner ones. Training reduces inner-outer asymmetry by reducing target/flanker confusion, and learning is persistent over months, suggesting that perceptual learning has the potential to improve visual performance by promoting neural plasticity.

The role of spatial attention in crowding and feature binding

Crowding refers to the failure to identify a peripheral object due to nearby objects (flankers). A hallmark of crowding is inner-outer asymmetry; that is, the outer flanker (more peripheral) produces stronger interference than the inner one. Here, by manipulating attention, we tested the predictions of two competing accounts: the attentional account, which predicts a positive attentional effect on the inner-outer asymmetry (i.e., attention to the outer flanker will increase asymmetry) and the receptive field size account, which predicts a negative attentional effect. In Experiment 1, observers estimated a Gabor target orientation. A peripheral pre-cue drew attention to one of three locations: target, inner flanker, or outer flanker. Probabilistic mixture modeling demonstrated asymmetry by showing that observers often misreported the outer-flanker orientation as the target. Interestingly, the outer cue led to a higher misreport rate of the outer flanker, and the inner cue led to a lower misreport rate of the outer flanker. Experiment 2 tested the effect of crowding and attention on incoherent object reports (i.e., binding errors, reporting the tilt of one presented item with the color of another item). In each trial, observers estimated both the tilt and color of the target. Attention merely increased coherent target reports, but not coherent flanker reports. The results suggest that the locus of spatial attention plays an essential role in crowding, as well as inner-outer asymmetry, and demonstrate that crowding and feature binding are closely related. However, our findings are inconsistent with the view that covert attention automatically binds features together.

Crowding changes appearance systematically in peripheral, amblyopic, and developing vision

Visual crowding is the disruptive effect of clutter on object recognition. Although most prominent in adult peripheral vision, crowding also disrupts foveal vision in typically developing children and those with strabismic amblyopia. Do these crowding effects share the same mechanism? Here we exploit observations that crowded errors in peripheral vision are not random: Target objects appear either averaged with the flankers (assimilation) or replaced by them (substitution). If amblyopic and developmental crowding share the same mechanism, then their errors should be similarly systematic. We tested foveal vision in children aged 3 to 8 years with typical vision or strabismic amblyopia and peripheral vision in typical adults. The perceptual effects of crowding were measured by requiring observers to adjust a reference stimulus to match the perceived orientation of a target “Vac-Man” element. When the target was surrounded by flankers that differed by ± 30°, all three groups (adults and children with typical or amblyopic vision) reported orientations between the target and flankers (assimilation). Errors were reduced with ± 90° differences but primarily matched the flanker orientation (substitution) when they did occur. A population pooling model of crowding successfully simulated this pattern of errors in all three groups. We conclude that the perceptual effects of amblyopic and developing crowding are systematic and resemble the near periphery in adults, suggesting a common underlying mechanism.

Mixture-modeling approach reveals global and local processes in visual crowding

Crowding refers to the inability to recognize objects in clutter, setting a fundamental limit on various perceptual tasks such as reading and facial recognition. While prevailing models suggest that crowding is a unitary phenomenon occurring at an early level of processing, recent studies have shown that crowding might also occur at higher levels of representation. Here we investigated whether local and global crowding interference co-occurs within the same display. To do so, we tested the distinctive contribution of local flanker features and global configurations of the flankers on the pattern of crowding errors. Observers (n = 27) estimated the orientation of a target when presented alone or surrounded by flankers. Flankers were grouped into a global configuration, forming an illusory rectangle when aligned or a rectangular configuration when misaligned. We analyzed the error distributions by fitting probabilistic mixture models. Results showed that participants often misreported the orientation of a flanker instead of that of the target. Interestingly, in some trials the orientation of the global configuration was misreported. These results suggest that crowding occurs simultaneously across multiple levels of visual processing and crucially depends on the spatial configuration of the stimulus. Our results pose a challenge to models of crowding with an early single pooling stage and might be better explained by models which incorporate the possibility of multilevel crowding and account for complex target-flanker interactions.

Spatial Attention Enhances Crowded Stimulus Encoding Across Modeled Receptive Fields by Increasing Redundancy of Feature Representations

Any visual system, biological or artificial, must make a trade-off between the number of units used to represent the visual environment and the spatial resolution of the sampling array. Humans and some other animals are able to allocate attention to spatial locations to reconfigure the sampling array of receptive fields (RFs), thereby enhancing the spatial resolution of representations without changing the overall number of sampling units. Here, we examine how representations of visual features in a fully convolutional neural network interact and interfere with each other in an eccentricity-dependent RF pooling array and how these interactions are influenced by dynamic changes in spatial resolution across the array. We study these feature interactions within the framework of visual crowding, a well-characterized perceptual phenomenon in which target objects in the visual periphery that are easily identified in isolation are much more difficult to identify when flanked by similar nearby objects. By separately simulating effects of spatial attention on RF size and on the density of the pooling array, we demonstrate that the increase in RF density due to attention is more beneficial than changes in RF size for enhancing target classification for crowded stimuli. Furthermore, by varying target/flanker spacing, as well as the spatial extent of attention, we find that feature redundancy across RFs has more influence on target classification than the fidelity of the feature representations themselves. Based on these findings, we propose a candidate mechanism by which spatial attention relieves visual crowding through enhanced feature redundancy that is mostly due to increased RF density.

Temporal crowding is a unique phenomenon reflecting impaired target encoding over large temporal intervals

Crowding refers to impaired object identification when presented with other objects, and it is well established that spatial crowding-crowding from adjacent objects-affects many aspects of visual perception and cognition. A similar interference also occurs across time-the identification of a target object is impaired when distracting objects precede and succeed it. When such interference is observed with relatively long interitem intervals it is termed temporal crowding. Thus far, little was known about temporal crowding and its underlying processes. Particularly it was unknown which aspects of visual processing are impaired by temporal crowding, and the answer to this question bears critical theoretical implications. To reveal the nature of this impairment we used a continuous-report task and a mixture-model analysis. In three experiments, observers viewed sequences of three oriented items separated by relatively long intervals (170-475ms). The target was the second item in the sequence, and the task was to reproduce its orientation. The findings suggest that temporal crowding impairs target encoding and increases substitution errors, but there was no evidence of a reduced signal-to-noise ratio. This pattern of results was similar regardless of stimuli duration and target-distractor similarity. However, it differed considerably from the pattern found for ordinary masking and spatial crowding, indicating that temporal crowding is a unique phenomenon. Moreover, the finding that temporal crowding affected the precision of target encoding even when the items were separated by almost half a second suggests that visual processing requires a surprisingly long time to complete.

Mixture model investigation of the inner-outer asymmetry in visual crowding reveals a heavier weight towards the visual periphery

Crowding, the failure to identify a peripheral item in clutter, is an essential bottleneck in visual information processing. A hallmark characteristic of crowding is the inner-outer asymmetry in which the outer flanker (more eccentric) produces stronger interference than the inner one (closer to the fovea). We tested the contribution of the inner-outer asymmetry to the pattern of crowding errors in a typical radial crowding display in which both flankers are presented simultaneously on the horizontal meridian. In two experiments, observers were asked to estimate the orientation of a Gabor target. Instead of the target, observers reported the outer flanker much more frequently than the inner one. When the target was the outer Gabor, crowding was reduced. Furthermore, when there were four flankers, two on each side of the target, observers misreported the outer flanker adjacent to the target, not the outermost flanker. Model comparisons suggested that orientation crowding reflects sampling over a weighted sum of the represented features, in which the outer flanker is more heavily weighted compared to the inner one. Our findings reveal a counterintuitive phenomenon: in a radial arrangement of orientation crowding, within a region of selection, the outer item dominates appearance more than the inner one.

Can rhythm-induced attention improve the perceptual representation?

Temporal attention can be entrained exogenously to rhythms. Indeed, faster and more accurate responses were previously found when the target appeared in-phase with a preceding rhythm in comparison to when it was out of phase. However, the nature of this rhythm-induced attentional effect is not well understood. To better understand the processes underlying rhythm-induced attention, we employed a continuous measure of perceived orientation and a mixture-model analysis. A trial in our study started with a sequence of auditory beeps separated by a fixed inter-beeps interval in the regular (rhythmic) condition or by variable inter-beeps intervals in the irregular condition. A visual target–a line embedded in a circle–followed the sequence. The ‘critical’ interval between the last beep and the target was chosen randomly from several possible Inter-Onset Intervals (IOIs), of which only one was in-phase with the rhythm. The target was followed by a probe line, and the participants were asked to rotate it to reproduce the target’s orientation. The measure of performance for a given trial was the difference in degrees between the orientation of the target and that reproduced by the observer. We found that guessing rate was lower with regular than irregular rhythms. However, there was no effect of rhythm type (regular vs irregular) on the quality of representation (measured as the variability in reproducing the target). Furthermore, the rhythm effect was present only when rhythm type was fixed within a block, and it was found with all IOIs, not just the in-phase IOI. This lack of specificity suggests that these results reflect a general effect of rhythm on alertness.

Response selection modulates crowding: a cautionary tale for invoking top-down explanations

Object recognition in the periphery is limited by clutter. This phenomenon of visual crowding is ameliorated when the objects are dissimilar. This effect of inter-object similarity has been extensively studied for low-level features and is thought to reflect bottom-up processes. Recently, crowding was also found to be reduced when objects belonged to explicitly distinct groups; that is, crowding was weak when they had low group membership similarity. It has been claimed that top-down knowledge is necessary to explain this effect of group membership, implying that the effect of similarity on crowding cannot be a purely bottom-up process. We tested the claim that the effect of group membership relies on knowledge in two experiments and found that neither explicit knowledge about differences in group membership nor the possibility of acquiring knowledge about target identities is necessary to produce the effects. These results suggest that top-down processes need not be invoked to explain the effect of group membership. Instead, we suggest that differences in flanker reportability that emerge from the differences in group membership are the source of the effect. That is, when targets and flankers are sampled from distinct groups, flankers cannot be inadvertently reported, leading to fewer errors and hence weaker crowding. Further, we argue that this effect arises at the stage of response selection. This conclusion is well supported by an analytical model based on these principles. We conclude that previously observed effects in crowding attributed to top-down or higher level processes might instead be due to post-perceptual response selection strategies.

Crowding and Binding: Not All Feature Dimensions Behave in the Same Way

Humans often fail to identify a target because of nearby flankers. The nature and stages at which this crowding occurs are unclear, and whether crowding operates via a common mechanism across visual dimensions is unknown. Using a dual-estimation report (N = 42), we quantitatively assessed the processing of features alone and in conjunction with another feature both within and between dimensions. Under crowding, observers misreported colors and orientations (i.e., reported a flanker value instead of the target's value) but averaged the target's and flankers' spatial frequencies (SFs). Interestingly, whereas orientation and color errors were independent, orientation and SF errors were interdependent. These qualitative differences of errors across dimensions revealed a tight link between crowding and feature binding, which is contingent on the type of feature dimension. These results and a computational model suggest that crowding and misbinding are due to pooling across a joint coding of orientations and SFs but not of colors.

Attentional guidance varies with display density

The aim of the present study was to investigate how display density affects attentional guidance in heterogeneous search displays. In Experiment 1 we presented observers with heterogeneous sparse and dense search displays which were adaptively changed over the course of the experiment using genetic algorithms. We generated random displays, and based upon fastest search times, the displays that allowed most efficient search were selected to generate new displays for the next generations, thus revealing which properties facilitated or inhibited target search across display densities. The results showed that the prevalence of distractors sharing the target color was substantially reduced over generations in sparse displays. Dense displays also evolved to contain less distractors sharing the target color but only when the orientation of the distractors resembled the target orientation. More importantly, spatial analyses revealed that changes across generations occurred across all areas in sparse displays but were confined to occur around the target location only in dense displays. In Experiment 2, in which we used a factorial design, we showed that the presence of potentially interfering distractors in the target area affected search in dense displays but not in sparse displays. Together the results suggest that the role of salience-driven attentional guidance is larger in dense than sparse displays even in the absence of display homogeneity.

Multi-level Crowding and the Paradox of Object Recognition in Clutter

In everyday life, we are constantly surrounded by complex and cluttered scenes. In such cluttered environments, visual perception is primarily limited by crowding, the deleterious influence of nearby objects on object recognition. For the past several decades, visual crowding was assumed to occur at a single stage, only between low-level features or object parts, thus dismantling, destroying, or filtering object information. A large and converging body of evidence has demonstrated that this assumption is false: crowding occurs at multiple stages of visual analysis, and information passes through crowding at each of these stages. This converging empirical evidence points to a seeming paradox: crowding happens at multiple levels, which would seem to impair object recognition, and yet visual information at each of those levels is maintained intact and influences subsequent higher-level visual processing. Thus, while crowding impairs the access we have to visual information at many levels, it does not impair the representation of that information. The resolution of this paradox reveals how the visual system strikes a balance between the limits of object selection and the desire to represent multiple levels of visual information throughout cluttered scenes. Understanding crowding is therefore key to resolving the relationship between the richness of object and scene representations and the limits of conscious object recognition.

Challenges to pooling models of crowding: Implications for visual mechanisms

A set of phenomena known as crowding reveal peripheral vision's vulnerability in the face of clutter. Crowding is important both because of its ubiquity, making it relevant for many real-world tasks and stimuli, and because of the window it provides onto mechanisms of visual processing. Here we focus on models of the underlying mechanisms. This review centers on a popular class of models known as pooling models, as well as the phenomenology that appears to challenge a pooling account. Using a candidate high-dimensional pooling model, we gain intuitions about whether a pooling model suffices and reexamine the logic behind the pooling challenges. We show that pooling mechanisms can yield substitution phenomena and therefore predict better performance judging the properties of a set versus a particular item. Pooling models can also exhibit some similarity effects without requiring mechanisms that pool at multiple levels of processing, and without constraining pooling to a particular perceptual group. Moreover, we argue that other similarity effects may in part be due to noncrowding influences like cuing. Unlike low-dimensional straw-man pooling models, high-dimensional pooling preserves rich information about the stimulus, which may be sufficient to support high-level processing. To gain insights into the implications for pooling mechanisms, one needs a candidate high-dimensional pooling model and cannot rely on intuitions from low-dimensional models. Furthermore, to uncover the mechanisms of crowding, experiments need to separate encoding from decision effects. While future work must quantitatively examine all of the challenges to a high-dimensional pooling account, insights from a candidate model allow us to conclude that a high-dimensional pooling mechanism remains viable as a model of the loss of information leading to crowding.

Object-based biased competition during covert spatial orienting

Biased-competition models assert that spatial attention facilitates visual perception by biasing competitive interactions in favor of relevant input. In line with this view, past work has shown that the benefits of covert spatial attention are greatest when targets must compete with interfering stimuli. Here we propose a boundary condition for the resolution of interference via exogenous attention: Attention resolves visual interference between targets and distractors, but only when they can be individuated into distinct representations. Thus, we propose that biased competition may be object-based. We replicated previous observations of larger attention effects when targets were flanked by irrelevant distractors (interference-present displays) than when targets were presented alone (interference-absent displays). Critically, we then showed that this amplification of cueing effects in the presence of interference is eliminated when strong crowding hampers individuation of the targets and distractors. Likewise, when targets were embedded within a noise mask that did not evoke the percept of an individuated distractor, the attention effects were equivalent across noise and lone-target displays. Thus, we conclude that exogenous spatial attention resolves interference in an object-based fashion that depends on the perception of individuated targets and distractors.

Beyond Bouma's window: How to explain global aspects of crowding?

In crowding, perception of an object deteriorates in the presence of nearby elements. Although crowding is a ubiquitous phenomenon, since elements are rarely seen in isolation, to date there exists no consensus on how to model it. Previous experiments showed that the global configuration of the entire stimulus must be taken into account. These findings rule out simple pooling or substitution models and favor models sensitive to global spatial aspects. In order to investigate how to incorporate global aspects into models, we tested a large number of models with a database of forty stimuli tailored for the global aspects of crowding. Our results show that incorporating grouping like components strongly improves model performance.

Feature contingencies when reading letter strings

Many models posit the use of distinctive spatial features to recognize letters of the alphabet, a fundamental component of reading. It has also been hypothesized that when letters are in close proximity, visual crowding may cause features to mislocalize between nearby letters, causing identification errors. Here, we took a data-driven approach to investigate these aspects of textual processing. Using data collected from subjects identifying each letter in thousands of lower-case letter trigrams presented in the peripheral visual field, we found characteristic error patterns in the results suggestive of the use of particular spatial features. Distinctive features were seldom entirely missed, and we found evidence for errors due to doubling, masking, and migration of features. Dependencies both amongst neighboring letters and in the responses revealed the contingent nature of processing letter strings, challenging the most basic models of reading that ignore either crowding or featural decomposition.

The Temporal Dynamic Relationship Between Attention and Crowding: Electrophysiological Evidence From an Event-Related Potential Study

Visual crowding is the difficulty experienced in identifying a target flanked by other objects within the peripheral visual field. Despite extensive research conducted on this topic, the precise relationship between attention and crowding is still debatable. One perspective suggests that crowding is a bottom-up and pre-attentive process, while another suggests that crowding is top-down and attentional. A third perspective proposes that crowding is a combination of bottom-up and top-down processes. To address this debate, the current study manipulated the attention and distance between targets and flankers, while simultaneously measuring event-related potentials, in human participants. Results indicated that, compared to uncrowded targets, crowded targets elicited more negative frontal N1 and P2 activity and a less negative occipital N1 activity, regardless of whether targets were attended or unattended, and a more positive occipital P2 activity when they were attended. Furthermore, the crowded minus uncrowded difference amplitude was more negative over the frontal region and more positive over the occipital region when the targets were attended, compared to when they were unattended during the N1 and P2 stages. This suggests that crowding, a concept that originates from Gestalt grouping, occurs automatically and can be modulated by attention.

Contextual-Dependent Attention Effect on Crowded Orientation Signals in Human Visual Cortex

A target becomes hard to identify with nearby visual stimuli. This phenomenon, known as crowding, places a fundamental limit on conscious perception and object recognition. To understand the neural representation of crowded stimuli, we used fMRI and a forward encoding model to reconstruct the target-specific feature from multivoxel activation patterns evoked by orientation patches. Orientation-selective response profiles were constructed in V1-V4 for a target embedded in different contexts. Subjects of both sexes either directed their attention over all the orientation patches or selectively to the target. In the context with a weak crowding effect, attending to the target enhanced the orientation selectivity of the response profile; such effect increased along the visual pathway. In the context with a strong crowding effect, attending to the target enhanced the orientation selectivity of the response profile in the earlier visual area, but not in V4. The increase and decrease of orientation selectivity along the visual hierarchy demonstrate a contextual-dependent attention effect on crowded orientation signals: in the context with a weak crowding effect, selective attention gradually resolves the target from nearby distractors along the hierarchy; in the context with a strong crowding effect, while selective attention maintains the target feature in the earlier visual area, its effect decreases in the downstream area. Our findings reveal how the human visual system represents the target-specific feature at multiple stages under the limit of attention selection in a cluttered scene.SIGNIFICANCE STATEMENT Using fMRI and a forward encoding model, we reconstructed orientation-selective response profiles for a target embedded in crowded contexts. In the context with a weak crowding effect, attention gradually resolves the target from nearby distractors along the visual hierarchy. In the context with a strong crowding effect, while the feature of the target is preserved in the early visual cortex, it degrades in the later visual processing stage. The increase and decrease of orientation selectivity along the visual hierarchy reveal how the human visual system strikes to present the target-specific feature under the limit of attention selection in a cluttered scene.

Critical resolution: A superior measure of crowding

Visual object recognition is essential for adaptive interactions with the environment. It is fundamentally limited by crowding, a breakdown of object recognition in clutter. The spatial extent over which crowding occurs is proportional to the eccentricity of the target object, but nevertheless varies substantially depending on various stimulus factors (e.g. viewing time, contrast). However, a lack of studies jointly manipulating such factors precludes predictions of crowding in more heterogeneous scenes, such as the majority of real life situations.

To establish how such co-occurring variations affect crowding, we manipulated combinations of 1) flanker contrast and backward masking, 2) flanker contrast and presentation duration, and 3) flanker preview and pop-out while measuring participants’ ability to correctly report the orientation of a target stimulus. In all three experiments, combining two manipulations consistently modulated the spatial extent of crowding in a way that could not be predicted from an additive combination. However, a simple transformation of the measurement scale completely abolished these interactions and all effects became additive. Precise quantitative predictions of the magnitude of crowding when combining multiple manipulations are thus possible when it is expressed in terms of what we label the ‘critical resolution’. Critical resolution is proportional to the inverse of the smallest flanker free area surrounding the target object necessary for its unimpaired identification. It offers a more parsimonious description of crowding than the traditionally used critical spacing and may thus constitute a measure of fundamental importance for understanding object recognition.

Visual Working Memory Is Independent of the Cortical Spacing Between Memoranda

The sensory recruitment hypothesis states that visual short-term memory is maintained in the same visual cortical areas that initially encode a stimulus' features. Although it is well established that the distance between features in visual cortex determines their visibility, a limitation known as crowding, it is unknown whether short-term memory is similarly constrained by the cortical spacing of memory items. Here, we investigated whether the cortical spacing between sequentially presented memoranda affects the fidelity of memory in humans (of both sexes). In a first experiment, we varied cortical spacing by taking advantage of the log-scaling of visual cortex with eccentricity, presenting memoranda in peripheral vision sequentially along either the radial or tangential visual axis with respect to the fovea. In a second experiment, we presented memoranda sequentially either within or beyond the critical spacing of visual crowding, a distance within which visual features cannot be perceptually distinguished due to their nearby cortical representations. In both experiments and across multiple measures, we found strong evidence that the ability to maintain visual features in memory is unaffected by cortical spacing. These results indicate that the neural architecture underpinning working memory has properties inconsistent with the known behavior of sensory neurons in visual cortex. Instead, the dissociation between perceptual and memory representations supports a role of higher cortical areas such as posterior parietal or prefrontal regions or may involve an as yet unspecified mechanism in visual cortex in which stimulus features are bound to their temporal order.

SIGNIFICANCE STATEMENT Although much is known about the resolution with which we can remember visual objects, the cortical representation of items held in short-term memory remains contentious. A popular hypothesis suggests that memory of visual features is maintained via the recruitment of the same neural architecture in sensory cortex that encodes stimuli. We investigated this claim by manipulating the spacing in visual cortex between sequentially presented memoranda such that some items shared cortical representations more than others while preventing perceptual interference between stimuli. We found clear evidence that short-term memory is independent of the intracortical spacing of memoranda, revealing a dissociation between perceptual and memory representations. Our data indicate that working memory relies on different neural mechanisms from sensory perception.

Visual crowding is a combination of an increase of positional uncertainty, source confusion, and featural averaging

Although we perceive a richly detailed visual world, our ability to identify individual objects is severely limited in clutter, particularly in peripheral vision. Models of such “crowding” have generally been driven by the phenomenological misidentifications of crowded targets: using stimuli that do not easily combine to form a unique symbol (e.g. letters or objects), observers typically confuse the source of objects and report either the target or a distractor, but when continuous features are used (e.g. orientated gratings or line positions) observers report a feature somewhere between the target and distractor. To reconcile these accounts, we develop a hybrid method of adjustment that allows detailed analysis of these multiple error categories. Observers reported the orientation of a target, under several distractor conditions, by adjusting an identical foveal target. We apply new modelling to quantify whether perceptual reports show evidence of positional uncertainty, source confusion, and featural averaging on a trial-by-trial basis. Our results show that observers make a large proportion of source-confusion errors. However, our study also reveals the distribution of perceptual reports that underlie performance in this crowding task more generally: aggregate errors cannot be neatly labelled because they are heterogeneous and their structure depends on target-distractor distance.

Can (should) theories of crowding be unified?

Objects in clutter are difficult to recognize, a phenomenon known as crowding. There is little consensus on the underlying mechanisms of crowding, and a large number of models have been proposed. There have also been attempts at unifying the explanations of crowding under a single model, such as the weighted feature model of Harrison and Bex (2015) and the texture synthesis model of Rosenholtz and colleagues (Balas, Nakano, & Rosenholtz, 2009; Keshvari & Rosenholtz, 2016). The goal of this work was to test various models of crowding and to assess whether a unifying account can be developed. Adopting Harrison and Bex's (2015) experimental paradigm, we asked observers to report the orientation of two concentric C-stimuli. Contrary to the predictions of their model, observers' recognition accuracy was worse for the inner C-stimulus. In addition, we demonstrated that the stimulus paradigm used by Harrison and Bex has a crucial confounding factor, eccentricity, which limits its usage to a very narrow range of stimulus parameters. Nevertheless, reporting the orientations of both C-stimuli in this paradigm proved very useful in pitting different crowding models against each other. Specifically, we tested deterministic and probabilistic versions of averaging, substitution, and attentional resolution models as well as the texture synthesis model. None of the models alone was able to explain the entire set of data. Based on these findings, we discuss whether the explanations of crowding can (should) be unified.

A Unifying Model of Orientation Crowding in Peripheral Vision

Peripheral vision is fundamentally limited not by the visibility of features, but by the spacing between them [1]. When too close together, visual features can become "crowded" and perceptually indistinguishable. Crowding interferes with basic tasks such as letter and face identification and thus informs our understanding of object recognition breakdown in peripheral vision [2]. Multiple proposals have attempted to explain crowding [3], and each is supported by compelling psychophysical and neuroimaging data [4-6] that are incompatible with competing proposals. In general, perceptual failures have variously been attributed to the averaging of nearby visual signals [7-10], confusion between target and distractor elements [11, 12], and a limited resolution of visual spatial attention [13]. Here we introduce a psychophysical paradigm that allows systematic study of crowded perception within the orientation domain, and we present a unifying computational model of crowding phenomena that reconciles conflicting explanations. Our results show that our single measure produces a variety of perceptual errors that are reported across the crowding literature. Critically, a simple model of the responses of populations of orientation-selective visual neurons accurately predicts all perceptual errors. We thus provide a unifying mechanistic explanation for orientation crowding in peripheral vision. Our simple model accounts for several perceptual phenomena produced by crowding of orientation and raises the possibility that multiple classes of object recognition failures in peripheral vision can be accounted for by a single mechanism.

Crowding in Visual Working Memory Reveals Its Spatial Resolution and the Nature of Its Representations

Spatial resolution fundamentally limits any image representation. Although this limit has been extensively investigated for perceptual representations by assessing how neighboring flankers degrade the perception of a peripheral target with visual crowding, the corresponding limit for representations held in visual working memory (VWM) is unknown. In the present study, we evoked crowding in VWM and directly compared resolution in VWM and perception. Remarkably, the spatial resolution of VWM proved to be no worse than that of perception. However, mixture modeling of errors caused by crowding revealed the qualitatively distinct nature of these representations. Perceptual crowding errors arose from both increased imprecision in target representations and substitution of flankers for targets. By contrast, VWM crowding errors arose exclusively from substitutions, which suggests that VWM transforms analog perceptual representations into discrete items. Thus, although perception and VWM share a common resolution limit, exceeding this limit reveals distinct mechanisms for perceiving images and holding them in mind.

Substitution and pooling in visual crowding induced by similar and dissimilar distractors

Visual crowding refers to a phenomenon whereby objects that appear in the periphery of the visual field are more difficult to identify when embedded within clutter. Pooling models assert that crowding results from an obligatory averaging or other combination of target and distractor features that occurs prior to awareness. One well-known manifestation of pooling is feature averaging, with which the features of target and nontarget stimuli are combined at an early stage of visual processing. Conversely, substitution models assert that crowding results from binding a target and nearby distractors to incorrect spatial locations. Recent evidence suggests that substitution predominates when target-flanker feature similarity is low, but it is unclear whether averaging or substitution best explains crowding when similarity is high. Here, we examined participants' orientation report errors for targets crowded by similar or dissimilar flankers. In two experiments, we found evidence inconsistent with feature averaging regardless of target-flanker similarity. However, the observed data could be accommodated by a probabilistic substitution model in which participants occasionally "swap" a target for a distractor. Thus, we conclude that-at least for the displays used here-crowding likely results from a probabilistic substitution of targets and distractors, regardless of target-distractor feature similarity.

Crowding, grouping, and object recognition: A matter of appearance

In crowding, the perception of a target strongly deteriorates when neighboring elements are presented. Crowding is usually assumed to have the following characteristics. (a) Crowding is determined only by nearby elements within a restricted region around the target (Bouma's law). (b) Increasing the number of flankers can only deteriorate performance. (c) Target-flanker interference is feature-specific. These characteristics are usually explained by pooling models, which are well in the spirit of classic models of object recognition. In this review, we summarize recent findings showing that crowding is not determined by the above characteristics, thus, challenging most models of crowding. We propose that the spatial configuration across the entire visual field determines crowding. Only when one understands how all elements of a visual scene group with each other, can one determine crowding strength. We put forward the hypothesis that appearance (i.e., how stimuli look) is a good predictor for crowding, because both crowding and appearance reflect the output of recurrent processing rather than interactions during the initial phase of visual processing.

Electrophysiological evidence for failures of item individuation in crowded visual displays

Visual perception is strongly impaired when peripheral targets are surrounded by nearby distractors, a phenomenon known as visual crowding. One common behavioral signature of visual crowding is an increased tendency for observers to mistakenly report the features of nearby distractors instead of the target item. Here, our goal was to distinguish between two possible explanations of such substitution errors. On the one hand, crowding may have its effects after the deployment of attention toward-and individuation of-targets and flankers, such that multiple individuated perceptual representations compete to guide the behavioral response. On the other hand, crowding may prevent the individuation of closely spaced stimuli, thereby reducing the number of apprehended items. We attempted to distinguish these alternatives using the N2pc, an ERP that has been shown to track the deployment of spatial attention and index the number of individuated items within a hemifield. N2pc amplitude increased monotonically with set size in uncrowded displays, but this set size effect was abolished in crowded visual displays. Moreover, these crowding-induced declines in N2pc amplitude predicted individual differences in the rate of substitution errors. Thus, crowding-induced confusions between targets and distractors may be a consequence of failures to individuate target and distractor stimuli during early stages of visual selection.