Objects with reduced visibility still contribute to size averaging

The anisotropic field of ensemble coding

Human observers can accurately estimate statistical summaries from an ensemble of multiple stimuli, including the average size, hue, and direction of motion. The efficiency and speed with which statistical summaries are extracted suggest an automatic mechanism of ensemble coding that operates beyond the capacity limits of attention and memory. However, the extent to which ensemble coding reflects a truly parallel and holistic mode of processing or a non-uniform and biased integration of multiple items is still under debate. In the present work, we used a technique, based on a Spatial Weighted Average Model (SWM), to recover the spatial profile of weights with which individual stimuli contribute to the estimated average during mean size adjustment tasks. In a series of experiments, we derived two-dimensional SWM maps for ensembles presented at different retinal locations, with different degrees of dispersion and under different attentional demands. Our findings revealed strong spatial anisotropies and leftward biases in ensemble coding that were organized in retinotopic reference frames and persisted under attentional manipulations. These results demonstrate an anisotropic spatial contribution to ensemble coding that could be mediated by the differential activation of the two hemispheres during spatial processing and scene encoding.

Feature-specificity in visual statistical summary processing

Visual statistical summary processing enables people to extract the average feature of a set of items rapidly and accurately. Previous studies have demonstrated independent mechanisms for summarizing low (e.g. color, orientation) and high-level (facial identity, emotion) visual information. However, no study to date has conclusively determined whether there are feature-specific summarization mechanisms for low-level features or whether there are low-level, feature agnostic summarization mechanisms. To address this issue, we asked participants to report either the average orientation or the average size from a set of lines where both features varied. Participants completed these tasks either in single-task or mixed-task conditions; in the latter, successful performance required extraction of both summaries concurrently. If there were feature-specific summarization mechanisms that could operate in parallel, then errors in mean size and mean orientation tasks should be independent, in both single and mixed task conditions. On the other hand, a central domain-general mechanism for low-level summarization would imply a correlation between errors for both features and greater error in the mixed than single task trials. In Experiment 1, we found that there was no correlation between the mean size and mean orientation errors and performance was similar across single and mixed-task conditions, suggesting that there may be independent summarization mechanisms for size and orientation features. To further test the feature-specificity account, in Experiment 2 and 3 (with mask), we manipulated the display duration to determine whether there were any differences in the summarization of earlier (orientation) vs. later (size) features. While these experiments replicated the pattern of results observed in Experiment 1, at shorter display durations, no differences emerged across features. We argue that our data is consistent with independent, multi-level feature-specific statistical summary mechanisms for low-level visual features.

Distributed attention model of perceptual averaging

The visual system efficiently processes complex and redundant information in a scene despite its limited capacity. One strategy for coping with the complexity and redundancy of a scene is to summarize it by using average information. However, despite its importance, the mechanism of averaging is not well understood. Here, a distributed attention model of averaging is proposed. Human percept for an object can be disturbed by various sources of internal noise, which can occur either before (early noise) or after (late noise) forming an ensemble perception. The model assumes these noises and reflects noise cancellation by averaging multiple items. The model predicts increased precision for more items with decelerated increments for large set-sizes resulting from late noise. Importantly, the model incorporates mechanisms of attention, which modulate each item's contribution to the averaging process. The attention in the model also results in saturation of performance increments for small set-sizes because the amount of attention allocated to each item is greater for small set-sizes than for large set-sizes. To evaluate the proposed model, a psychophysical experiment was conducted in which observers' ability to discriminate average sizes of two displays was measured. The observers' averaging performance increased at a decreasing rate with small set-sizes and it approached an asymptote for large set-sizes. The model accurately predicted the observed pattern of data. It provides a theoretical framework for interpreting behavioral data and leads to an understanding of the characteristics of ensemble perception.

Perceptual averaging of facial expressions requires visual awareness and attention

Humans, as highly social animals, are regularly exposed to the faces of conspecifics-often more than one at a time. This feature of social living is important for understanding face perception, not just because it means that information from faces is available in bulk, but also because it changes the way individuals are perceived. For instance, when two faces are seen nearby one another, they tend to look like each other. This phenomenon of perceptual averaging is robust when both faces are seen and attended. But in everyday life, some faces may not receive the full benefit of attention, or they may not be visible at all. We evaluated whether perceptual averaging of relatively complex and simple information on faces, including facial expression and head orientation, can still occur even in these circumstances. In particular, we used object-substitution masking (OSM) and a dual-task designed to disrupt visual awareness and attention, respectively, during evaluations of briefly presented face pairs. Disruptions of awareness or attention eliminated averaging of facial expression, whereas orientation averaging persisted in spite of these challenges. These results demonstrate boundary conditions for the process of perceptual averaging. More generally, they provide insight into how the visual system processes multitudes of objects, both simple and complex, both with and without attention and awareness.

Differential hemispheric and visual stream contributions to ensemble coding of crowd emotion

In crowds, where scrutinizing individual facial expressions is inefficient, humans can make snap judgments about the prevailing mood by reading "crowd emotion". We investigated how the brain accomplishes this feat in a set of behavioral and fMRI studies. Participants were asked to either avoid or approach one of two crowds of faces presented in the left and right visual hemifields. Perception of crowd emotion was improved when crowd stimuli contained goal-congruent cues and was highly lateralized to the right hemisphere. The dorsal visual stream was preferentially activated in crowd emotion processing, with activity in the intraparietal sulcus and superior frontal gyrus predicting perceptual accuracy for crowd emotion perception, whereas activity in the fusiform cortex in the ventral stream predicted better perception of individual facial expressions. Our findings thus reveal significant behavioral differences and differential involvement of the hemispheres and the major visual streams in reading crowd versus individual face expressions.

Ensemble Perception

To understand visual consciousness, we must understand how the brain represents ensembles of objects at many levels of perceptual analysis. Ensemble perception refers to the visual system's ability to extract summary statistical information from groups of similar objects-often in a brief glance. It defines foundational limits on cognition, memory, and behavior. In this review, we provide an operational definition of ensemble perception and demonstrate that ensemble perception spans across multiple levels of visual analysis, incorporating both low-level visual features and high-level social information. Further, we investigate the functional usefulness of ensemble perception and its efficiency, and we consider possible physiological and cognitive mechanisms that underlie an individual's ability to make accurate and rapid assessments of crowds of objects.

Object-substitution masking weakens but does not eliminate shape interactions

At any moment, some objects in the environment are seen clearly, whereas others go unnoticed. Whether or not these gaps in awareness are actually problematic may depend on the extent that information about unseen objects is lost. Determining when and how visual awareness and visual processing become linked is thus of great importance. Previous research using object-substitution masking (OSM) demonstrated that relatively simple visual features, such as size or orientation, are still processed even when they are not visible. Yet this does not appear to be the case for more complex features like faces. This suggests that, during OSM, disruptions of visual processing and awareness may tend to co-occur beginning at some intermediate stage along the ventral pathway. We tested this hypothesis by evaluating the extent to which OSM disrupted the perception and processing of two-dimensional objects. Specifically, we evaluated whether an unseen shape's aspect ratio would influence the appearance of another shape that was briefly visible nearby. As expected, the aspect ratios of two shapes appeared to be more similar to each other when both were visible. This averaging effect was weakened, but not eliminated, when one ellipse in each pair received OSM. These shape interactions persisted even when one ellipse from each pair was invisible. When combined with previous work, these results suggest that during object-substitution masking, disruptions of visual processing tend to strengthen with increases in stimulus complexity, becoming more tightly bound to the mechanisms of visual awareness at intermediate stages of visual analysis.

Perceptual Averaging in Individuals with Autism Spectrum Disorder

There is mounting evidence that observers rely on statistical summaries of visual information to maintain stable and coherent perception. Sensitivity to the mean (or other prototypical value) of a visual feature (e.g., mean size) appears to be a pervasive process in human visual perception. Previous studies in individuals diagnosed with Autism Spectrum Disorder (ASD) have uncovered characteristic patterns of visual processing that suggest they may rely more on enhanced local representations of individual objects instead of computing such perceptual averages. To further explore the fundamental nature of abstract statistical representation in visual perception, we investigated perceptual averaging of mean size in a group of 12 high-functioning individuals diagnosed with ASD using simplified versions of two identification and adaptation tasks that elicited characteristic perceptual averaging effects in a control group of neurotypical participants. In Experiment 1, participants performed with above chance accuracy in recalling the mean size of a set of circles (mean task) despite poor accuracy in recalling individual circle sizes (member task). In Experiment 2, their judgments of single circle size were biased by mean size adaptation. Overall, these results suggest that individuals with ASD perceptually average information about sets of objects in the surrounding environment. Our results underscore the fundamental nature of perceptual averaging in vision, and further our understanding of how autistic individuals make sense of the external environment.

Rapid proportion comparison with spatial arrays of frequently used meaningful visual symbols

It has been shown that when two arrays of Arabic numerals were briefly presented, observers could accurately indicate which array contained the larger number of a target numeral. This study investigated whether this rapid proportion comparison can be extended to other meaningful symbols that share some of notable properties of Arabic numerals. We tested mainly several Japanese Kanji letters, each of which represents a meaning and can work as a word. Using physically identical stimulus sets that could be interpreted as different types of letters, Experiment 1 first confirmed the rapid proportion comparison with Arabic numerals for Japanese participants. Experiment 2 showed that the rapid proportion comparison can be extended to Kanji numerals. Experiment 3 successfully demonstrated that rapid proportion judgments can be found with non-quantitative Kanji letters that are used frequently. Experiment 4 further demonstrated the rapid proportion comparison with frequently used meaningful non-letter symbols (gender icons). The rapid processing cannot be attributed to fluent processing of familiar items, because it was not found with familiar phonograms (Japanese Kana letters). These findings suggest that the rapid proportion comparison can be commonly found with frequently used meaningful symbols, even though their meaning is not relevant to the task.

Processing of Individual Items during Ensemble Coding of Facial Expressions

There is growing evidence that human observers are able to extract the mean emotion or other type of information from a set of faces. The most intriguing aspect of this phenomenon is that observers often fail to identify or form a representation for individual faces in a face set. However, most of these results were based on judgments under limited processing resource. We examined a wider range of exposure time and observed how the relationship between the extraction of a mean and representation of individual facial expressions would change. The results showed that with an exposure time of 50 ms for the faces, observers were more sensitive to mean representation over individual representation, replicating the typical findings in the literature. With longer exposure time, however, observers were able to extract both individual and mean representation more accurately. Furthermore, diffusion model analysis revealed that the mean representation is also more prone to suffer from the noise accumulated in redundant processing time and leads to a more conservative decision bias, whereas individual representations seem more resistant to this noise. Results suggest that the encoding of emotional information from multiple faces may take two forms: single face processing and crowd face processing.

A closer look at four-dot masking of a foveated target

Four-dot masking with a common onset mask was recently demonstrated in a fully attended and foveated target (Filmer, Mattingley & Dux, 2015). Here, we replicate and extend this finding by directly comparing a four-dot mask with an annulus mask while probing masking as a function of mask duration, and target-mask separation. Our results suggest that while an annulus mask operates via spatially local contour interactions, a four-dot mask operates through spatially global mechanisms. We also measure how the visual system’s representation of an oriented bar is impacted by a four-dot mask, and find that masking here does not degrade the precision of perceived targets, but instead appears to be driven exclusively by rendering the target completely invisible.

Statistical Properties Demand as Much Attention as Object Features

Recent studies have argued that the statistical properties of a set of visual items can be extracted with little or even no cost. In the present study, observers (N = 188) performed a color task and an orientation task, and the attention effect was measured as the advantage of pre-cueing one of the two tasks. The color and orientation tasks required participants to report either an object feature or the mean of a 4×4 array (i.e., statistical property). The pre-cueing advantages were approximately equal regardless of the nature of the tasks (object features vs. statistical properties), providing evidence that statistical properties are not perceived with zero cost, but demand as much attention as object features.

Characterizing ensemble statistics: mean size is represented across multiple frames of reference

The visual system represents the overall statistical, not individual, properties of sets. Here we tested the spatial nature of ensemble statistics. We used a mean-size adaptation paradigm (Corbett et al. in Visual Cognition, 20, 211-231, 2012) to examine whether average size is encoded in multiple reference frames. We adapted observers to patches of small- and large-sized dots in opposite regions of the display (left/right or top/bottom) and then tested their perceptions of the sizes of single test dots presented in regions that corresponded to retinotopic, spatiotopic, and hemispheric coordinates within the adapting displays. We observed retinotopic, spatiotopic, and hemispheric adaptation aftereffects, such that participants perceived a test dot as being larger when it was presented in the area adapted to the patch of small dots than when it was presented in the area adapted to large dots. This aftereffect also transferred between eyes. Our results demonstrate that mean size is represented across multiple spatial frames of reference, supporting the proposal that ensemble statistics play a fundamental role in maintaining perceptual stability.

Statistical extraction affects visually guided action

The visual system summarizes average properties of ensembles of similar objects. We demonstrated an adaptation aftereffect of one such property, mean size, suggesting it is encoded along a single visual dimension (Corbett, et al., 2012), in a similar manner as basic stimulus properties like orientation and direction of motion. To further explore the fundamental nature of ensemble encoding, here we mapped the evolution of mean size adaptation over the course of visually guided grasping. Participants adapted to two sets of dots with different mean sizes. After adaptation, two test dots replaced the adapting sets. Participants first reached to one of these dots, and then judged whether it was larger or smaller than the opposite dot. Grip apertures were inversely dependent on the average dot size of the preceding adapting patch during the early phase of movements, and this aftereffect dissipated as reaches neared the target. Interestingly, perceptual judgments still showed a marked aftereffect, even though they were made after grasping was completed more-or-less veridically. This effect of mean size adaptation on early visually guided kinematics provides novel evidence that mean size is encoded fundamentally in both perception and action domains, and suggests that ensemble statistics not only influence our perceptions of individual objects but can also affect our physical interactions with the external environment.

Global statistical regularities modulate the speed of visual search in patients with focal attentional deficits

There is growing evidence that the statistical properties of ensembles of similar objects are processed in a qualitatively different manner than the characteristics of individual items. It has recently been proposed that these types of perceptual statistical representations are part of a strategy to complement focused attention in order to circumvent the visual system’s limited capacity to represent more than a few individual objects in detail. Previous studies have demonstrated that patients with attentional deficits are nonetheless sensitive to these sorts of statistical representations. Here, we examined how such global representations may function to aid patients in overcoming focal attentional limitations by manipulating the statistical regularity of a visual scene while patients performed a search task. Three patients previously diagnosed with visual neglect searched for a target Gabor tilted to the left or right of vertical in displays of horizontal distractor Gabors. Although the local sizes of the distractors changed on every trial, the mean size remained stable for several trials. Patients made faster correct responses to targets in neglected regions of the visual field when global statistics remained constant over several trials, similar to age-matched controls. Given neglect patients’ attentional deficits, these results suggest that stable perceptual representations of global statistics can establish a context to speed search without the need to represent individual elements in detail.

Size averaging of irrelevant stimuli cannot be prevented

Research suggests that subjects can compute the mean size of two sets of interspersed objects concurrently, but that doing so incurs a cost of dividing attention across the two sets. Alternatively, costs may arise from failing to exclude irrelevant items from the calculation of mean size. Here, we examined whether attention can be selectively deployed to prevent the inclusion of items from an irrelevant, concurrently displayed set in the computation of the relevant set's mean size. The results suggest that mean size is computed prior to the deployment of attention, failing to exclude processing of items that are irrelevant to the task. The influence of the irrelevant items is evident both with brief exposures of the set (200ms) and in a simultaneous judgment task with unlimited viewing time, suggesting that attention cannot be effectively deployed to facilitate selective averaging of the size of the relevant set. Size averaging appears to precede the deployment of selective attention, suggesting that it may be carried out automatically, without intention.

The whole is indeed more than the sum of its parts: perceptual averaging in the absence of individual item representation

We tested Ariely's (2001) proposal that the visual system represents the overall statistical properties of sets of objects against alternative accounts of rapid averaging involving sub-sampling strategies. In four experiments, observers could rapidly extract the mean size of a set of circles presented in an RSVP sequence, but could not reliably identify individual members. Experiment 1 contrasted performance on a member identification task with performance on a mean judgment task, and showed that the tasks could be dissociated based on whether the test probe was presented before or after the sequence, suggesting that member identification and mean judgment are subserved by different mechanisms. In Experiment 2, we confirmed that when given a choice between a probe corresponding to the mean size of the set and a foil corresponding to the mean of the smallest and largest items only, the former is preferred to the latter, even when observers are explicitly instructed to average only the smallest and largest items. Experiment 3 showed that a test item corresponding to the mean size of the set could be reliably discriminated from a foil but the largest item in the set, differing by an equivalent amount, could not. In Experiment 4, observers rejected test items dissimilar to the mean size of the set in a member identification task, favoring test items that corresponded to the mean of the set over items that were actually shown. These findings suggest that mean representation is accomplished without explicitly encoding individual items.