Variance aftereffect within and between sensory modalities for visual and auditory domains

We can grasp various features of the outside world using summary statistics efficiently. Among these statistics, variance is an index of information homogeneity or reliability. Previous research has shown that visual variance information in the context of spatial integration is encoded directly as a unique feature, and currently perceived variance can be distorted by that of the preceding stimuli. In this study, we focused on variance perception in temporal integration. We investigated whether any variance aftereffects occurred in visual size and auditory pitch. Furthermore, to examine the mechanism of cross-modal variance perception, we also investigated whether variance aftereffects occur between different modalities. Four experimental conditions (a combination of sensory modalities of adaptor and test: visual-to-visual, visual-to-auditory, auditory-to-auditory, and auditory-to-visual) were conducted. Participants observed a sequence of visual or auditory stimuli perturbed in size or pitch with certain variance and performed a variance classification task before and after the variance adaptation phase. We found that in visual size, within modality adaptation to small or large variance, resulted in a variance aftereffect, indicating that variance judgments are biased in the direction away from that of the adapting stimulus. In auditory pitch, within modality adaptation to small variance caused variance aftereffect. For cross-modal combinations, adaptation to small variance in visual size resulted in variance aftereffect. However, the effect was weak, and variance aftereffect did not occur in other conditions. These findings indicate that the variance information of sequentially presented stimuli is encoded independently in visual and auditory domains.

Mean orientation discrimination based on proximal stimuli

Ensemble perception refers to the ability to accurately and rapidly perceive summary statistical representations of specific features from a group of similar objects. However, the specific type of representation involved in this perception within a three-dimensional (3-D) environment remains unclear. In the context of perspective viewing with stereopsis, distal stimuli can be projected onto the retina as different forms of proximal stimuli based on their distances, despite sharing similar properties, such as object size and spatial frequency. This study aimed to investigate the effects of distal and proximal stimuli on the perception of summary statistical information related to orientation. In our experiment, we presented multiple Gabor patches in a stereoscopic environment, allowing us to measure the discrimination threshold of the mean orientation. The object size and spatial frequency were fixed for all patches regardless of depth. However, the physical angular size and absolute spatial frequency covaried with the depth. The results revealed the threshold elevation with depth expansion, especially when the patches formed two clusters at near and far distances, leading to large variations in their retinotopic representations. This finding indicates a minor contribution of similarity of the distal stimuli. Subsequent experiments demonstrated that the variability in physical angular size of the patches significantly influenced the threshold elevation in contrast to that of binocular disparity and absolute spatial frequency. These findings highlight the critical role of physical angular size variability in perceiving mean orientations within the 3-D space.

Interpersonal prior information informs ensemble coding through the co-representation process

Humans have the ability to rapidly extract summary statistics from object groupings through a specific capability known as ensemble coding. Previous literature has reported that this ability can become biased by prior perceptual experiences at the individual level. However, it remains unknown whether interpersonal prior information could also bias ensemble perception through a co-representation process. Experiment 1 found that participants' summary estimations were biased toward their co-actor's stimuli. Experiment 2 confirmed a causal relationship between the bias effect and the co-representation process by showing a reduction in biased estimation after pairing participants with an out-group partner. These findings extend the sources of prior information exploited by humans during perceptual average from individual-level information (i.e., self-tasks) to interpersonal-level information (i.e., co-actor's tasks). More specifically, interpersonal prior information is shown to act in a top-down and implicit manner, biasing ensemble perception.

The functional role of spatial anisotropies in ensemble perception

BACKGROUND

The human brain can rapidly represent sets of similar stimuli by their ensemble summary statistics, like the average orientation or size. Classic models assume that ensemble statistics are computed by integrating all elements with equal weight. Challenging this view, here, we show that ensemble statistics are estimated by combining parafoveal and foveal statistics in proportion to their reliability. In a series of experiments, observers reproduced the average orientation of an ensemble of stimuli under varying levels of visual uncertainty.

RESULTS

Ensemble statistics were affected by multiple spatial biases, in particular, a strong and persistent bias towards the center of the visual field. This bias, evident in the majority of subjects and in all experiments, scaled with uncertainty: the higher the uncertainty in the ensemble statistics, the larger the bias towards the element shown at the fovea.

CONCLUSION

Our findings indicate that ensemble perception cannot be explained by simple uniform pooling. The visual system weights information anisotropically from both the parafovea and the fovea, taking the intrinsic spatial anisotropies of vision into account to compensate for visual uncertainty.

Outlier rejection in the process of pooling

Ensemble perception allows our visual system to process large amounts of information efficiently by summarizing its statistical properties. A key aspect of ensemble perception is the devaluation of outlying elements, which leads to more informative summary statistics with reduced variance and a more representative mean. However, the mechanisms underlying this outlier rejection process are not well understood. One possibility is that outliers are selectively excluded before summarization. To test this, we investigated whether only weaker items were excluded from averaging. We manipulated the encoding strength of items in a display by changing the emotional intensities of faces, the spatial location of emotional outliers, and the spatial distribution of emotional faces. We found that the response to outliers varied depending on their location. Specifically, outliers were more likely to be excluded from averaging when presented in more peripheral regions, while their exclusion was partial in parafoveal regions. In other words, outlier rejection in ensemble processing is more flexible than the supposed rigid designation of weighting against outliers. Alternatively, the results fit well with hierarchically structured pooling, during which outliers are discounted more dynamically without positing any separate selective mechanism before summarization. We propose an explanation for outlier rejection in light of a recently proposed population response model of ensemble processing.

Estimating lighting direction in scenes with multiple objects

To recover the reflectance and shape of an object in a scene, the human visual system must account for the properties of the light illuminating the object. Here, we examine the extent to which multiple objects within a scene are utilised to estimate the direction of lighting in a scene. In Experiment 1, we presented participants with rendered scenes that contained 1, 9, or 25 unfamiliar blob-like objects and measured their capacity to discriminate whether a directional light source was left or right of the participants' vantage point. Trends reported for ensemble perception suggest that the number of utilised objects-and, consequently, discrimination sensitivity-would increase with set size. However, we find little indication that increasing the number of objects in a scene increased discrimination sensitivity. In Experiment 2, an equivalent noise analysis was used to measure participants' internal noise and the number of objects used to judge the average light source direction in a scene, finding that participants relied on 1 or 2 objects to make their judgement regardless of whether 9 or 25 objects were present. In Experiment 3, participants completed a shape identification task that required an implicit judgement of light source direction, rather than an explicit judgement as in Experiments 1 and 2. We find that sensitivity for identifying surface shape was comparable for scenes containing 1, 9, and 25 objects. Our results suggest that the visual system relied on a small number of objects to estimate the direction of lighting in our rendered scenes.

Three's a crowd: Fast ensemble perception of first impressions of trustworthiness

Trustworthiness impressions are fundamental social judgements with far-reaching consequences in many aspects of society, including criminal justice, leadership selection and partner preferences. Thus far, most research has focused on facial characteristics that make a face individually appear more or less trustworthy. However, in everyday life, faces are not always perceived in isolation but are often encountered in crowds. It has been proposed that we deal with the large amount of facial information in a group by extracting summary statistics of the crowd, a phenomenon called ensemble perception. Prior research showed that ensemble perception occurs for various facial features, such as emotional expression, facial identity, and attractiveness. Here, we investigated whether observers can integrate the level of trustworthiness from multiple faces to extract an average impression of the crowd. Across four studies, participants were presented with crowds of faces and were asked to report their average level of trustworthiness with an adjustment (Experiment 1) and a rating task (Experiments 2 and 3). Participants were able to extract an ensemble perception of trustworthiness impressions from multiple faces. Moreover, observers were able to form a summary statistic of trustworthiness impressions from a group of faces as quickly as 250 ms (Experiment 4). Taken together, these results demonstrate that ensemble perception can occur at the level of impressions of trustworthiness. Thus, these critical social judgements not only occur for individual faces but are also integrated into a unique ensemble impression of crowds. Our findings contribute to the development of a more ecological approach to the study of trust impressions, since they provide an understanding of trustworthiness judgements not only on an individual level, but on a much broader social group level. Furthermore, our results drive forward new theory because they demonstrate for the first time that ensemble representations cover a broad range of phenomena than previously recognized, including complex high-level facial trait judgements such as trustworthiness impressions.

Role of facial familiarity and emotional expression intensity in ensemble emotion perception

When looking at groups of people, we can extract information from the different faces to derive properties of the group, such as its average facial emotion, although how this average is computed remains a matter of debate. Here, we examined whether our participants' personal familiarity with the faces in the group, as well as the intensity of the facial expressions, biased ensemble perception. Participants judged the average emotional expression of ensembles of four different identities whose expressions depicted either neutral, angry, or happy emotions. For the angry and happy expressions, the intensity of the emotion could be either low (e.g., slightly happy) or high (very happy). When all the identities in the ensemble were unfamiliar, the presence of any high intensity emotional face biased ensemble perception towards its emotion. However, when a familiar face was present in the ensemble, perception was biased towards the familiar face's emotion regardless of its intensity. These findings reveal that how we perceive the average emotion of a group is influenced by both the emotional intensity and familiarity of the faces comprising the group, supporting the idea that different faces may be weighted differently in ensemble perception. These findings have important implications for the judgements we make about a group's overall emotional state may be biased by individuals within the group.

Efficient sensory encoding predicts robust averaging

Not every item in a stimulus ensemble equally contributes to the perceived ensemble average. Rather, items with feature values close to the ensemble mean (inlying items) contribute stronger compared to those items whose feature values are further away from the mean (outlying items). This nonuniform weighting process, named robust averaging, has been interpreted as evidence against an optimal integration of sensory information. Here, however, we show that robust averaging naturally emerges from an optimal integration process when sensory encoding is efficiently adapted to the ensemble statistics in the experiment. We demonstrate that such a model can accurately fit several existing datasets showing robust perceptual averaging in discriminating low-level stimulus features such as orientation. Across various feature domains, our model accurately predicts subjects' decision accuracy and nonuniform weighting profile, and both their dependency on the specific stimulus distribution in the experiments. Our results suggest that the human visual system forms efficient sensory representations on short time-scales to improve overall decision performance.

Variability of dot spread is overestimated

Previous research has demonstrated that individuals exhibit a tendency to overestimate the variability of both low-level features (e.g., color, orientation) and mid-level features (e.g., size) when items are presented dynamically in a sequential order, a finding we will refer to as the variability overestimation effect. Because previous research on this bias used sequential displays, an open question is whether the effect is due to a memory-related bias or a vision-related bias. To assess whether the bias would also be apparent with static, simultaneous displays, and to examine whether the bias generalizes to spatial properties, we tested participants' perception of the variability of a cluster of dots. Results showed a consistent overestimation bias: Participants judged the dots as being more spread than they actually were. The variability overestimation effect was observed when there were 10 or 20 dots but not when there were 50 dots. Taken together, the results of the current study contribute to the ensemble perception literature by providing evidence that simultaneously presented stimuli are also susceptible to the variability overestimation effect. The use of static displays further demonstrates that this bias is present in both dynamic and static contexts, suggesting an inherent bias existent in the human visual system. A potential theoretical account-boundary effect-is discussed as a potential underlying mechanism. Moreover, the present study has implications for common visual tasks carried out in real-world scenarios, such as a radiologist making judgments about distribution of calcification in breast cancer diagnoses.

The relationship between ensemble representations of facial information

People accurately evaluate various types of facial information (gaze direction, facial expression, facial identity, and gender) of multiple faces. Considering such varieties, summarizing abilities of facial information might vary depending on its type because it is either changeable (e.g., gaze direction and expression) or invariant (e.g., identity and gender). The current study investigated the relationship between the averaging abilities of facial information using an individual difference approach and a dual-task paradigm to understand the effect of facial information type on the ensemble coding of facial information. We conducted two online experiments on the relationship between the averaging abilities of facial expressions and gaze direction (Experiment 1) and those of facial expressions and gender (Experiment 2). Participants were asked to estimate the average of each piece of facial information in the first and second blocks (single task), respectively, and both sequentially in the third and fourth blocks (dual task). We found that most of the error autocorrelations of facial information were high, indicating high measurement reliability. Participants' abilities to average facial expressions were correlated with those to average gaze directions, but not with those to average gender information. That is, the ensemble processing of facial expressions is related to gaze directions, but not genders. These results suggest that ensemble representations of facial information regarding changeable properties differ from those of invariant ones.

Understanding Mood of the Crowd with Facial Expressions: Majority Judgment for Evaluation of Statistical Summary Perception

We intuitively perceive mood or collective information of facial expressions without much effort. Although it is known that statistical summarization occurs even for faces instantaneously, it might be hard to perceive precise summary statistics of facial expressions (i.e., using all of them equally) since recognition of them requires the binding of multiple features of a face. This study assessed which information is extracted from the crowd to understand mood. In a series of experiments, twelve individual faces with happy and neutral expressions (or angry and neutral expressions) were presented simultaneously, and participants reported which expression appeared more frequently. To perform this task correctly, participants must perceive precise distribution of facial expressions in the crowd. If participants could perceive ensembles based on every face instantaneously, expressions presented on more than half of the faces (in a single ensemble/trial) would have been identified as more frequently presented and the just noticeable difference would be small. The results showed that participants did not always report seeing emotional faces more frequently until much more emotional than neutral faces appeared, suggesting that facial expression ensembles were not perceived from all faces. Manipulating the presentation layout revealed that participants' judgments highly weight only a part of the faces in the center of the crowd regardless of their visual size. Moreover, individual differences in the precision of summary statistical perception were related to visual working memory. Based on these results, this study provides a speculative explanation of summary perception of real distinctive faces. (247 words).

How are local orientation signals pooled?

Visual perception is capable of pooling multiple local orientation signals into a single more accurate summary orientation. However, there is still a lack of systematic inquiry into which summary statistics are implemented in that process. Here, the task was to recognize in which direction, clockwise or counter-clockwise, the mean orientation of a set of randomly distributed Gabor patches (N = 1, 2, 4, and 8) was rotated from the implicit vertical. The mean orientation discrimination accuracy did not improve with the increase of the number N of elements in proportion to the square-root-N, as could be expected if noisy internal representations were arithmetically averaged. The Proportion of Informative Elements (PIE), defined as the percentage of elements having an orientation different from the vertical, also affected the discrimination precision, violating the arithmetic averaging rules. The decrease in the orientation discrimination precision with the increase of the PIE would suggest that the orientation pooling could be more adequately described by a quadratic or higher power mean. Thus, we parameterized the averaging process for the power parameter of the generalized mean formula. The results indicate that different pooling rules in different trials may apply, for example, the arithmetic mean in some and the maximal deviation rule in others. It is concluded that pooling of orientation information is a relatively inaccurate process for which different perceptual cues and their combination rules can be used.

The aftereffect of the ensemble average of facial expressions on subsequent facial expression recognition

An ensemble or statistical summary can be extracted from facial expressions presented in different spatial locations simultaneously. However, how such complicated objects are represented in the mind is not clear. It is known that the aftereffect of facial expressions, in which prolonged viewing of facial expressions biases the perception of subsequent facial expressions of the same category, occurs only when a visual representation is formed. Using this methodology, we examined whether an ensemble can be represented with visualized information. Experiment 1 revealed that the presentation of multiple facial expressions biased the perception of subsequent facial expressions to less happy as much as the presentation of a single face did. Experiment 2 compared the presentation of faces comprising strong and weak intensities of emotional expressions with an individual face as the adaptation stimulus. The results indicated that the perceptual biases were found after the presentation of four faces and a strong single face, but not after the weak single face presentation. Experiment 3 employed angry expressions, a distinct category from the test expression used as an adaptation stimulus; no aftereffect was observed. Finally, Experiment 4 clearly demonstrated the perceptual bias with a higher number of faces. Altogether, these results indicate that an ensemble average extracted from multiple faces leads to the perceptual bias, and this effect is similar in terms of its properties to that of a single face. This supports the idea that an ensemble of faces is represented with visualized information as a single face.

Multivariate summary of a complex scene

The current study investigated how people summarize and represent objects with multiple features to cope with the complexity due to the number of objects and feature dimensions. We presented a set of circles whose color and size were either correlated perfectly (r = 1) or not correlated at all (r = 0). Using a membership identification task, we found that participants formed a statistical representation that included information about conjunctions as well as each color and size dimensions. In addition, we found that participants represented different set boundaries depending on the correlation between features of a set. Lastly, a pair-matching task revealed that participants predicted one feature value from the other feature value based on the correlation between features of a set. Our findings suggest that people represent a multi-feature ensemble statistically as a multivariate feature distribution, which is an efficient strategy to cope with scene complexity.

Oversampling of minority categories drives misperceptions of group compositions

The ability to estimate proportions informs our immediate impressions of social environments (e.g., of the diversity of races or genders within a crowded room). This study examines how the distribution of attention during brief glances shapes estimates of group gender proportions. Performance-wise, subjects exhibit a canonical pattern of judgment errors: small proportions are overestimated while large values are underestimated. Subjects' eye movements at sub-second timescales reveal that these biases follow from a tendency to visually oversample members of the gender minority. Rates of oversampling dovetail with average levels of error magnitudes, response variability, and response times. Visual biases are thus associated with the inherent difficulty in estimating particular proportions. All results are replicated at a within-subjects level with non-human ensembles using natural scene stimuli; the observed attentional patterns and judgment biases are thus not exclusively guided by face-specific visual properties. Our results reveal the biased distribution of attention underlying typical judgment errors of group proportions.

Contribution of a common ability in average and variability judgments

People can make judgments about statistical properties of visual features within groups of objects, such as the average size, size variability, or numerosity of circles. Emerging from recent work is the view that these kinds of visual estimations, collectively dubbed ensemble perception, rely on independent abilities that are specific to the statistical property being estimated (mean, variance, range, numerosity). Here we revisit evidence for the claim that different statistical judgments (i.e., average and variability) for the same object feature are based on independent abilities. We tested a large sample of people, using a pre-registered open-ended sequential design to avoid ending up with weak evidence. We estimated the shared variance in ability across different ensemble judgments, with common constraints for the different tasks. We found that the abilities to judge the average size and the size variability for an array of circles are positively correlated, even after controlling for the ability to discriminate the size of single circles. Our results refute the idea that judgments of average and variability for the same object feature rely on completely independent abilities.

Ensemble coding of average length and average orientation are correlated

Viewers can summarize redundant features in groups of objects into an ensemble percept. There appears to be separate mechanisms underlying ensemble perception of low- and high-level visual features, but it is unclear whether ensemble perception of different low-level features is supported by common mechanisms. Yörük and Boduroglu, in Attention, Perception, & Psychophysics 82 (2020) 852-864, investigated whether length and orientation summarization tap common mechanisms by examining the correlation between errors on length- and orientation-averaging tasks and concluded that because they did not find any correlations, the two features are summarized by different, feature-specific mechanisms. However, their study was conducted with a small sample size and included sources of individual performance variance that may have diminished correlations. We report two studies that tested the correlation between performance in the length- and orientation-averaging tasks, with larger samples and modifications that sought to reduce the sources of variance. Study 1 used ensembles that varied in both feature dimensions and Study 2 used ensembles that only varied in the task-relevant dimension. Both studies showed that errors in length- and orientation-averaging are correlated, suggesting that ensemble perception of these low-level features is supported, at least to some extent, by a common ability.

Ensemble perception without attention depends upon attentional control settings

People are able to rapidly extract summary statistical information about common patterns, or ensembles, that may exist in a scene, such as repeated textures or colors. Here we examined the extent to which such an ensemble perception can occur in the absence of focal visual attention using a method that has some advantages over methods previously used to study the issue. In particular, we assessed the extent to which ensembles can be processed without attention by measuring the indirect effect of a to-be-ignored ensemble on judgments of an attended ensemble. The results show that ensembles outside the focus of attention do influence judgments of attended ensembles when the to-be-ignored ensemble contains summary statistics that match a sought-for target category. Thus, an attentional control setting for specific summary statistical information permits the processing of ensembles outside of focal attention, facilitating the rapid perception of visual scenes.

Variability leads to overestimation of mean summaries

Research on ensemble perception has shown that people can extract both mean and variance information, but much less is understand how these two different types of summaries interact with one another. Some research has argued that people are more erroneous in extracting the mean of displays that have greater variability. In all three experiments, we manipulated the variability in the displays. Participants reported the mean size of a set of circles (Experiment 1) and mean length of horizontally placed (Experiment 2a) and randomly oriented lines (Experiment 2b). In all experiments, we found that mean size estimations were more erroneous for higher than smaller variance displays. More critically, there was a tendency to overestimate the mean, driven by variance in both task-relevant and task-irrelevant features. We discuss these findings in relation to limitations in concurrent summarization ability and outlier discounting in ensemble perception.

Contributions of ensemble perception to outlier representation precision

It is known that the visual system can efficiently extract mean and variance information, facilitating the detection of outliers. However, no research to date has directly investigated whether ensemble perception mechanisms contribute to outlier representation precision. We specifically were interested in how the distinctiveness of outliers impacts their precision. Across two experiments, we compared how accurately viewers represented the orientation of spatial outliers that varied in distinctiveness and found that increased outlier distinctiveness resulted in greater precision. Based on comparisons of our data to simulations reflecting particular selective strategies, we eliminated the possibility that participants were selectively processing the outlier, at the expense of the ensemble. Thus, we argued that participants separately represented distinct outliers along with ensemble summaries of the remaining items in a display. We also found that outlier distinctiveness moderated the precision of how the remaining items were summarized. We discuss these findings in relation to computational capacity and constraints of ensemble perception mechanisms.

Perceived variability reflects the reliability of individual items

When confronted with many visual items, people can compute their variability accurately and rapidly, which facilitates efficient information processing and optimal decision making. However, how the visual system computes variability is still unclear. To investigate this, we implemented situations whereby estimates of variability based on several possible variability measures (e.g., range, standard deviation, and weighted standard deviation) could be differentiated, and then examined which best accounted for human variability perception. In three psychophysical experiments, participants watched two arrays of items with various orientations and judged which had more variable orientations. Results showed that perceived variability was most consistent with the weighted standard deviation based on the reliability of individual items. Specifically, participants gave less consideration to deviant orientations that were likely to be outliers, and greater consideration to salient orientations that were likely to be encoded precisely. This reliability-based weighted standard deviation suggests an efficient and flexible way of representing visual variability.

A method for detection of inattentional feature blindness

In ensemble displays, two principal factors determine the precision with which the mean value of some perceptual attribute, such as size and orientation, can be discriminated: inefficiency and representational noise of each element. Inefficiency is mainly caused by biased inference, or by inattentional (feature) blindness (i.e., some elements or their features are not processed). Here, we define inattentional feature blindness as an inability to perceive the value(s) of certain feature(s) of an object while the presence of the object itself may be registered. Separation of the effects of inattentional (feature) blindness and perceptual noise has escaped traditional analytic methods because of their trade-off effects on the slope of the psychometric discrimination function. Here, we propose a method that can separate the effects of inattentional feature blindness from that of the representational noise. The basic idea is to display a set of elements from which only one contains information relevant for solving the task, while all other elements are "dummies" carrying no useful information because they do not differ from the reference. If the single informative element goes unprocessed, the correct answer can only be given by a random guess. The guess rate can be modeled similarly to the lapse rate, traditionally represented by λ. As an illustration, we present evidence that the presence versus lack of inattentional feature blindness in orientation pooling depends on the feature types present in the display.

Set size and ensemble perception of numerical value

There is a growing body of research on ensemble perception, or our ability to form ensemble representations based on perceptual features for stimuli of varying levels of complexity, and more recently, on ensemble cognition, which refers to our ability to perceive higher-level properties of stimuli such as facial attractiveness or gaze direction. Less is known about our ability to form ensemble representations based on more abstract properties such as the semantic meaning associated with items in a scene. Previous work examining whether the meaning associated with digits can be incorporated into summary statistical representations suggests that numerical information from digit ensembles can be extracted rapidly, and likely using a parallel processing mechanism. Here, we further investigate whether participants can accurately generate summary representations of numerical value from digit sets and explore the effect of set size on their ability to do so, by comparing psychometric functions based on a numerical averaging task in which set size varied. Steeper slopes for ten- and seven-item compared to five-item digit sets provide evidence that displays with more digits yield more reliable discrimination between larger and smaller numerical averages. Additionally, consistent with previous reports, we observed a response bias such that participants were more likely to report that the numerical average was "greater than 5" for larger compared to smaller sets. Overall, our results contribute to evidence that ensemble representations for semantic attributes may be carried out via similar mechanisms as those reported for perceptual features.

Synergy between research on ensemble perception, data visualization, and statistics education: A tutorial review

In the age of big data, we are constantly inventing new data visualizations to consolidate massive amounts of numerical information into smaller and more digestible visual formats. These data visualizations use various visual features to convey quantitative information, such as spatial position in scatter plots, color saturation in heat maps, and area in dot maps. These data visualizations are typically composed of ensembles, or groups of related objects, that together convey information about a data set. Ensemble perception, or one's ability to perceive summary statistics from an ensemble, such as the mean, has been used as a foundation for understanding and explaining the effectiveness of certain data visualizations. However, research in data visualization has revealed some perceptual biases and conceptual difficulties people face when trying to utilize the information in these graphs. In this tutorial review, we will provide a broad overview of research conducted in ensemble perception, discuss how principles of ensemble encoding have been applied to the research in data visualization, and showcase the barriers graphs can pose to learning statistical concepts, using histograms as a specific example. The goal of this tutorial review is to highlight possible connections between three areas of research-ensemble perception, data visualization, and statistics education-and to encourage research in the practical applications of ensemble perception in solving real-world problems in statistics education.

Uncertainty-based overestimation in the perception of group actions

Individuals are adept at estimating average properties of group visual stimuli, even following brief presentations. In estimating the directional heading of walking human figures, judgments are biased in a peculiar manner: groups facing intermediate directions are perceived to be more leftward- or rightward-facing than actual averages. This effect was previously explained as a repulsive bias away from a central category boundary; groups along this boundary (directly facing the observer) are estimated with lower variability and with relatively greater accuracy. Here we show that: (i) the original effect replicates and is constant over time in a novel estimation task with persistent directional states; and, (ii) novel patterns of response variability and durations align with the entire range of overestimation. A simple model of additive errors proportional to viewer uncertainty matches the observed bias magnitudes. We furthermore show that the bias generalizes beyond approaching walkers with the use of rearward-facing walkers presented at a nonparallel angle. Overall, the recurring relation between bias and uncertainty is also consistent with top-down and post-perceptual causes of misestimation.

Do group ensemble statistics bias visual working memory for individual items? A registered replication of Brady and Alvarez (2011)

We performed a registered and precise replication of Experiment 1 reported in Brady and Alvarez (Psychological Science, 22, 384–392, 2011). The original experiment found that participants, who were asked to memorize the size of differently colored circles, reported the size of a probed circle biased toward the mean size of the same-colored group. Because our previous three unpublished replication attempts failed to find this effect, we powered the present registered replication using a Bayes Factor Design Analysis such that it provided compelling evidence regarding the presence or absence of the reported bias with a high probability, even under the assumption of smaller effect sizes. Thus, we recruited 663 participants through Amazon Mechanical Turk. We observed both a significant bias and strong Bayesian evidence in favor of the existence of a bias over the null hypothesis. Thus, our results can be considered a successful replication of the original findings, although with a considerably smaller effect size. We discuss the role of data quality when recruiting participants with Amazon Mechanical Turk. The present findings corroborate the idea that memory representations of individual objects are influenced by summary statistics.

Temporal bisection is influenced by ensemble statistics of the stimulus set

Although humans are well capable of precise time measurement, their duration judgments are nevertheless susceptible to temporal context. Previous research on temporal bisection has shown that duration comparisons are influenced by both stimulus spacing and ensemble statistics. However, theories proposed to account for bisection performance lack a plausible justification of how the effects of stimulus spacing and ensemble statistics are actually combined in temporal judgments. To explain the various contextual effects in temporal bisection, we develop a unified ensemble-distribution account (EDA), which assumes that the mean and variance of the duration set serve as a reference, rather than the short and long standards, in duration comparison. To validate this account, we conducted three experiments that varied the stimulus spacing (Experiment 1), the frequency of the probed durations (Experiment 2), and the variability of the probed durations (Experiment 3). The results revealed significant shifts of the bisection point in Experiments 1 and 2, and a change of the sensitivity of temporal judgments in Experiment 3-which were all well predicted by EDA. In fact, comparison of EDA to the extant prior accounts showed that using ensemble statistics can parsimoniously explain various stimulus set-related factors (e.g., spacing, frequency, variance) that influence temporal judgments.

Holistic ensemble perception

In a glance, observers can evaluate gist characteristics from crowds of faces, such as the average emotional tenor or the average family resemblance. Prior research suggests that high-level ensemble percepts rely on holistic and viewpoint-invariant information. However, it is also possible that feature-based analysis was sufficient to yield successful ensemble percepts in many situations. To confirm that ensemble percepts can be extracted holistically, we asked observers to report the average emotional valence of Mooney face crowds. Mooney faces are two-tone, shadow-defined images that cannot be recognized in a part-based manner. To recognize features in a Mooney face, one must first recognize the image as a face by processing it holistically. Across experiments, we demonstrated that observers successfully extracted the average emotional valence from crowds that were spatially distributed or viewed in a rapid temporal sequence. In a subsequent set of experiments, we maximized holistic processing by including only those Mooney faces that were difficult to recognize when inverted. Under these conditions, participants remained highly sensitive to the average emotional valence of Mooney face crowds. Taken together, these experiments provide evidence that ensemble perception can operate selectively on holistic representations of human faces, even when feature-based information is not readily available.

Task-driven and flexible mean judgment for heterogeneous luminance ensembles

Spatial averaging of luminances over a variegated region has been assumed in visual processes such as light adaptation, texture segmentation, and lightness scaling. Despite the importance of these processes, how mean brightness can be computed remains largely unknown. We investigated how accurately and precisely mean brightness can be compared for two briefly presented heterogeneous luminance arrays composed of different numbers of disks. The results demonstrated that mean brightness judgments can be made in a task-dependent and flexible fashion. Mean brightness judgments measured via the point of subjective equality (PSE) exhibited a consistent bias, suggesting that observers relied strongly on a subset of the disks (e.g., the highest- or lowest-luminance disks) in making their judgments. Moreover, the direction of the bias flexibly changed with the task requirements, even when the stimuli were completely the same. When asked to choose the brighter array, observers relied more on the highest-luminance disks. However, when asked to choose the darker array, observers relied more on the lowest-luminance disks. In contrast, when the task was the same, observers' judgments were almost immune to substantial changes in apparent contrast caused by changing the background luminance. Despite the bias in PSE, the mean brightness judgments were precise. The just-noticeable differences measured for multiple disks were similar to or even smaller than those for single disks, which suggested a benefit of averaging. These findings implicated flexible weighted averaging; that is, mean brightness can be judged efficiently by flexibly relying more on a few items that are relevant to the task.

Statistical summary representations of bound features

The visual system can efficiently summarize various lower-level and higher-level features of ensembles. However, no research to date has directly investigated how different features interact with each other within a single summary and whether people can efficiently report an integrated summary of two feature dimensions. In the first two experiments, we specifically investigated whether individuals can integrate spatial and size information to report a bound spatial summary, the center of mass (CoM), as efficiently as the centroid, which is devoid of size information. Both experiments revealed that viewers were equally accurate in extracting the centroid and the CoM, with the size distribution inadvertently affecting the centroid estimates. In the final experiment, we investigated whether encouraging observers to attend to individual item size would cause the centroid estimates to be biased toward the CoM. When item size was task-irrelevant, as in the centroid task, observers were able to selectively focus on spatial location, eliminating any impact from the size distribution. Taken together, these findings demonstrate that viewers are capable of extracting integrated summaries, possibly through a mechanism that allows them to represent the spatial distribution of sizes. We discuss possible mechanisms that may support the extraction of integrated summaries, and highlight the need to consider multilevel mechanisms extending beyond simple feature- and object-based mechanisms.

The role of category- and exemplar-specific experience in ensemble processing of objects

People can relatively easily report summary properties for ensembles of objects, suggesting that this information can enrich visual experience and increase the efficiency of perceptual processing. Here, we ask whether the ability to judge diversity within object arrays improves with experience. We surmised that ensemble judgments would be more accurate for commonly experienced objects, and perhaps even more for objects of expertise like faces. We also expected improvements in ensemble processing with practice with a novel category, and perhaps even more with repeated experience with specific exemplars. We compared the effect of experience on diversity judgments for arrays of objects, with participants being tested with either a small number of repeated exemplars or with a large number of exemplars from the same object category. To explore the role of more prolonged experience, we tested participants with completely novel objects (random blobs), with objects familiar at the category level (cars), and with objects with which observers are experts at subordinate-level recognition (faces). For objects that are novel, participants showed evidence of improved ability to distribute attention. In contrast, for object categories with long-term experience, i.e., faces and cars, performance improved during the experiment but not necessarily due to improved ensemble processing. Practice with specific exemplars did not result in better diversity judgments for all object categories. Considered together, these results suggest that ensemble processing improves with experience. However, experience operates rapidly, the role of experience does not rely on exemplar-level knowledge and may not benefit from subordinate-level expertise.

Feature-specificity in visual statistical summary processing

Visual statistical summary processing enables people to extract the average feature of a set of items rapidly and accurately. Previous studies have demonstrated independent mechanisms for summarizing low (e.g. color, orientation) and high-level (facial identity, emotion) visual information. However, no study to date has conclusively determined whether there are feature-specific summarization mechanisms for low-level features or whether there are low-level, feature agnostic summarization mechanisms. To address this issue, we asked participants to report either the average orientation or the average size from a set of lines where both features varied. Participants completed these tasks either in single-task or mixed-task conditions; in the latter, successful performance required extraction of both summaries concurrently. If there were feature-specific summarization mechanisms that could operate in parallel, then errors in mean size and mean orientation tasks should be independent, in both single and mixed task conditions. On the other hand, a central domain-general mechanism for low-level summarization would imply a correlation between errors for both features and greater error in the mixed than single task trials. In Experiment 1, we found that there was no correlation between the mean size and mean orientation errors and performance was similar across single and mixed-task conditions, suggesting that there may be independent summarization mechanisms for size and orientation features. To further test the feature-specificity account, in Experiment 2 and 3 (with mask), we manipulated the display duration to determine whether there were any differences in the summarization of earlier (orientation) vs. later (size) features. While these experiments replicated the pattern of results observed in Experiment 1, at shorter display durations, no differences emerged across features. We argue that our data is consistent with independent, multi-level feature-specific statistical summary mechanisms for low-level visual features.

Quality of average representation can be enhanced by refined individual items

Ensemble perception is efficient because it summarizes redundant and complex information. However, it loses the fine details of individual items during the averaging process. Such characteristics of ensemble perception are similar to those of coarse processing. Here, we tested whether extracting an average of a set was similar to coarse processing. To manipulate coarse processing, we used the fast flicker adaptation known as suppressing coarse information processed by the magnocellular pathway. We hypothesized that if computing the average of a set relied on coarse processing, the precision of an averaging task should decrease after adaptation compared to baseline (no-adaptation). Across experiments with various features (orientation in Experiment 1, size in Experiment 2, and facial expression in Experiment 3), we found that suppressing coarse information did not disrupt the performance of the averaging tasks. Rather, adaptation increased the precision of mean representation. The precision of mean representation might have increased because fine information was relatively enhanced after adaptation. Our results suggest that the quality of ensemble representation relies on that of individual items.

Value associations bias ensemble perception

Ensemble perception refers to awareness of average properties, e.g. size, of “noisy” elements that often comprise visual arrays in natural scenes. Here, we asked how ensemble perception might be influenced when some but not all array elements are associated with monetary reward. Previous studies show that reward associations can speed object processing, facilitate selection, and enhance working-memory maintenance, suggesting they may bias ensemble judgments. To investigate, participants reported the average element size of brief arrays of different-sized circles. In the learning phase, all circles had the same color, but different colors produced high or low performance-contingent rewards. Then, in an unrewarded test phase, arrays comprised three spatially inter-mixed subsets, each with a different color, including the high-reward color. In different trials, the mean size of the subset with the high-reward color was smaller, larger, or the same as the ensemble mean. Ensemble size estimates were significantly biased by the high-reward-associated subset, showing that value associations modulate ensemble perception. In the test phase of a second experiment, a pattern mask appeared immediately after array presentation to limit top-down processing. Not only was value-biasing eliminated, ensemble accuracy improved, suggesting that value associations distort consciously available ensemble representation via late high-level processing.

Roles of saliency and set size in ensemble averaging

Ensemble statistics are often thought of as a reliable impression of numerous items despite limited capacities to consciously represent each individual. However, whether all items equally contribute to ensemble summaries (e.g., mean) and whether they might be affected by known limited-capacity processes, such as focused attention, is still debated. We addressed these questions via a recently described "amplification effect," a systematic bias of perceived mean (e.g., average size) towards the more salient "tail" of a feature distribution (e.g., larger items). In our experiments, observers adjusted the mean orientation of sets of items varying in set size. We made some of the items more salient or less salient by changing their size. While the whole orientation distribution was fixed, the more salient subset could be shifted relative to the set mean or differ in range. We measured the bias away from the set mean and the standard deviation (SD) of errors, as it is known to reflect the physical range from which ensemble information is sampled. We found that bias and SD changes followed the shifts and range changes in salient subsets, providing evidence for amplification. However, these changes were weaker than those expected from sampling only salient items, suggesting that less salient items were also sampled. Importantly, the SD decreased as a function of set size, which is only possible if the number of sampled elements increased with set size. Overall, we conclude that orientation summary statistics are sampled from an entire ensemble and modulated by the amplification effect of attention.

Emotional judgments of scenes are influenced by unintentional averaging

Background

The visual system uses ensemble perception to summarize visual input across a variety of domains. This heuristic operates at multiple levels of vision, compressing information as basic as oriented lines or as complex as emotional faces. Given its pervasiveness, the ensemble unsurprisingly can influence how an individual item is perceived, and vice versa.

Methods

In the current experiments, we tested whether the perceived emotional valence of a single scene could be influenced by surrounding, simultaneously presented scenes. Observers first rated the emotional valence of a series of individual scenes. They then saw ensembles of the original images, presented in sets of four, and were cued to rate, for a second time, one of four.

Results

Results confirmed that the perceived emotional valence of the cued image was pulled toward the mean emotion of the surrounding ensemble on the majority of trials, even though the ensemble was task-irrelevant. Control experiments and analyses confirmed that the pull was driven by high-level, ensemble information.

Conclusion

We conclude that high-level ensemble information can influence how we perceive individual items in a crowd, even when working memory demands are low and the ensemble information is not directly task-relevant.

Ensemble perception and focused attention: Two different modes of visual processing to cope with limited capacity

The visual system has a limited capacity for dealing with complex and redundant information in a scene. Here, we propose that a distributed attention mode of processing is necessary for coping with this limit, together with a focused attention mode of processing. The distributed attention mode provides a statistical summary of a scene, whereas the focused attention mode provides relevant information for object recognition. In this paper, we claim that a distributed mode of processing is necessary because (1) averaging performance improves with increased set-sizes, (2) even unselected items are likely to contribute to averaging, and (3) the assumption of variable capacity limits in averaging over different set-sizes is not plausible. We then propose how the averaging process can access multiple items over the capacity limit of focused attention. The visual system can represent multiple items as population responses and read out relevant information using the two modes of attention. It can summarize population responses with a broad application of a Gaussian profile (i.e., distributed attention) and represent its peak as the mean. It can focus on relevant population responses with a narrow application of a Gaussian profile (i.e., focused attention) and select important information for object recognition. The two attention modes of processing provide a framework for incorporating two seemingly opposing fields of study (ensemble perception and selective attention) and a unified theory of a coping strategy with our limited capacity.

Adaptation to mean and variance: Interrelationships between mean and variance representations in orientation perception

When there are many visual items, the visual system could represent their summary statistics (e.g., mean, variance) to process them efficiently. Although many previous studies have investigated the mean or variance representation itself, a relationship between these two ensemble representations has not been investigated much. In this study, we tested the potential interaction between mean and variance representations by using a visual adaptation method. We reasoned that if mean and variance representations interact with each other, an adaptation aftereffect to either mean or variance would influence the perception of the other. Participants watched a sequence of orientation arrays containing a specific statistical property during the adaptation period. To produce an adaptation aftereffect specific to variance or mean, one property of the adaptor arrays (variance or mean) had a fixed value while the other property was randomly varied. After the adaptation, participants were asked to discriminate the property of the test array that was randomly varied during the adaptation. We found that the adaptation aftereffect of orientation variance influenced the sensitivity of mean orientation discrimination (Experiment 1), and that the adaptation aftereffect of mean orientation influenced the bias of orientation variance discrimination (Experiment 2). These results suggest that mean and variance representations do closely interact with each other. Considering that mean and variance reflect the representative value and dispersion of multiple items respectively, the interactions between mean and variance representations may reflect their complementary roles to summarize complex visual information effectively.

Tracking two pleasures

Can people track several pleasures? In everyday life, pleasing stimuli rarely appear in isolation. Yet, experiments on aesthetic pleasure usually present only one image at a time. Here, we ask whether people can reliably report the pleasure of either of two images seen in a single glimpse. Participants (N = 13 in the original; +25 in the preregistered replication) viewed 36 Open Affective Standardized Image Set (OASIS) images that span the entire range of pleasure and beauty. On each trial, the observer saw two images, side by side, for 200 ms. An arrow cue pointed, randomly, left, right, or bidirectionally. Left or right indicated which image (the target) to rate while ignoring the other (the distractor); bidirectional requested rating the combined pleasure of both images. In half the blocks, the cue came before the images (precuing). Otherwise, it came after (postcuing). Precuing allowed the observer to ignore the distractor, while postcuing demanded tracking both images. Finally, we obtained single-pleasure ratings for each image shown alone. Our replication confirms the original study. People have unbiased access to their felt pleasure from each image and the average of both. Furthermore, the variance of the observer’s report is similar whether reporting the pleasure of one image or the average pleasure of two. The undiminished variance for reports of the average pleasure of two images indicates either that the underlying pleasure variances are highly correlated, or, more likely, that the variance arises in the common reporting process. In brief, observers can faithfully track at least two visual pleasures.

Correcting "confusability regions" in face morphs

The visual system represents summary statistical information from a set of similar items, a phenomenon known as ensemble perception. In exploring various ensemble domains (e.g., orientation, color, facial expression), researchers have often employed the method of continuous report, in which observers select their responses from a gradually changing morph sequence. However, given their current implementation, some face morphs unintentionally introduce noise into the ensemble measurement. Specifically, some facial expressions on the morph wheel appear perceptually similar even though they are far apart in stimulus space. For instance, in a morph wheel of happy-sad-angry-happy expressions, an expression between happy and sad may not be discriminable from an expression between sad and angry. Without accounting for this confusability, observer ability will be underestimated. In the present experiments we accounted for this by delineating the perceptual confusability of morphs of multiple expressions. In a two-alternative forced choice task, eight observers were asked to discriminate between anchor images (36 in total) and all 360 facial expressions on the morph wheel. The results were visualized on a "confusability matrix," depicting the morphs most likely to be confused for one another. The matrix revealed multiple confusable images between distant expressions on the morph wheel. By accounting for these "confusability regions," we demonstrated a significant improvement in performance estimation on a set of independent ensemble data, suggesting that high-level ensemble abilities may be better than has been previously thought. We also provide an alternative computational approach that may be used to determine potentially confusable stimuli in a given morph space.

Reprint of "Investigating ensemble perception of emotions in autistic and typical children and adolescents"

Ensemble perception, the ability to assess automatically the summary of large amounts of information presented in visual scenes, is available early in typical development. This ability might be compromised in autistic children, who are thought to present limitations in maintaining summary statistics representations for the recent history of sensory input. Here we examined ensemble perception of facial emotional expressions in 35 autistic children, 30 age- and ability-matched typical children and 25 typical adults. Participants received three tasks: a) an ‘ensemble’ emotion discrimination task; b) a baseline (single-face) emotion discrimination task; and c) a facial expression identification task. Children performed worse than adults on all three tasks. Unexpectedly, autistic and typical children were, on average, indistinguishable in their precision and accuracy on all three tasks. Computational modelling suggested that, on average, autistic and typical children used ensemble-encoding strategies to a similar extent; but ensemble perception was related to non-verbal reasoning abilities in autistic but not in typical children. Eye-movement data also showed no group differences in the way children attended to the stimuli. Our combined findings suggest that the abilities of autistic and typical children for ensemble perception of emotions are comparable on average.

Ensemble perception of emotions in autistic and typical children and adolescents

Ensemble perception, the ability to assess automatically the summary of large amounts of information presented in visual scenes, is available early in typical development. This ability might be compromised in autistic children, who are thought to present limitations in maintaining summary statistics representations for the recent history of sensory input. Here we examined ensemble perception of facial emotional expressions in 35 autistic children, 30 age- and ability-matched typical children and 25 typical adults. Participants received three tasks: a) an 'ensemble' emotion discrimination task; b) a baseline (single-face) emotion discrimination task; and c) a facial expression identification task. Children performed worse than adults on all three tasks. Unexpectedly, autistic and typical children were, on average, indistinguishable in their precision and accuracy on all three tasks. Computational modelling suggested that, on average, autistic and typical children used ensemble-encoding strategies to a similar extent; but ensemble perception was related to non-verbal reasoning abilities in autistic but not in typical children. Eye-movement data also showed no group differences in the way children attended to the stimuli. Our combined findings suggest that the abilities of autistic and typical children for ensemble perception of emotions are comparable on average.