Spatial limitations in averaging social cues

The combination operation of grouping and ensemble coding for structured biological motion crowds in working memory

Massive studies have explored biological motion (BM) crowds processing for their remarkable social significance, primarily focused on uniformly distributed ones. However, real-world BM crowds often exhibit hierarchical structures rather than uniform arrangements. How such structured BM crowds are processed remains a subject of inquiry. This study investigates the representation of structured BM crowds in working memory (WM), recognizing the pivotal role WM plays in our social interactions involving BM. We propose the group-based ensemble hypothesis and test it through a member identification task. Participants were required to discern whether a presented BM belonged to a prior memory display of eight BM, each with distinct walking directions. Drawing on prominent Gestalt principles as organizational cues, we constructed structured groups within BM crowds by applying proximity and similarity cues in Experiments 1 and 2, respectively. In Experiment 3, we deliberately weakened the visibility of stimuli structures by increasing the similarity between subsets, probing the robustness of results. Consistently, our findings indicate that BM aligned with the mean direction of the subsets was more likely to be recognized as part of the memory stimuli. This suggests that WM inherently organizes structured BM crowds into separate ensembles based on organizational cues. In essence, our results illuminate the simultaneous operation of grouping and ensemble encoding mechanisms for BM crowds within WM.

What is the basis of ensemble subset selection?

The visual system can rapidly calculate the ensemble statistics of a set of objects; for example, people can easily estimate an average size of apples on a tree. To accomplish this, it is not always useful to summarize all the visual information. If there are various types of objects, the visual system should select a relevant subset: only apples, not leaves and branches. Here, we ask what kind of visual information makes a "good" ensemble that can be selectively attended to provide an accurate summary estimate. We tested three candidate representations: basic features, preattentive object files, and full-fledged bound objects. In four experiments, we presented a target and several distractors' sets of differently colored objects. We found that conditions where a target ensemble had at least one unique color (basic feature) provided ensemble averaging performance comparable to the baseline displays without distractors. When the target subset was defined as a conjunction of two colors or color-shape partly shared with distractors (so that they could be differentiated only as preattentive object files), subset averaging was also possible but less accurate than in the baseline and feature conditions. Finally, performance was very poor when the target subset was defined by an exact feature relationship, such as in the spatial conjunction of two colors (spatially bound object). Overall, these results suggest that distinguishable features and, to a lesser degree, preattentive object files can serve as the representational basis of ensemble selection, while bound objects cannot.

Estimating lighting direction in scenes with multiple objects

To recover the reflectance and shape of an object in a scene, the human visual system must account for the properties of the light illuminating the object. Here, we examine the extent to which multiple objects within a scene are utilised to estimate the direction of lighting in a scene. In Experiment 1, we presented participants with rendered scenes that contained 1, 9, or 25 unfamiliar blob-like objects and measured their capacity to discriminate whether a directional light source was left or right of the participants' vantage point. Trends reported for ensemble perception suggest that the number of utilised objects-and, consequently, discrimination sensitivity-would increase with set size. However, we find little indication that increasing the number of objects in a scene increased discrimination sensitivity. In Experiment 2, an equivalent noise analysis was used to measure participants' internal noise and the number of objects used to judge the average light source direction in a scene, finding that participants relied on 1 or 2 objects to make their judgement regardless of whether 9 or 25 objects were present. In Experiment 3, participants completed a shape identification task that required an implicit judgement of light source direction, rather than an explicit judgement as in Experiments 1 and 2. We find that sensitivity for identifying surface shape was comparable for scenes containing 1, 9, and 25 objects. Our results suggest that the visual system relied on a small number of objects to estimate the direction of lighting in our rendered scenes.

Understanding Mood of the Crowd with Facial Expressions: Majority Judgment for Evaluation of Statistical Summary Perception

We intuitively perceive mood or collective information of facial expressions without much effort. Although it is known that statistical summarization occurs even for faces instantaneously, it might be hard to perceive precise summary statistics of facial expressions (i.e., using all of them equally) since recognition of them requires the binding of multiple features of a face. This study assessed which information is extracted from the crowd to understand mood. In a series of experiments, twelve individual faces with happy and neutral expressions (or angry and neutral expressions) were presented simultaneously, and participants reported which expression appeared more frequently. To perform this task correctly, participants must perceive precise distribution of facial expressions in the crowd. If participants could perceive ensembles based on every face instantaneously, expressions presented on more than half of the faces (in a single ensemble/trial) would have been identified as more frequently presented and the just noticeable difference would be small. The results showed that participants did not always report seeing emotional faces more frequently until much more emotional than neutral faces appeared, suggesting that facial expression ensembles were not perceived from all faces. Manipulating the presentation layout revealed that participants' judgments highly weight only a part of the faces in the center of the crowd regardless of their visual size. Moreover, individual differences in the precision of summary statistical perception were related to visual working memory. Based on these results, this study provides a speculative explanation of summary perception of real distinctive faces. (247 words).

How are local orientation signals pooled?

Visual perception is capable of pooling multiple local orientation signals into a single more accurate summary orientation. However, there is still a lack of systematic inquiry into which summary statistics are implemented in that process. Here, the task was to recognize in which direction, clockwise or counter-clockwise, the mean orientation of a set of randomly distributed Gabor patches (N = 1, 2, 4, and 8) was rotated from the implicit vertical. The mean orientation discrimination accuracy did not improve with the increase of the number N of elements in proportion to the square-root-N, as could be expected if noisy internal representations were arithmetically averaged. The Proportion of Informative Elements (PIE), defined as the percentage of elements having an orientation different from the vertical, also affected the discrimination precision, violating the arithmetic averaging rules. The decrease in the orientation discrimination precision with the increase of the PIE would suggest that the orientation pooling could be more adequately described by a quadratic or higher power mean. Thus, we parameterized the averaging process for the power parameter of the generalized mean formula. The results indicate that different pooling rules in different trials may apply, for example, the arithmetic mean in some and the maximal deviation rule in others. It is concluded that pooling of orientation information is a relatively inaccurate process for which different perceptual cues and their combination rules can be used.

Person Perception, Meet People Perception: Exploring the Social Vision of Groups

Groups, teams, and collectives-people-are incredibly important to human behavior. People live in families, work in teams, and celebrate and mourn together in groups. Despite the huge variety of human group activity and its fundamental importance to human life, social-psychological research on person perception has overwhelmingly focused on its namesake, the person, rather than expanding to consider people perception. By looking to two unexpected partners, the vision sciences and organization behavior, we find emerging work that presents a path forward, building a foundation for understanding how people perceive other people. And yet this nascent field is missing critical insights that scholars of social vision might offer: specifically, for example, the chance to connect perception to behavior through the mediators of cognition and motivational processes. Here, we review emerging work across the vision and social sciences to extract core principles of people perception: efficiency, capacity, and complexity. We then consider complexity in more detail, focusing on how people perception modifies person-perception processes and enables the perception of group emergent properties as well as group dynamics. Finally, we use these principles to discuss findings and outline areas fruitful for future work. We hope that fellow scholars take up this people-perception call.

Holistic ensemble perception

In a glance, observers can evaluate gist characteristics from crowds of faces, such as the average emotional tenor or the average family resemblance. Prior research suggests that high-level ensemble percepts rely on holistic and viewpoint-invariant information. However, it is also possible that feature-based analysis was sufficient to yield successful ensemble percepts in many situations. To confirm that ensemble percepts can be extracted holistically, we asked observers to report the average emotional valence of Mooney face crowds. Mooney faces are two-tone, shadow-defined images that cannot be recognized in a part-based manner. To recognize features in a Mooney face, one must first recognize the image as a face by processing it holistically. Across experiments, we demonstrated that observers successfully extracted the average emotional valence from crowds that were spatially distributed or viewed in a rapid temporal sequence. In a subsequent set of experiments, we maximized holistic processing by including only those Mooney faces that were difficult to recognize when inverted. Under these conditions, participants remained highly sensitive to the average emotional valence of Mooney face crowds. Taken together, these experiments provide evidence that ensemble perception can operate selectively on holistic representations of human faces, even when feature-based information is not readily available.

Ensemble coding of crowd speed using biological motion

The accurate perception of human crowds is integral to social understanding and interaction. Previous studies have shown that observers are sensitive to several crowd characteristics such as average facial expression, gender, identity, joint attention, and heading direction. In two experiments, we examined ensemble perception of crowd speed using standard point-light walkers (PLW). Participants were asked to estimate the average speed of a crowd consisting of 12 figures moving at different speeds. In Experiment 1, trials of intact PLWs alternated with trials of scrambled PLWs with a viewing duration of 3 seconds. We found that ensemble processing of crowd speed could rely on local motion alone, although a globally intact configuration enhanced performance. In Experiment 2, observers estimated the average speed of intact-PLW crowds that were displayed at reduced viewing durations across five blocks of trials (between 2500 ms and 500 ms). Estimation of fast crowds was precise and accurate regardless of viewing duration, and we estimated that three to four walkers could still be integrated at 500 ms. For slow crowds, we found a systematic deterioration in performance as viewing time reduced, and performance at 500 ms could not be distinguished from a single-walker response strategy. Overall, our results suggest that rapid and accurate ensemble perception of crowd speed is possible, although sensitive to the precise speed range examined.

Gaze deflection reveals how gaze cueing is tuned to extract the mind behind the eyes

Suppose you are surreptitiously looking at someone, and then when they catch you staring at them, you immediately turn away. This is a social phenomenon that almost everyone experiences occasionally. In such experiences-which we will call gaze deflection-the "deflected" gaze is not directed at anything in particular but simply away from the other person. As such, this is a rare instance where we may turn to look in a direction without intending to look there specifically. Here we show that gaze cues are markedly less effective at orienting an observer's attention when they are seen as deflected in this way-even controlling for low-level visual properties. We conclude that gaze cueing is a sophisticated mental phenomenon: It is not merely driven by perceived eye or head motions but is rather well tuned to extract the "mind" behind the eyes.

Incidental learning of group trust: Predictive gaze cue matters

Human gaze is a subtle cue to deliver information and helps impression formation in social interactions. People automatically follow the gaze direction of others and shift their attention accordingly, as well as determine the trustworthiness of others based on the predictable validity of their gaze behavior, yet it remains unclear how this works at the collective level. Therefore, the current study is the first to explore the incidental learning of trust from a group’s gaze behavior. To simulate different patterns of perceiving collective information in real life, two ways of presenting group member gazes were used in the object categorization task, the simultaneous way in Experiment 1 and the sequential way in Experiment 3, and a sampling strategy was ruled out in Experiment 2. Converging findings in experiments demonstrated a typical gaze-cueing effect, and more importantly, the Predictive-valid group obtained more trust compared to the Predictive-invalid group. To enrich and expand the applicability of the incidental trust learning effect from gazes, the current study provides supportive evidence at the collective level, confirming that humans have an efficient capability to process gaze information of groups.

Independent and parallel visual processing of ensemble statistics: Evidence from dual tasks

The visual system can represent multiple objects in a compressed form of ensemble summary statistics (such as object numerosity, mean, and feature variance/range). Yet the relationships between the different types of visual statistics remain relatively unclear. Here, we tested whether two summaries (mean and numerosity, or mean and range) are calculated independently from each other and in parallel. Our participants performed dual tasks requiring a report about two summaries in each trial, and single tasks requiring a report about one of the summaries. We estimated trial-by-trial correlations between the precision of reports as well as correlations across observers. Both analyses showed the absence of correlations between different types of ensemble statistics, suggesting their independence. We also found no decrement (except that related to the order of report explained by memory retrieval) in performance in dual compared to single tasks, which suggests that two statistics of one ensemble can be processed in parallel.

Elucidating the Neural Representation and the Processing Dynamics of Face Ensembles

Extensive behavioral work has documented the ability of the human visual system to extract summary representations from face ensembles (e.g., the average identity of a crowd of faces). Yet, the nature of such representations, their underlying neural mechanisms, and their temporal dynamics await elucidation. Here, we examine summary representations of facial identity in human adults (of both sexes) with the aid of pattern analyses, as applied to EEG data, along with behavioral testing. Our findings confirm the ability of the visual system to form such representations both explicitly and implicitly (i.e., with or without the use of specific instructions). We show that summary representations, rather than individual ensemble constituents, can be decoded from neural signals elicited by ensemble perception, we describe the properties of such representations by appeal to multidimensional face space constructs, and we visualize their content through neural-based image reconstruction. Further, we show that the temporal profile of ensemble processing diverges systematically from that of single faces consistent with a slower, more gradual accumulation of perceptual information. Thus, our findings reveal the representational basis of ensemble processing, its fine-grained visual content, and its neural dynamics.SIGNIFICANCE STATEMENT Humans encounter groups of faces, or ensembles, in a variety of environments. Previous behavioral research has investigated how humans process face ensembles as well as the types of summary representations that can be derived from them, such as average emotion, gender, and identity. However, the neural mechanisms mediating these processes are unclear. Here, we demonstrate that ensemble representations, with different facial identity summaries, can be decoded and even visualized from neural data through multivariate analyses. These results provide, to our knowledge, the first detailed investigation into the status and the visual content of neural ensemble representations of faces. Further, the current findings shed light on the temporal dynamics of face ensembles and its relationship with single-face processing.

Perceiving gaze from head and eye rotations: An integrative challenge for children and adults

Gaze is an emergent visual feature. A person's gaze direction is perceived not just based on the rotation of their eyes, but also their head. At least among adults, this integrative process appears to be flexible such that one feature can be weighted more heavily than the other depending on the circumstances. Yet it is unclear how this weighting might vary across individuals or across development. When children engage emergent gaze, do they prioritize cues from the head and eyes similarly to adults? Is the perception of gaze among individuals with autism spectrum disorder (ASD) emergent, or is it reliant on a single feature? Sixty adults (M = 29.86 years-of-age), thirty-seven typically developing children and adolescents (M = 9.3 years-of-age; range = 7-15), and eighteen children with ASD (M = 9.72 years-of-age; range = 7-15) viewed faces with leftward, rightward, or direct head rotations in conjunction with leftward or rightward pupil rotations, and then indicated whether the face was looking leftward or rightward. All individuals, across development and ASD status, used head rotation to infer gaze direction, albeit with some individual differences. However, the use of pupil rotation was heavily dependent on age. Finally, children with ASD used pupil rotation significantly less than typically developing (TD) children when inferring gaze direction, even after accounting for age. Our approach provides a novel framework for understanding individual and group differences in gaze as it is actually perceived-as an emergent feature. Furthermore, this study begins to address an important gap in ASD literature, taking the first look at emergent gaze perception in this population.

Exaggerated groups: amplification in ensemble coding of temporal and spatial features

The human visual system represents summary statistical information (e.g. average) along many visual dimensions efficiently. While studies have indicated that approximately the square root of the number of items in a set are effectively integrated through this ensemble coding, how those samples are determined is still unknown. Here, we report that salient items are preferentially weighted over the other less salient items, by demonstrating that the perceived means of spatial (i.e. size) and temporal (i.e. flickering temporal frequency (TF)) features of the group of items are positively biased as the number of items in the group increases. This illusory 'amplification effect' was not the product of decision bias but of perceptual bias. Moreover, our visual search experiments with similar stimuli suggested that this amplification effect was due to attraction of visual attention to the salient items (i.e. large or high TF items). These results support the idea that summary statistical information is extracted from sets with an implicit preferential weighting towards salient items. Our study suggests that this saliency-based weighting may reflect a more optimal and efficient integration strategy for the extraction of spatio-temporal statistical information from the environment, and may thus be a basic principle of ensemble coding.

Mean emotion from multiple facial expressions can be extracted with limited attention: Evidence from visual ERPs

Human observers can readily extract the mean emotion from multiple faces shown briefly. However, it remains currently debated whether this ability depends on attention or not. To address this question, in this study, we recorded lateralized event-related brain potentials (i.e., N2pc and SPCN) to track covert shifts of spatial attention, while healthy adult participants discriminated the mean emotion of four faces shown in the periphery at an attended or unattended spatial location, using a cueing technique. As a control condition, they were asked to discriminate the emotional expression of a single face shown in the periphery. Analyses of saccade-free data showed that the mean emotion discrimination ability was above chance level but statistically undistinguishable between the attended and unattended location, suggesting that attention was not a pre-requisite for averaging. Interestingly, at the ERP level, covert shifts of spatial attention were captured by the N2pc and SPCN components. All together, these novel findings suggest that averaging multiple facial expressions shown in the periphery can operate with limited attention.

Perceiving crowd attention: Gaze following in human crowds with conflicting cues

People automatically redirect their visual attention by following others' gaze orientation, a phenomenon called "gaze following." This is an evolutionarily generated socio-cognitive process that provides people with information about their environments. Often, however, people in crowds can have rather different gaze orientations. This study investigated how gaze following occurs in situations with many conflicting gazes. In two experiments, we modified the gaze cueing paradigm to use a crowd rather than a single individual. Specifically, participants were presented with a group of human avatars with differing gaze orientations, and the target appeared randomly on the left or right side of a display. We found that (a) when a marked difference existed in the number of avatars with divergent gaze orientations, participants automatically followed the majority's gaze orientation, and (b) the strongest gaze cue effect occurred when all gazes shared the same orientation, with the response superiority of the majority's oriented location monotonically diminishing with the number of gazes with divergent orientations. These findings suggested that the majority rule plays a role in gaze following behavior when individuals are confronted with conflicting multigaze scenes, and that an increasing subgroup size appears to enlarge the strength of the gaze cueing effect.

Cross-cultural and hemispheric laterality effects on the ensemble coding of emotion in facial crowds

In many social situations, we make a snap judgment about crowds of people relying on their overall mood (termed "crowd emotion"). Although reading crowd emotion is critical for interpersonal dynamics, the sociocultural aspects of this process have not been explored. The current study examined how culture modulates the processing of crowd emotion in Korean and American observers. Korean and American (non-East Asian) participants were briefly presented with two groups of faces that were individually varying in emotional expressions and asked to choose which group between the two they would rather avoid. We found that Korean participants were more accurate than American participants overall, in line with the framework on cultural viewpoints: Holistic versus analytic processing in East Asians versus Westerners. Moreover, we found a speed advantage for other-race crowds in both cultural groups. Finally, we found different hemispheric lateralization patterns: American participants were more accurate to perceive the facial crowd to be avoided when it was presented in the left visual field than the right visual field, indicating a right hemisphere advantage for processing crowd emotion of both European American and Korean facial crowds. However, Korean participants showed weak or nonexistent laterality effects, with a slight right hemisphere advantage for European American facial crowds and no advantage in perceiving Korean facial crowds. Instead, Korean participants showed positive emotion bias for own-race faces. This work suggests that culture plays a role in modulating our crowd emotion perception of groups of faces and responses to them.

Differential hemispheric and visual stream contributions to ensemble coding of crowd emotion

In crowds, where scrutinizing individual facial expressions is inefficient, humans can make snap judgments about the prevailing mood by reading "crowd emotion". We investigated how the brain accomplishes this feat in a set of behavioral and fMRI studies. Participants were asked to either avoid or approach one of two crowds of faces presented in the left and right visual hemifields. Perception of crowd emotion was improved when crowd stimuli contained goal-congruent cues and was highly lateralized to the right hemisphere. The dorsal visual stream was preferentially activated in crowd emotion processing, with activity in the intraparietal sulcus and superior frontal gyrus predicting perceptual accuracy for crowd emotion perception, whereas activity in the fusiform cortex in the ventral stream predicted better perception of individual facial expressions. Our findings thus reveal significant behavioral differences and differential involvement of the hemispheres and the major visual streams in reading crowd versus individual face expressions.

Ensemble Perception

To understand visual consciousness, we must understand how the brain represents ensembles of objects at many levels of perceptual analysis. Ensemble perception refers to the visual system's ability to extract summary statistical information from groups of similar objects-often in a brief glance. It defines foundational limits on cognition, memory, and behavior. In this review, we provide an operational definition of ensemble perception and demonstrate that ensemble perception spans across multiple levels of visual analysis, incorporating both low-level visual features and high-level social information. Further, we investigate the functional usefulness of ensemble perception and its efficiency, and we consider possible physiological and cognitive mechanisms that underlie an individual's ability to make accurate and rapid assessments of crowds of objects.

Orientation-defined boundaries are detected with low efficiency

When compared with other summary statistics (mean size, size variance, orientation variance), visual estimates of average orientation are inefficient. Observers act as if they use information from no more than two or three items. We hypothesised that observers would attain greater sampling efficiency when their task did not require an explicit representation of mean orientation. We tested this hypothesis using a texture-segmentation task. Two arrays of 32 wavelets each were presented; one left and one right of fixation. Orientations in the target array were sampled from wrapped normal distributions having two different means with the same variance. One distribution defined orientations above the horizontal meridian, the other defined orientations below the meridian. All orientations in the other array were defined by a single wrapped normal distribution having the same variance as each of the distributions in the target array. Contrary to our hypothesis, results indicate that observers effectively ignored all but one item from the top and bottom of each array. In fact, we found no change in the threshold difference between the target's two means when all but one item from the top and bottom of each array were removed. We are forced to conclude that the visual system does not compute the average of more than a few orientations, even for texture segmentation.

The center of attention: Metamers, sensitivity, and bias in the emergent perception of gaze

A person's gaze reveals much about their focus of attention and intentions. Sensitive perception of gaze is thus highly relevant for social interaction, especially when it is directed toward the viewer. Yet observers also tend to overestimate the likelihood that gaze is directed toward them. How might the visual system balance these competing goals, maximizing sensitivity for discriminating gazes that are relatively direct, while at the same time allowing many gazes to appear as if they look toward the viewer? Perceiving gaze is an emergent visual process that involves integrating information from the eyes with the rotation of the head. Here, we examined whether the visual system leverages emergent representation to balance these competing goals. We measured perceived gaze for a large range of pupil and head combinations and found that head rotation has a nonlinear influence on a person's apparent direction of looking, especially when pupil rotations are relatively direct. These perceptual distortions could serve to expand representational space and thereby enhance discriminability of gazes that are relatively direct. We also found that the emergent perception of gaze supports an abundance of direct gaze metamers-different combinations of head and pupil rotations that combine to generate the appearance of gaze directed toward the observer. Our results thus demonstrate a way in which the visual system flexibly integrates information from facial features to optimize social perception. Many gazes can be made to look toward you, yet similar gazes need not appear alike.