Positive sequential dependency for face attractiveness perception

Serial dependence: A matter of memory load

In serial dependence, perceptual decisions are biased towards stimuli encountered in the recent past. Here, we investigate whether and how serial dependence is affected by the availability of visual working memory (VWM) resources. In two experiments, participants reproduced the orientation of a series of stimuli. On alternating trials, we included an additional VWM task with randomly varying levels of load. Serial dependence was not only affected by the additional load task but also clearly modulated by the level of load: a high load in the previous trial reduced serial dependence while a high load in the present increased it. These results were independent of the effects of VWM load on the precision of reproduction responses. Our findings provide new insights into the mechanisms that may regulate serial dependence, revealing its intimate link with VWM resources.

Significance statement

Our perception, thoughts, and behavior are continuously influenced by recent events. For instance, the way we process and understand current visual information depends on what we have seen in the preceding seconds, a phenomenon known as serial dependence. The precise mechanisms and factors involved in serial dependence are still unclear. Here, we demonstrated that working memory resources are a crucial component. Specifically, when we are currently experiencing a heavy memory load, the influence of prior stimuli becomes stronger. Conversely, when prior stimuli were shown under a high memory load, their influence was reduced. These findings highlight the importance of working memory resources in shaping our interpretation of the present based on the recent past.

Serial dependence requires visual awareness: Evidence from continuous flash suppression

The visual system often undergoes a relatively stable perception even in a noisy visual environment. This crucial function was reflected in a visual perception phenomenon-serial dependence, in which recent stimulus history systematically biases current visual decisions. Although serial dependence effects have been revealed in numerous studies, few studies examined whether serial dependence would require visual awareness. By using the continuous flash suppression (CFS) technique to render grating stimuli invisible, we investigated whether serial dependence effects could emerge at the unconscious levels. In an orientation adjustment task, subjects viewed a randomly oriented grating and reported their orientation perception via an adjustment response. Subjects performed a series of three type trial pairs. The first two trial pairs, in which subjects were instructed to make a response or no response toward the first trial of the pairs, respectively, were used to measure serial dependence at the conscious levels; the third trial pair, in which the grating stimulus in the first trial of the pair was masked by a CFS stimulus, was used to measure the serial dependence at the unconscious levels. One-back serial dependence effects for the second trial of the pairs were evaluated. We found significant serial dependence effects at the conscious levels, whether absence (Experiment 1) or presence (Experiment 2) of CFS stimuli, but failed to find the effects at the unconscious levels, corroborating the view that serial dependence requires visual awareness.

Opposing serial dependencies revealed for sequences of auditory emotional stimuli

Our percept of the world is not solely determined by what we perceive and process at a given moment in time, but also depends on what we processed recently. In the present study, we investigate whether the perceived emotion of a spoken sentence is contingent upon the emotion of an auditory stimulus on the preceding trial (i.e., serial dependence). Thereto, participants were exposed to spoken sentences that varied in emotional affect by changing the prosody that ranged from 'happy' to 'fearful'. Participants were instructed to rate the emotion. We found a positive serial dependence for emotion processing whereby the perceived emotion was biased towards the emotion on the preceding trial. When we introduced 'no-go' trials (i.e., no rating was required), we found a negative serial dependence when participants knew in advance to withhold their response on a given trial (Experiment 2) and a positive serial dependence when participants received the information to withhold their response after the stimulus presentation (Experiment 3). We therefore established a robust serial dependence for emotion processing in speech and introduce a methodology to disentangle perceptual from post-perceptual processes. This approach can be applied to the vast majority of studies investigating sequential dependencies to separate positive from negative serial dependence.

Tuning perception and decisions to temporal context

Recent work suggests that serial dependence, where perceptual decisions are biased toward previous stimuli, arises from the prior that sensory input is temporally correlated. However, existing studies have mostly used random stimulus sequences that do not involve such temporal consistencies. Here, we manipulated the temporal statistics of visual stimuli to examine the role of true temporal correlations in serial dependence. In two experiments, observers reproduced the orientation of the last stimulus in a sequence, while we varied temporal correlations in the stimulus features at two timescales: stimulus history within the trial and decision history across trials. We found a clear dissociation: increasing temporal correlation in the stimulus history led to adaptation-like repulsive biases, whereas increasing temporal correlation in the decision history reduced attractive biases. Thus, we suggest that temporal correlation enhances the discriminative ability of the visual system, revealing the fundamental role of the broader temporal context.

Task feedback suggests a post-perceptual component to serial dependence

Decisions across a range of perceptual tasks are biased toward past stimuli. Such serial dependence is thought to be an adaptive low-level mechanism that promotes perceptual stability across time. However, recent studies suggest post-perceptual mechanisms may also contribute to serially biased responses, calling into question a single locus of serial dependence and the nature of integration of past and present sensory inputs. We measured serial dependence in the context of a three-dimensional (3D) motion perception task where uncertainty in the sensory information varied substantially from trial to trial. We found that serial dependence varied with stimulus properties that impact sensory uncertainty on the current trial. Reduced stimulus contrast was associated with an increased bias toward the stimulus direction of the previous trial. Critically, performance feedback, which reduced sensory uncertainty, abolished serial dependence. These results provide clear evidence for a post-perceptual locus of serial dependence in 3D motion perception and support the role of serial dependence as a response strategy in the face of substantial sensory uncertainty.

Stable perceptual phenotype of the magnitude of history biases even in the face of global task complexity

According to a Bayesian framework, visual perception requires active interpretation of noisy sensory signals in light of prior information. One such mechanism, serial dependence, is thought to promote perceptual stability by assimilating current percepts with recent stimulus history. Combining a delayed orientation-adjustment paradigm with predictable (study 1) or unpredictable (study 2) task structure, we test two key predictions of this account in a novel context: first, that serial dependence should persist even in variable environments, and, second, that, within a given observer and context, this behavioral bias should be stable from one occasion to the next. Relying on data of 41 human volunteers and two separate experimental sessions, we confirm both hypotheses. Group-level, attractive serial dependence remained strong even in the face of volatile settings with multiple, unpredictable types of tasks, and, despite considerable interindividual variability, within-subject patterns of attractive and repulsive stimulus-history biases were highly stable from one experimental session to the next. In line with the hypothesized functional role of serial dependence, we propose that, together with previous work, our findings suggest the existence of a more general individual-specific fingerprint with which the past shapes current perception. Congruent with the Bayesian account, interindividual differences may then result from differential weighting of sensory evidence and prior information.

Serial dependence in visual perception: A meta-analysis and review

Positive sequential dependencies are phenomena in which actions, perception, decisions, and memory of features or objects are systematically biased toward visual experiences from the recent past. Among many labels, serial dependencies have been referred to as priming, sequential dependencies, sequential effects, or serial effects. Despite extensive research on the topic, the field still lacks an operational definition of what counts as serial dependence. In this meta-analysis, we review the vast literature on serial dependence and quantitatively assess its key diagnostic characteristics across several different domains of visual perception. The meta-analyses fully characterize serial dependence in orientation, face, and numerosity perception. They show that serial dependence is defined by four main kinds of tuning: serial dependence decays with time (temporal-tuning), it depends on relative spatial location (spatial-tuning), it occurs only between similar features and objects (feature-tuning), and it is modulated by attention (attentional-tuning). We also review studies of serial dependence that report single observer data, highlighting the importance of individual differences in serial dependence. Finally, we discuss a range of outstanding questions and novel research avenues that are prompted by the meta-analyses. Together, the meta-analyses provide a full characterization of serial dependence as an operationally defined family of visual phenomena, and they outline several of the key diagnostic criteria for serial dependence that should serve as guideposts for future research.

Independent short- and long-term dependencies in perception

Perception is biased by stimulus history. Both long-term effects such as the central-tendency bias (CTB) and short-term effects such as serial dependence (SD) have been described, but research into the two has remained largely separate. The sources of these effects, however, are highly correlated in stimulus statistics, which can result in a misinterpretation of experimental data. We compared CTB and SD in the perception of color and line length. Observers judged the relative hue or length of consecutive stimuli in a delayed-matching task. Two interstimulus intervals were used to investigate whether elapsed time or the number of stimulus occurrences was more important for SD. We estimated biases by fitting psychometric functions to the data split based on the history features, and we also fit generalized linear mixed models with either CTB, SD, or both included as regressors. We found biases to both recent stimulus history and the cumulative average of stimulus values for both color and line length judgments. The strength and pattern of each of the biases depended on whether all sources of bias were included in the analysis. Within the range of interstimulus intervals tested, the number of intervening stimuli was more important than elapsed time for SD. We conclude that both SD and CTB independently affect perceptual judgments, and that one effect is not an artifact caused by the other. Failing to consider both effects in data analysis can give an erroneous picture of the phenomenon under study.

Modeling facial perception in group context from a serial perception perspective

By utilizing statistical properties and summary statistics, the visual system can efficiently integrate perception of spatially and temporally adjacent stimuli into perception of a given target. For instance, perception of a target face can either be biased positively toward previous faces (e.g. the serial dependence effect) or be biased negatively by surrounding faces in the same trial/space (e.g. spatial ensemble averaging). However, both aspects were investigated separately. As spatial and temporal processing share the same purpose to reduce redundancy in visual processing, if one statistical processing occurs, would the statistical processing in the other domain still exist or be discarded? We investigated this question by exploring whether serial dependence of face perception (of attractiveness and averageness) survives when the changed face perception in the group context occurs. The results of Markov Chain modeling and conventional methods suggested that serial dependence (the temporal aspect) co-occurs with changed face perception in the group context (the spatial aspect). We also utilized the Hidden Markov modeling, as a new mathematical method, to model statistical processing from both domains. The results confirmed the co-occurrence of temporal effect and changed face perception in the group context for both attractiveness and averageness, suggesting potentially different spatial and temporal compression mechanisms in high-level vision. Further modeling and cluster analysis further revealed that the detailed computation of spatially and temporally adjacent faces in the attractiveness and averageness processing were similar yet different among different individuals. This work builds a bridge to understanding mathematical principles underlying changed face perception in the group context from the serial perspective.

Number-time interaction: Search for a common magnitude system in a cross-modal setting

A theory of magnitude (ATOM) suggests that a generalized magnitude system in the brain processes magnitudes such as space, time, and numbers. Numerous behavioral and neurocognitive studies have provided support to ATOM theory. However, the evidence for common magnitude processing primarily comes from the studies in which numerical and temporal information are presented visually. Our current understanding of such cross-dimensional magnitude interactions is limited to visual modality only. However, it is still unclear whether the ATOM-framework accounts for the integration of cross-modal magnitude information. To examine the cross-modal influence of numerical magnitude on temporal processing of the tone, we conducted three experiments using a temporal bisection task. We presented the numerical magnitude information in the visual domain and the temporal information in the auditory either simultaneously with duration judgment task (Experiment-1), before duration judgment task (Experiment-2), and before duration judgment task but with numerical magnitude also being task-relevant (Experiment-3). The results suggest that the numerical information presented in the visual domain affects temporal processing of the tone only when the numerical magnitudes were task-relevant and available while making a temporal judgment (Experiments-1 and 3). However, numerical information did not interfere with temporal information when presented temporally separated from the duration information (Experiments-2). The findings indicate that the influence of visual numbers on temporal processing in cross-modal settings may not arise from the common magnitude system but instead from general cognitive mechanisms like attention and memory.

Stimulus uncertainty predicts serial dependence in orientation judgements

How is what you see influenced by what you saw? The visual system may use recent perception to inform responses to current stimuli. This can cause the perception of current stimuli to be attracted toward previous observations, an effect termed serial dependence. This misperception might well be useful in a noisy visual environment, where minor image distortions over time may not actually represent meaningful change. Previous work has suggested that Bayesian perceptual inference may underlie serial dependence. For this to be true, the relative uncertainty associated with both prior and current sensory input should be taken into account. In an experiment manipulating the level of noise present in orientation stimuli, we found an effect of current stimulus uncertainty on serial dependence. We found no good evidence for an effect of previous stimulus uncertainty. Our results provide only partial evidence for the Bayesian interpretation of serial dependence. Non-Bayesian models may provide a better account of the phenomenon.

A general serial dependence among various facial traits: Evidence from Markov Chain and derivative of Gaussian

The human vision system can extract a stable representation of the always-changing visual world. However, the mechanism underlying such perceptual continuity remains unclear. A possible candidate is the serial dependence: visual perception of an object is positively biased toward the visual input from the recent past. Does the visual system use one pattern of serial dependence for general purposes? Or different patterns of serial dependence for different visual tasks? Because different social facial traits (e.g., trustworthiness and dominance) are dissociable, it is reasonable to assume that the perception of different facial characteristics would require different patterns of serial dependences. In this study, we examine the existence and the similarities of the serial dependence(s) in the evaluation of seven facial characteristics (i.e., attractiveness, trustworthiness, confidence, dominance, intelligence, age, and aggressiveness). The convergent evidence from conventional Derivative of Gaussian fitting and Markov Chain modeling demonstrated that (1) serial dependence exists in judgments of all seven social facial characteristic, (2) the serial dependences of them are highly similar, and (3) the serial dependence follows the efficient coding. Thus it is highly possible that there exists a general serial dependence mechanism for (at least high-level) vision processing. Moreover, we used the Markov Chain modeling to better describe the transitional pattern of serial dependence, which is a kind of Markov process. These findings may shed light on future works regarding serial dependence, as well as face perception.

Dynamics of history-dependent perceptual judgment

Identical physical inputs do not always evoke identical percepts. To investigate the role of stimulus history in tactile perception, we designed a task in which rats had to judge each vibrissal vibration, in a long series, as strong or weak depending on its mean speed. After a low-speed stimulus (trial n - 1), rats were more likely to report the next stimulus (trial n) as strong, and after a high-speed stimulus, they were more likely to report the next stimulus as weak, a repulsive effect that did not depend on choice or reward on trial n - 1. This effect could be tracked over several preceding trials (i.e., n - 2 and earlier) and was characterized by an exponential decay function, reflecting a trial-by-trial incorporation of sensory history. Surprisingly, the influence of trial n - 1 strengthened as the time interval between n - 1 and n grew. Human subjects receiving fingertip vibrations showed these same key findings. We are able to account for the repulsive stimulus history effect, and its detailed time scale, through a single-parameter model, wherein each new stimulus gradually updates the subject's decision criterion. This model points to mechanisms underlying how the past affects the ongoing subjective experience.

A shared mechanism for facial expression in human faces and face pareidolia

Facial expressions are vital for social communication, yet the underlying mechanisms are still being discovered. Illusory faces perceived in objects (face pareidolia) are errors of face detection that share some neural mechanisms with human face processing. However, it is unknown whether expression in illusory faces engages the same mechanisms as human faces. Here, using a serial dependence paradigm, we investigated whether illusory and human faces share a common expression mechanism. First, we found that images of face pareidolia are reliably rated for expression, within and between observers, despite varying greatly in visual features. Second, they exhibit positive serial dependence for perceived facial expression, meaning an illusory face (happy or angry) is perceived as more similar in expression to the preceding one, just as seen for human faces. This suggests illusory and human faces engage similar mechanisms of temporal continuity. Third, we found robust cross-domain serial dependence of perceived expression between illusory and human faces when they were interleaved, with serial effects larger when illusory faces preceded human faces than the reverse. Together, the results support a shared mechanism for facial expression between human faces and illusory faces and suggest that expression processing is not tightly bound to human facial features.

Serial dependence in time and numerosity perception is dimension-specific

The perception of a visual event (e.g., a flock of birds) at the present moment can be biased by a previous perceptual experience (e.g., the perception of an earlier flock). Serial dependence is a perceptual bias whereby a current stimulus appears more similar to a previous one than it actually is. Whereas serial dependence emerges within several visual stimulus dimensions, whether it could simultaneously operate across different dimensions of the same stimulus (e.g., the numerosity and the duration of a visual pattern) remains unclear. Here we address this question by assessing the presence of serial dependence across duration and numerosity, two stimulus dimensions that are often associated and can bias each other. Participants performed either a duration or a numerosity discrimination task, in which they compared a constant reference with a variable test stimulus, varying along the task-relevant dimension (either duration or numerosity). Serial dependence was induced by a task-irrelevant inducer, that is, a stimulus presented before the reference and always varying in both duration and numerosity. The results show systematic serial dependencies only within the task-relevant stimulus dimension, that is, stimulus numerosity affects numerosity perception only, and duration affects duration perception only. Additionally, at least in the numerosity condition, the task-irrelevant dimension of the inducer (duration) had an opposite, repulsive effect. These findings thus show that attractive serial dependence operates in a highly specific fashion and does not transfer across different stimulus dimensions. Instead, the repulsive influence, possibly reflecting perceptual adaptation, can transfer from one dimension to another.

A Bayesian and efficient observer model explains concurrent attractive and repulsive history biases in visual perception

Human perceptual decisions can be repelled away from (repulsive adaptation) or attracted towards recent visual experience (attractive serial dependence). It is currently unclear whether and how these repulsive and attractive biases interact during visual processing and what computational principles underlie these history dependencies. Here we disentangle repulsive and attractive biases by exploring their respective timescales. We find that perceptual decisions are concurrently attracted towards the short-term perceptual history and repelled from stimuli experienced up to minutes into the past. The temporal pattern of short-term attraction and long-term repulsion cannot be captured by an ideal Bayesian observer model alone. Instead, it is well captured by an ideal observer model with efficient encoding and Bayesian decoding of visual information in a slowly changing environment. Concurrent attractive and repulsive history biases in perceptual decisions may thus be the consequence of the need for visual processing to simultaneously satisfy constraints of efficiency and stability.