Context information supports serial dependence of multiple visual objects across memory episodes

Priming effect of individual similarity and ensemble perception in visual search and working memory

Perceptual priming is a well-known phenomenon showing that the repetition of an object's feature can facilitate subsequent detection of that item. Although the priming effect has been rigorously studied in visual search, less is known about its effect on working memory and it is unclear whether the repetition of similar features, and furthermore, ensemble perception created by a large set of similar features, can induce priming. In this study, we investigated the priming effects of individual similarity and ensemble perception in visual search and visual working memory (VWM). We replicated the classic perceptual priming effect (Experiment 1a) and found that visual search was enhanced when the current target had a similar color to the previous target (Experiment 1b), but not when the similar color had been shown as a distractor before (Experiment 1c). However, if the target and distractors of similar colors formed ensemble perception, the search efficiency was again promoted even when the current target shared the same color with the previous distractor (Experiment 1d). For VWM, repeating the ensembles of the target- and nontarget-color subsets did not significantly affect the memory capacity, while switching the two harmed the memory fidelity but not capacity (Experiment 2). We suggest different underlying mechanisms for priming in visual search and VWM: in the former, the perception history of individual similarity and stimuli ensemble exert their effects on through the priority map, by forming a gradient distribution of attentional weights that peak at the previous target feature and diminish as stimulus diverges from the previously selected one; while in the latter, perception history of memory ensemble may influence the deployment of existing memory resources across trials, thereby affecting the memory fidelity but not its capacity.

The distinct development of stimulus and response serial dependence

Serial dependence (SD) is a phenomenon wherein current perceptions are biased by the previous stimulus and response. This helps to attenuate perceptual noise and variability in sensory input and facilitates stable ongoing perceptions of the environment. However, little is known about the developmental trajectory of SD. This study investigates how the stimulus and response biases of the SD effect develop across three age groups. Conventional analyses, in which previous stimulus and response biases were assessed separately, revealed significant changes in the biases over time. Previous stimulus bias shifted from repulsion to attraction, while previous response bias evolved from attraction to greater attraction. However, there was a strong correlation between stimulus and response orientations. Therefore, a generalized linear mixed-effects (GLME) analysis that simultaneously considered both previous stimulus and response, outperformed separate analyses. This revealed that previous stimulus and response resulted in two distinct biases with different developmental trajectories. The repulsion bias of previous stimulus remained relatively stable across all age groups, whereas the attraction bias of previous response was significantly stronger in adults than in children and adolescents. These findings demonstrate that the repulsion bias towards preceding stimuli is established early in the developing brain (at least by around 10 years old), while the attraction bias towards responses is not fully developed until adulthood. Our findings provide new insights into the development of the SD phenomenon and how humans integrate two opposing mechanisms into their perceptual responses to external input during development.

The push-pull of serial dependence effects: Attraction to the prior response and repulsion from the prior stimulus

In the "serial dependence" effect, responses to visual stimuli appear biased toward the last trial's stimulus. However, several kinds of serial dependence exist, with some reflecting prior stimuli and others reflecting prior responses. One-factor analyses consider the prior stimulus alone or the prior response alone and can consider both variables only via separate analyses. We demonstrate that one-factor analyses are potentially misleading and can reach conclusions that are opposite from the truth if both dependencies exist. To address this limitation, we developed two-factor analyses (model comparison with hierarchical Bayesian modeling and an empirical "quadrant analysis"), which consider trial-by-trial combinations of prior response and prior stimulus. Two-factor analyses can tease apart the two dependencies if applied to a sufficiently large dataset. We applied these analyses to a new study and to four previously published studies. When applying a model that included the possibility of both dependencies, there was no evidence of attraction to the prior stimulus in any dataset, but there was evidence of attraction to the prior response in all datasets. Two of the datasets contained sufficient constraint to determine that both dependencies were needed to explain the results. For these datasets, the dependency on the prior stimulus was repulsive rather than attractive. Our results are consistent with the claim that both dependencies exist in most serial dependence studies (the two-dependence model was not ruled out for any dataset) and, furthermore, that the two dependencies work against each other.

Memory reports are biased by all relevant contents of working memory

Sensory input is inherently noisy while the world is inherently predictable. When multiple observations of the same object are available, integration of the available information necessarily increases the reliability of a world estimate. Optimal integration of multiple instances of sensory evidence has already been demonstrated during multisensory perception but could benefit unimodal perception as well. In the present study 330 participants observed a sequence of four orientations and were cued to report one of them. Reports were biased by all simultaneously memorized items that were similar and relevant to the target item, weighted by their reliability (signal-to-noise ratio). Orientations presented before and presented after the target biased report, demonstrating that the bias emerges in memory and not (exclusively) during perception or encoding. Only attended, task-relevant items biased report. We suggest that these results reflect how the visual system integrates information that is sampled from the same object at consecutive timepoints to promote perceptual stability and behavioural effectiveness in a dynamic world. We suggest that similar response biases, such as serial dependence, might be instances of a more general mechanism of working memory averaging. Data is available at https://osf.io/embcf/ .

Serial Dependence in Perception

Much evidence has shown that perception is biased towards previously presented similar stimuli, an effect recently termed serial dependence. Serial dependence affects nearly every aspect of perception, often causing gross perceptual distortions, especially for weak and ambiguous stimuli. Despite unwanted side-effects, empirical evidence and Bayesian modeling show that serial dependence acts to improve efficiency and is generally beneficial to the system. Consistent with models of predictive coding, the Bayesian priors of serial dependence are generated at high levels of cortical analysis, incorporating much perceptual experience, but feed back to lower sensory areas. These feedback loops may drive oscillations in the alpha range, linked strongly with serial dependence. The discovery of top-down predictive perceptual processes is not new, but the new, more quantitative approach characterizing serial dependence promises to lead to a deeper understanding of predictive perceptual processes and their underlying neural mechanisms.

Attention influences the effects of the previous form orientation on the current motion direction estimation

Recent studies have found that the estimates of motion directions are biased toward the previous form orientations, showing serial dependence, and the serial dependence does not involve cognitive abilities. In the current study, we conducted two experiments to investigate whether and how attention-a cognitive ability-affected the serial dependence. The results showed that serial dependence was present in the current study, reproducing the previous findings. Importantly, when the attentional load reduced the reliability (i.e., estimation accuracy and precision) of previous form orientations (Experiment 1), the serial dependence decreased, meaning that the biases of motion direction estimates toward previous form orientations were reduced; in contrast, when the attentional load reduced the reliability of current motion directions (Experiment 2), the serial dependence increased, meaning that the biases of motion direction estimates toward previous form orientations were increased. These trends were well consistent with the prediction of the Bayesian inference theory. Therefore, the current study revealed the involvement of attention in the serial dependence of current motion direction estimation on the previous form orientation, demonstrating that the serial dependence was cognitive and the attentional effect can be a Bayesian inference process, initially revealing its computational mechanism.

Serial dependence bias can predict the overall estimation error in visual perception

Although visual feature estimations are accurate and precise, overall estimation errors (i.e., the difference between estimates and actual values) tend to show systematic patterns. For example, estimates of orientations are systematically biased away from horizontal and vertical orientations, showing an oblique illusion. Additionally, many recent studies have demonstrated that estimations of current visual features are systematically biased toward previously seen features, showing a serial dependence. However, no study examined whether the overall estimation errors were correlated with the serial dependence bias. To address this question, we enrolled three groups of participants to estimate orientation, motion speed, and point-light-walker direction. The results showed that the serial dependence bias explained over 20% of overall estimation errors in the three tasks, indicating that we could use the serial dependence bias to predict the overall estimation errors. The current study first demonstrated that the serial dependence bias was not independent from the overall estimation errors. This finding could inspire researchers to investigate the neural bases underlying the visual feature estimation and serial dependence.

Spatial correspondence in relative space regulates serial dependence

Our perception is often attracted to what we have seen before, a phenomenon called 'serial dependence.' Serial dependence can help maintain a stable perception of the world, given the statistical regularity in the environment. If serial dependence serves this presumed utility, it should be pronounced when consecutive elements share the same identity when multiple elements spatially shift across successive views. However, such preferential serial dependence between identity-matching elements in dynamic situations has never been empirically tested. Here, we hypothesized that serial dependence between consecutive elements is modulated more effectively by the spatial correspondence in relative space than by that in absolute space because spatial correspondence in relative coordinates can warrant identity matching invariantly to changes in absolute coordinates. To test this hypothesis, we developed a task where two targets change positions in unison between successive views. We found that serial dependence was substantially modulated by the correspondence in relative coordinates, but not by that in absolute coordinates. Moreover, such selective modulation by the correspondence in relative space was also observed even for the serial dependence defined by previous non-target elements. Our findings are consistent with the view that serial dependence subserves object-based perceptual stabilization over time in dynamic situations.

Tuning perception and decisions to temporal context

Recent work suggests that serial dependence, where perceptual decisions are biased toward previous stimuli, arises from the prior that sensory input is temporally correlated. However, existing studies have mostly used random stimulus sequences that do not involve such temporal consistencies. Here, we manipulated the temporal statistics of visual stimuli to examine the role of true temporal correlations in serial dependence. In two experiments, observers reproduced the orientation of the last stimulus in a sequence, while we varied temporal correlations in the stimulus features at two timescales: stimulus history within the trial and decision history across trials. We found a clear dissociation: increasing temporal correlation in the stimulus history led to adaptation-like repulsive biases, whereas increasing temporal correlation in the decision history reduced attractive biases. Thus, we suggest that temporal correlation enhances the discriminative ability of the visual system, revealing the fundamental role of the broader temporal context.

The effects of visual distractors on serial dependence

Attractive serial dependence occurs when perceptual decisions are attracted toward previous stimuli. This effect is mediated by spatial attention and is most likely to occur when similar stimuli are attended at nearby locations. Attention, however, also involves the suppression of distracting information and of spatial locations where distracting stimuli have frequently appeared. Although distractors form an integral part of our visual experience, how they affect the processing of subsequent stimuli is unknown. Here, in two experiments, we tested serial dependence from distractor stimuli during an orientation adjustment task. We interleaved adjustment trials with a discrimination task requiring observers to ignore a peripheral distractor randomly appearing on half of the trials. Distractors were either similar to the adjustment probe (Experiment 1) or differed in spatial frequency and contrast (Experiment 2) and were shown at predictable or random locations in separate blocks. The results showed that the distractor caused considerable attentional capture in the discrimination task, with observers likely using proactive strategies to anticipate distractors at predictable locations. However, there was no evidence that the distractors affected the perceptual stream leading to positive serial dependence. Instead, they left a weak repulsive trace in Experiment 1 and more generally interfered with the effect of the previous adjustment probe in the serial dependence task. We suggest that this repulsive bias may reflect the operation of mechanisms involved in attentional suppression.

Stable perceptual phenotype of the magnitude of history biases even in the face of global task complexity

According to a Bayesian framework, visual perception requires active interpretation of noisy sensory signals in light of prior information. One such mechanism, serial dependence, is thought to promote perceptual stability by assimilating current percepts with recent stimulus history. Combining a delayed orientation-adjustment paradigm with predictable (study 1) or unpredictable (study 2) task structure, we test two key predictions of this account in a novel context: first, that serial dependence should persist even in variable environments, and, second, that, within a given observer and context, this behavioral bias should be stable from one occasion to the next. Relying on data of 41 human volunteers and two separate experimental sessions, we confirm both hypotheses. Group-level, attractive serial dependence remained strong even in the face of volatile settings with multiple, unpredictable types of tasks, and, despite considerable interindividual variability, within-subject patterns of attractive and repulsive stimulus-history biases were highly stable from one experimental session to the next. In line with the hypothesized functional role of serial dependence, we propose that, together with previous work, our findings suggest the existence of a more general individual-specific fingerprint with which the past shapes current perception. Congruent with the Bayesian account, interindividual differences may then result from differential weighting of sensory evidence and prior information.

Serial dependence in visual perception: A meta-analysis and review

Positive sequential dependencies are phenomena in which actions, perception, decisions, and memory of features or objects are systematically biased toward visual experiences from the recent past. Among many labels, serial dependencies have been referred to as priming, sequential dependencies, sequential effects, or serial effects. Despite extensive research on the topic, the field still lacks an operational definition of what counts as serial dependence. In this meta-analysis, we review the vast literature on serial dependence and quantitatively assess its key diagnostic characteristics across several different domains of visual perception. The meta-analyses fully characterize serial dependence in orientation, face, and numerosity perception. They show that serial dependence is defined by four main kinds of tuning: serial dependence decays with time (temporal-tuning), it depends on relative spatial location (spatial-tuning), it occurs only between similar features and objects (feature-tuning), and it is modulated by attention (attentional-tuning). We also review studies of serial dependence that report single observer data, highlighting the importance of individual differences in serial dependence. Finally, we discuss a range of outstanding questions and novel research avenues that are prompted by the meta-analyses. Together, the meta-analyses provide a full characterization of serial dependence as an operationally defined family of visual phenomena, and they outline several of the key diagnostic criteria for serial dependence that should serve as guideposts for future research.

Attractive and repulsive serial dependence: The role of task relevance, the passage of time, and the number of stimuli

Visual decisions are attracted toward features of previous stimuli. This phenomenon, termed serial dependence, has been related to a mechanism that integrates present visual input with stimuli seen up to 10 to 15 s in the past. It is believed that this mechanism is "temporally tuned" and the effect of prior stimuli fades with time. Here, we investigated whether the temporal window of serial dependence is influenced by the number of stimuli shown. Observers performed an orientation adjustment task where the interval between the past and present stimulus and the number of intervening stimuli varied. First, we found that the direction-repulsive or attractive-and duration of the effect of a past stimulus depends on whether the past stimulus was relevant to behavior. Second, we show that the number of stimuli, and not only the passage of time, plays a role: The effect of a stimulus at a fixed interval depends on the number of other stimuli shown after. Our results demonstrate that neither a single mechanism nor a general tuning window can fully capture the complexity of serial dependence.

Independent short- and long-term dependencies in perception

Perception is biased by stimulus history. Both long-term effects such as the central-tendency bias (CTB) and short-term effects such as serial dependence (SD) have been described, but research into the two has remained largely separate. The sources of these effects, however, are highly correlated in stimulus statistics, which can result in a misinterpretation of experimental data. We compared CTB and SD in the perception of color and line length. Observers judged the relative hue or length of consecutive stimuli in a delayed-matching task. Two interstimulus intervals were used to investigate whether elapsed time or the number of stimulus occurrences was more important for SD. We estimated biases by fitting psychometric functions to the data split based on the history features, and we also fit generalized linear mixed models with either CTB, SD, or both included as regressors. We found biases to both recent stimulus history and the cumulative average of stimulus values for both color and line length judgments. The strength and pattern of each of the biases depended on whether all sources of bias were included in the analysis. Within the range of interstimulus intervals tested, the number of intervening stimuli was more important than elapsed time for SD. We conclude that both SD and CTB independently affect perceptual judgments, and that one effect is not an artifact caused by the other. Failing to consider both effects in data analysis can give an erroneous picture of the phenomenon under study.

The influence of the tested item on serial dependence in perceptual decisions

Serial dependence in vision reflects how perceptual decisions can be biased by what we have recently perceived. Serial dependence studies test single items' effects on perceptual decisions. However, our visual world contains multiple objects at any given moment, so it's important to understand how past experiences affect not only a single object but also perception in a more general sense. Here we asked the question: What effect does a single item have when there is more than one subsequently presented test item? We displayed a single line (inducer) at the screen center, then either a single test-line or two simultaneous test-lines, varying in orientation space to the inducer. Next, participants reported test-line orientation using a left or right located response circle (to indicate which test-line should be reported). The results demonstrated that the inducer influenced subsequent perceptual judgments of a test-line even when two test-lines were presented. Distant items caused repulsive serial dependence, while close items caused attractive serial dependence. This shows how a single inducer can influence test-line judgments, even when two test-lines are presented, and can produce attractive and repulsive serial dependence biases when the item to report is revealed after it has disappeared.

Modality-specific sensory and decisional carryover effects in duration perception

BACKGROUND

The brain uses recent history when forming perceptual decisions. This results in carryover effects in perception. Although separate sensory and decisional carryover effects have been shown in many perceptual tasks, their existence and nature in temporal processing are unclear. Here, we investigated whether and how previous stimuli and previous choices affect subsequent duration perception, in vision and audition.

RESULTS

In a series of three experiments, participants were asked to classify visual or auditory stimuli into "shorter" or "longer" duration categories. In experiment 1, visual and auditory stimuli were presented in separate blocks. Results showed that current duration estimates were repelled away from the previous trial's stimulus duration, but attracted towards the previous choice, in both vision and audition. In experiment 2, visual and auditory stimuli were pseudorandomly presented in one block. We found that sensory and decisional carryover effects occurred only when previous and current stimuli were from the same modality. Experiment 3 further investigated the stimulus dependence of carryover effects within each modality. In this experiment, visual stimuli with different shape topologies (or auditory stimuli with different audio frequencies) were pseudorandomly presented in one visual (or auditory) block. Results demonstrated sensory carryover (within each modality) despite task-irrelevant differences in visual shape topology or audio frequency. By contrast, decisional carryover was reduced (but still present) across different visual topologies and completely absent across different audio frequencies.

CONCLUSIONS

These results suggest that serial dependence in duration perception is modality-specific. Moreover, repulsive sensory carryover effects generalize within each modality, whereas attractive decisional carryover effects are contingent on contextual details.

Serial dependencies between locomotion and visual space

How do we know the spatial distance of objects around us? Only by physical interaction within an environment can we measure true physical distances. Here, we investigated the possibility that travel distances, measured during walking, could be used to calibrate visual spatial perception. The sensorimotor contingencies that arise during walking were carefully altered using virtual reality and motion tracking. Participants were asked to walk to a briefly highlighted location. During walking, we systematically changed the optic flow, i.e., the ratio between the visual and physical motion speed. Although participants remained unaware of this manipulation, they walked a shorter or longer distance as a function of the optic flow speed. Following walking, participants were required to estimate the perceived distance of visual objects. We found that visual estimates were serially dependent on the experience of the manipulated flow in the previous trial. Additional experiments confirmed that to affect visual perception, both visual and physical motion are required. We conclude that the brain constantly uses movements to measure space for both, actions, and perception.

Effects of satisfying and violating expectations on serial dependence

Serial dependence refers to the phenomenon that observers tend to report stimuli as being more similar to previous stimuli than they really are (attractive dependence) or, in some cases, as more different than they really are (repulsive dependence). Numerous experiments have demonstrated serial dependence for a range of modalities and stimulus features, highlighting the role of bottom-up sensory interactions. However, comparatively less research has focused on how higher-level cognitive factors, such as expectations, might influence serial dependence. Here, we manipulated expectations by having observers respond to target luminance gratings that occurred at the end of a sequence of non-target gratings. The sequence either rotated predictably (inducing an expectation), varied randomly (inducing no expectation), or rotated predictably but had a random target orientation (violating expectations). We found that observers produced less errors and indicated less uncertainty in their estimations of expected stimuli but their responses were biased away from the penultimate stimulus in the sequence (repulsive dependence). In contrast, following random sequences, responses showed an attractive bias to the penultimate stimulus in the sequence. Unexpected targets showed a mixture of both biases, such that when targets happened (by chance) to appear as expected, responses were repulsed, but responses to target orientations that more clearly violated expectations were attracted. These results indicate that, whereas attraction to previous stimuli may be a default strategy employed in response to random and unexpected events, certain expectations can reverse the default bias into a repulsive one.

Serial dependence in visual perception: A review

How does the visual system represent continuity in the constantly changing visual input? A recent proposal is that vision is serially dependent: Stimuli seen a moment ago influence what we perceive in the present. In line with this, recent frameworks suggest that the visual system anticipates whether an object seen at one moment is the same as the one seen a moment ago, binding visual representations across consecutive perceptual episodes. A growing body of work supports this view, revealing signatures of serial dependence in many diverse visual tasks. Yet, the variety of disparate findings and interpretations calls for a more general picture. Here, we survey the main paradigms and results over the past decade. We also focus on the challenge of finding a relationship between serial dependence and the concept of "object identity," taking centuries-long history of research into account. Among the seemingly contrasting findings on serial dependence, we highlight common patterns that may elucidate the nature of this phenomenon and attempt to identify questions that are unanswered.

Attractive and repulsive effects of sensory history concurrently shape visual perception

Background

Sequential effects of environmental stimuli are ubiquitous in most behavioral tasks involving magnitude estimation, memory, decision making, and emotion. The human visual system exploits continuity in the visual environment, which induces two contrasting perceptual phenomena shaping visual perception. Previous work reported that perceptual estimation of a stimulus may be influenced either by attractive serial dependencies or repulsive aftereffects, with a number of experimental variables suggested as factors determining the direction and magnitude of sequential effects. Recent studies have theorized that these two effects concurrently arise in perceptual processing, but empirical evidence that directly supports this hypothesis is lacking, and it remains unclear whether and how attractive and repulsive sequential effects interact in a trial. Here we show that the two effects concurrently modulate estimation behavior in a typical sequence of perceptual tasks.

Results

We first demonstrate that observers’ estimation error as a function of both the previous stimulus and response cannot be fully described by either attractive or repulsive bias but is instead well captured by a summation of repulsion from the previous stimulus and attraction toward the previous response. We then reveal that the repulsive bias is centered on the observer’s sensory encoding of the previous stimulus, which is again repelled away from its own preceding trial, whereas the attractive bias is centered precisely on the previous response, which is the observer’s best prediction about the incoming stimuli.

Conclusions

Our findings provide strong evidence that sensory encoding is shaped by dynamic tuning of the system to the past stimuli, inducing repulsive aftereffects, and followed by inference incorporating the prediction from the past estimation, leading to attractive serial dependence.

A key role of orientation in the coding of visual motion direction

Despite the fundamental importance of visual motion processing, our understanding of how the brain represents basic aspects of motion is incomplete. While it is generally believed that direction is the main representational feature of motion, motion processing is also influenced by nondirectional orientation signals that are present in most motion stimuli. Here, we aimed to test whether this nondirectional motion axis contributes motion perception even when orientation is completely absent from the stimulus. Using stimuli with and without orientation signals, we found that serial dependence in a simple motion direction estimation task was predominantly determined by the orientation of the previous motion stimulus. Moreover, the observed attraction profiles closely matched the characteristic pattern of serial attraction found in orientation perception. Evidently, the sequential integration of motion signals strongly depends on the orientation of motion, indicating a fundamental role of nondirectional orientation in the coding of visual motion direction.

Cartesian coordinates scaffold stable spatial perception over time

Visual systems exploit temporal continuity principles to achieve stable spatial perception, manifested as the serial dependence and central tendency effects. These effects are posited to reflect a smoothing process whereby past and present information integrates over time to decrease noise and stabilize perception. Meanwhile, the basic spatial coordinate—Cartesian versus polar—that scaffolds the integration process in two-dimensional continuous space remains unknown. The spatial coordinates are largely related to the allocentric and egocentric reference frames and presumably correspond with early and late processing stages in spatial perception. Here, four experiments consistently demonstrate that Cartesian outperforms polar coordinates in characterizing the serial bias—serial dependence and central tendency effect—in two-dimensional continuous spatial perception. The superiority of Cartesian coordinates is robust, independent of task environment (online and offline task), experimental length (short and long blocks), spatial context (shape of visual mask), and response modality (keyboard and mouse). Taken together, the visual system relies on the Cartesian coordinates for spatiotemporal integration to facilitate stable representation of external information, supporting the involvement of allocentric reference frame and top-down modulation in spatial perception over long time intervals.

The effect of abstract representation and response feedback on serial dependence in numerosity perception

Serial dependence entails an attractive bias based on the recent history of stimulation, making the current stimulus appear more similar to the preceding one. Although serial dependence is ubiquitous in perception, its nature and mechanisms remain unclear. Here, in two independent experiments, we test the hypothesis that this bias originates from high-level processing stages at the level of abstract information processing (Exp. 1) or at the level of judgment (Exp. 2). In Experiment 1, serial dependence was induced by a task-irrelevant "inducer" stimulus in a numerosity discrimination task, similarly to previous studies. Importantly, in this experiment, the inducers were either arrays of dots similar to the task-relevant stimuli (e.g., 12 dots), or symbolic numbers (e.g., the numeral "12"). Both dots and symbol inducers successfully yielded attractive serial dependence biases, suggesting that abstract information about an image is sufficient to bias the perception of the current stimulus. In Experiment 2, participants received feedback about their responses in each trial of a numerosity estimation task, which was designed to assess whether providing external information about the accuracy of judgments would modulate serial dependence. Providing feedback significantly increased the attractive serial dependence effect, suggesting that external information at the level of judgment may modulate the weight of past perceptual information during the processing of the current image. Overall, our results support the idea that, although serial dependence may operate at a perceptual level, it originates from high-level processing stages at the level of abstract information processing and at the level of judgment.

Serial dependence tracks objects and scenes in parallel and independently

The visual world is made up of objects and scenes. Object perception requires both discriminating an individual object from others and binding together different perceptual samples of that object across time. Such binding manifests by serial dependence, the attraction of the current perception of a visual attribute toward values of that attribute seen in the recent past. Scene perception is subserved by global mechanisms such as ensemble perception, the rapid extraction of the average feature value of a group of objects. The current study examined to what extent the perception of single objects in multi-object scenes depended on previous feature values of that object or on the average previous attribute of all objects in the ensemble. Results show that serial dependence occurs independently on two simultaneously present objects, that ensemble perception depends only on previous ensembles, and that serial dependence of an individual object occurs only for the features of that particular object. These results suggest that the temporal integration of successive perceptual samples operates simultaneously at independent levels of visual processing.

Working Memory Content Is Distorted by Its Use in Perceptual Comparisons

Visual information around us is rarely static. To perform a task in such a dynamic environment, we often have to compare current visual input with our working memory (WM) representation of the immediate past. However, little is known about what happens to a WM representation when it is compared with perceptual input. To test this, we asked young adults (N = 170 total in three experiments) to compare a new visual input with a WM representation prior to reporting the WM representation. We found that the perceptual comparison biased the WM report, especially when the input was subjectively similar to the WM representation. Furthermore, using computational modeling and individual-differences analyses, we found that this similarity-induced memory bias was driven by representational integration, rather than incidental confusion, between the WM representation and subjectively similar input. Together, our findings highlight a novel source of WM distortion and suggest a general mechanism that determines how WM interacts with new visual input.

You see what you look for: Targets and distractors in visual search can cause opposing serial dependencies

Visual perception is, at any given moment, strongly influenced by its temporal context-what stimuli have recently been perceived and in what surroundings. We have previously shown that to-be-ignored items produce a bias upon subsequent perceptual decisions that acts in parallel with other biases induced by attended items. However, our previous investigations were confined to biases upon the perceived orientation of a visual search target, and it is unclear whether these biases influence perceptual decisions in a more general sense. Here, we test whether the biases from visual search targets and distractors affect the perceived orientation of a neutral test line, one that is neither a target nor a distractor. To do so, we asked participants to search for an oddly oriented line among distractors and report its location for a few trials and next presented a test line irrelevant to the search task. Participants were asked to report the orientation of the test line. Our results indicate that in tasks involving visual search, targets induce a positive bias upon a neutral test line if their orientations are similar, whereas distractors produce an attractive bias for similar test lines and a repulsive bias if the orientations of the test line and the average orientation of the distractors are far apart in feature space. In sum, our results show that both attentional role and proximity in feature space between previous and current stimuli determine the direction of biases in perceptual decisions.

Capturing the objects of vision with neural networks

Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioural studies have documented how object representations emerge through grouping, amodal completion, proto-objects and object files. By contrast, deep neural network models of visual object recognition remain largely tethered to sensory input, despite achieving human-level performance at labelling objects. Here, we review related work in both fields and examine how these fields can help each other. The cognitive literature provides a starting point for the development of new experimental tasks that reveal mechanisms of human object perception and serve as benchmarks driving the development of deep neural network models that will put the object into object recognition.

Temporal dynamics of implicit memory underlying serial dependence

Serial dependence is the effect in which the immediately preceding trial influences participants' responses to the current stimulus. But for how long does this bias last in the absence of interference from other stimuli? Here, we had 20 healthy young adult participants (12 women) perform a coincident timing task using different inter-trial intervals to characterize the serial dependence effect as the time between trials increases. Our results show that serial dependence abruptly decreases from 0.1 s to 1 s inter-trial interval, but it remains pronounced after that for up to 8 s. In addition, participants' response variability slightly decreases over longer intervals. We discuss these results in light of recent models suggesting that serial dependence might rely on a short-term memory trace kept through changes in synaptic weights, which might explain its long duration and apparent stability over time.

Spontaneous perception: a framework for task-free, self-paced perception

Flipping through social media feeds, viewing exhibitions in a museum, or walking through the botanical gardens, people consistently choose to engage with and disengage from visual content. Yet, in most laboratory settings, the visual stimuli, their presentation duration, and the task at hand are all controlled by the researcher. Such settings largely overlook the spontaneous nature of human visual experience, in which perception takes place independently from specific task constraints and its time course is determined by the observer as a self-governing agent. Currently, much remains unknown about how spontaneous perceptual experiences unfold in the brain. Are all perceptual categories extracted during spontaneous perception? Does spontaneous perception inherently involve volition? Is spontaneous perception segmented into discrete episodes? How do different neural networks interact over time during spontaneous perception? These questions are imperative to understand our conscious visual experience in daily life. In this article we propose a framework for spontaneous perception. We first define spontaneous perception as a task-free and self-paced experience. We propose that spontaneous perception is guided by four organizing principles that grant it temporal and spatial structures. These principles include coarse-to-fine processing, continuity and segmentation, agency and volition, and associative processing. We provide key suggestions illustrating how these principles may interact with one another in guiding the multifaceted experience of spontaneous perception. We point to testable predictions derived from this framework, including (but not limited to) the roles of the default-mode network and slow cortical potentials in underlying spontaneous perception. We conclude by suggesting several outstanding questions for future research, extending the relevance of this framework to consciousness and spontaneous brain activity. In conclusion, the spontaneous perception framework proposed herein integrates components in human perception and cognition, which have been traditionally studied in isolation, and opens the door to understand how visual perception unfolds in its most natural context.

Sequence structure organizes items in varied latent states of working memory neural network

In memory experiences, events do not exist independently but are linked with each other via structure-based organization. Structure context largely influences memory behavior, but how it is implemented in the brain remains unknown. Here, we combined magnetoencephalogram (MEG) recordings, computational modeling, and impulse-response approaches to probe the latent states when subjects held a list of items in working memory (WM). We demonstrate that sequence context reorganizes WM items into distinct latent states, that is, being reactivated at different latencies during WM retention, and the reactivation profiles further correlate with recency behavior. In contrast, memorizing the same list of items without sequence task requirements weakens the recency effect and elicits comparable neural reactivations. Computational modeling further reveals a dominant function of sequence context, instead of passive memory decaying, in characterizing recency effect. Taken together, sequence structure context shapes the way WM items are stored in the human brain and essentially influences memory behavior.

Transsaccadic integration operates independently in different feature dimensions

Our knowledge about objects in our environment reflects an integration of current visual input with information from preceding gaze fixations. Such a mechanism may reduce uncertainty but requires the visual system to determine which information obtained in different fixations should be combined or kept separate. To investigate the basis of this decision, we conducted three experiments. Participants viewed a stimulus in their peripheral vision and then made a saccade that shifted the object into the opposite hemifield. During the saccade, the object underwent changes of varying magnitude in two feature dimensions (Experiment 1, color and location; Experiments 2 and 3, color and orientation). Participants reported whether they detected any change and estimated one of the postsaccadic features. Integration of presaccadic with postsaccadic input was observed as a bias in estimates toward the presaccadic feature value. In all experiments, presaccadic bias weakened as the magnitude of the transsaccadic change in the estimated feature increased. Changes in the other feature, despite having a similar probability of detection, had no effect on integration. Results were quantitatively captured by an observer model where the decision whether to integrate information from sequential fixations is made independently for each feature and coupled to awareness of a feature change.

Decoding Concurrent Representations of Pitch and Location in Auditory Working Memory

Multivariate analyses of hemodynamic signals serve to identify the storage of specific stimulus contents in working memory (WM). Representations of visual stimuli have been demonstrated both in sensory regions and in higher cortical areas. While previous research has typically focused on the WM maintenance of a single content feature, it remains unclear whether two separate features of a single object can be decoded concurrently. Also, much less evidence exists for representations of auditory compared with visual stimulus features. To address these issues, human participants had to memorize both pitch and perceived location of one of two sample sounds. After a delay phase, they were asked to reproduce either pitch or location. At recall, both features showed comparable levels of discriminability. Region of interest (ROI)-based decoding of functional magnetic resonance imaging (fMRI) data during the delay phase revealed feature-selective activity for both pitch and location of a memorized sound in auditory cortex and superior parietal lobule. The latter region showed higher decoding accuracy for location than pitch. In addition, location could be decoded from angular and supramarginal gyrus and both superior and inferior frontal gyrus. The latter region also showed a trend for decoding of pitch. We found no region exclusively coding pitch memory information. In summary, the present study yielded evidence for concurrent representations of pitch and location of a single object both in sensory cortex and in hierarchically higher regions, pointing toward representation formats that enable feature integration within the same anatomic brain regions.SIGNIFICANCE STATEMENT Decoding of hemodynamic signals serves to identify brain regions involved in the storage of stimulus-specific information in working memory (WM). While to-be-remembered information typically consists of several features, most previous investigations have focused on the maintenance of one memorized feature belonging to one visual object. The present study assessed the concurrent storage of two features of the same object in auditory WM. We found that both pitch and location of memorized sounds were decodable both in early sensory areas, in higher-level superior parietal cortex and, to a lesser extent, in inferior frontal cortex. While auditory cortex is known to process different features in parallel, their concurrent representation in parietal regions may support the integration of object features in WM.

Serial dependence in time and numerosity perception is dimension-specific

The perception of a visual event (e.g., a flock of birds) at the present moment can be biased by a previous perceptual experience (e.g., the perception of an earlier flock). Serial dependence is a perceptual bias whereby a current stimulus appears more similar to a previous one than it actually is. Whereas serial dependence emerges within several visual stimulus dimensions, whether it could simultaneously operate across different dimensions of the same stimulus (e.g., the numerosity and the duration of a visual pattern) remains unclear. Here we address this question by assessing the presence of serial dependence across duration and numerosity, two stimulus dimensions that are often associated and can bias each other. Participants performed either a duration or a numerosity discrimination task, in which they compared a constant reference with a variable test stimulus, varying along the task-relevant dimension (either duration or numerosity). Serial dependence was induced by a task-irrelevant inducer, that is, a stimulus presented before the reference and always varying in both duration and numerosity. The results show systematic serial dependencies only within the task-relevant stimulus dimension, that is, stimulus numerosity affects numerosity perception only, and duration affects duration perception only. Additionally, at least in the numerosity condition, the task-irrelevant dimension of the inducer (duration) had an opposite, repulsive effect. These findings thus show that attractive serial dependence operates in a highly specific fashion and does not transfer across different stimulus dimensions. Instead, the repulsive influence, possibly reflecting perceptual adaptation, can transfer from one dimension to another.

Keeping it real: Looking beyond capacity limits in visual cognition

Research within visual cognition has made tremendous strides in uncovering the basic operating characteristics of the visual system by reducing the complexity of natural vision to artificial but well-controlled experimental tasks and stimuli. This reductionist approach has for example been used to assess the basic limitations of visual attention, visual working memory (VWM) capacity, and the fidelity of visual long-term memory (VLTM). The assessment of these limits is usually made in a pure sense, irrespective of goals, actions, and priors. While it is important to map out the bottlenecks our visual system faces, we focus here on selected examples of how such limitations can be overcome. Recent findings suggest that during more natural tasks, capacity may be higher than reductionist research suggests and that separable systems subserve different actions, such as reaching and looking, which might provide important insights about how pure attentional or memory limitations could be circumvented. We also review evidence suggesting that the closer we get to naturalistic behavior, the more we encounter implicit learning mechanisms that operate "for free" and "on the fly." These mechanisms provide a surprisingly rich visual experience, which can support capacity-limited systems. We speculate whether natural tasks may yield different estimates of the limitations of VWM, VLTM, and attention, and propose that capacity measurements should also pass the real-world test within naturalistic frameworks. Our review highlights various approaches for this and suggests that our understanding of visual cognition will benefit from incorporating the complexities of real-world cognition in experimental approaches.

Perceptual decisions are biased toward relevant prior choices

Perceptual decisions are biased by recent perceptual history-a phenomenon termed 'serial dependence.' Here, we investigated what aspects of perceptual decisions lead to serial dependence, and disambiguated the influences of low-level sensory information, prior choices and motor actions. Participants discriminated whether a brief visual stimulus lay to left/right of the screen center. Following a series of biased 'prior' location discriminations, subsequent 'test' location discriminations were biased toward the prior choices, even when these were reported via different motor actions (using different keys), and when the prior and test stimuli differed in color. By contrast, prior discriminations about an irrelevant stimulus feature (color) did not substantially influence subsequent location discriminations, even though these were reported via the same motor actions. Additionally, when color (not location) was discriminated, a bias in prior stimulus locations no longer influenced subsequent location discriminations. Although low-level stimuli and motor actions did not trigger serial-dependence on their own, similarity of these features across discriminations boosted the effect. These findings suggest that relevance across perceptual decisions is a key factor for serial dependence. Accordingly, serial dependence likely reflects a high-level mechanism by which the brain predicts and interprets new incoming sensory information in accordance with relevant prior choices.

Perceptual history propagates down to early levels of sensory analysis

One function of perceptual systems is to construct and maintain a reliable representation of the environment. A useful strategy intrinsic to modern “Bayesian” theories of perception1, 2, 3, 4, 5, 6 is to take advantage of the relative stability of the input and use perceptual history (priors) to predict current perception. This strategy is efficient1, 2, 3, 4, 5, 6, 7 but can lead to stimuli being biased toward perceptual history, clearly revealed in a phenomenon known as serial dependence.8, 9, 10, 11, 12, 13, 14 However, it is still unclear whether serial dependence biases sensory encoding or only perceptual decisions.^15,16 We leveraged on the “surround tilt illusion”—where tilted flanking stimuli strongly bias perceived orientation—to measure its influence on the pattern of serial dependence, which is typically maximal for similar orientations of past and present stimuli.^7,10 Maximal serial dependence for a neutral stimulus preceded by an illusory one occurred when the perceived, not the physical, orientations of the two stimuli matched, suggesting that the priors biasing current perception incorporate the effect of the illusion. However, maximal serial dependence of illusory stimuli induced by neutral stimuli occurred when their physical (not perceived) orientations were matched, suggesting that priors interact with incoming sensory signals before they are biased by flanking stimuli. The evidence suggests that priors are high-level constructs incorporating contextual information, which interact directly with early sensory signals, not with highly processed perceptual representations.

•

Perception is heavily biased by perceptual history and expectations

•

Perceptual history includes illusory effects driven by spatial context

•

This representation propagates back to sensory areas preceding context effects

•

The results point to a neural architecture consistent with predictive coding

A Bayesian and efficient observer model explains concurrent attractive and repulsive history biases in visual perception

Human perceptual decisions can be repelled away from (repulsive adaptation) or attracted towards recent visual experience (attractive serial dependence). It is currently unclear whether and how these repulsive and attractive biases interact during visual processing and what computational principles underlie these history dependencies. Here we disentangle repulsive and attractive biases by exploring their respective timescales. We find that perceptual decisions are concurrently attracted towards the short-term perceptual history and repelled from stimuli experienced up to minutes into the past. The temporal pattern of short-term attraction and long-term repulsion cannot be captured by an ideal Bayesian observer model alone. Instead, it is well captured by an ideal observer model with efficient encoding and Bayesian decoding of visual information in a slowly changing environment. Concurrent attractive and repulsive history biases in perceptual decisions may thus be the consequence of the need for visual processing to simultaneously satisfy constraints of efficiency and stability.