Bottlenecks of motion processing during a visual glance: the leaky flask model

The perceptual timescape: Perceptual history on the sub-second scale

There is a high-capacity store of brief time span (∼1000 ms) which information enters from perceptual processing, often called iconic memory or sensory memory. It is proposed that a main function of this store is to hold recent perceptual information in a temporally segregated representation, named the perceptual timescape. The perceptual timescape is a continually active representation of change and continuity over time that endows the perceived present with a perceived history. This is accomplished primarily by two kinds of time marking information: time distance information, which marks all items of information in the perceptual timescape according to how far in the past they occurred, and ordinal temporal information, which organises items of information in terms of their temporal order. Added to that is information about connectivity of perceptual objects over time. These kinds of information connect individual items over a brief span of time so as to represent change, persistence, and continuity over time. It is argued that there is a one-way street of information flow from perceptual processing either to the perceived present or directly into the perceptual timescape, and thence to working memory. Consistent with that, the information structure of the perceptual timescape supports postdictive reinterpretations of recent perceptual information. Temporal integration on a time scale of hundreds of milliseconds takes place in perceptual processing and does not draw on information in the perceptual timescape, which is concerned with temporal segregation, not integration.

Capacity and Allocation across Sensory and Short-Term Memories

Human memory consists of sensory memory (SM), short-term memory (STM), and long-term memory (LTM). SM enables a large capacity, but decays rapidly. STM has limited capacity, but lasts longer. The traditional view of these memory systems resembles a leaky hourglass, the large top and bottom portions representing the large capacities of SM and LTM, whereas the narrow portion in the middle represents the limited capacity of STM. The “leak” in the top part of the hourglass depicts the rapid decay of the contents of SM. However, recently, it was shown that major bottlenecks for motion processing exist prior to STM, and the “leaky hourglass” model was replaced by a “leaky flask” model with a narrower top part to capture bottlenecks prior to STM. The leaky flask model was based on data from one study, and the first goal of the current paper was to test if the leaky flask model would generalize by using a different set of data. The second goal of the paper was to explore various block diagram models for memory systems and determine the one best supported by the data. We expressed these block diagram models in terms of statistical mixture models and, by using the Bayesian information criterion (BIC), found that a model with four components, viz., SM, attention, STM, and guessing, provided the best fit to our data. In summary, we generalized previous findings about early qualitative and quantitative bottlenecks, as expressed in the leaky flask model and showed that a four-process model can provide a good explanation for how visual information is processed and stored in memory.

The Short-Term Retention of Depth

We review research on the visual working memory for information portrayed by items arranged in depth (i.e., distance to the observer) within peri-personal space. Most items lose their metric depths within half a second, even though their identities and spatial positions are retained. The paradoxical loss of depth information may arise because visual working memory retains the depth of a single object for the purpose of actions such as pointing or grasping which usually apply to only one thing at a time.

Perception of Happening: How the Brain Deals with the No-History Problem

In physics, the temporal dimension has units of infinitesimally brief duration. Given this, how is it possible to perceive things, such as motion, music, and vibrotactile stimulation, that involve extension across many units of time? To address this problem, it is proposed that there is what is termed an "information construct of happening" (ICOH), a simultaneous representation of recent, temporally differentiated perceptual information on the millisecond time scale. The main features of the ICOH are (i) time marking, semantic labeling of all information in the ICOH with ordinal temporal information and distance from what is informationally identified as the present moment, (ii) vector informational features that specify kind, direction, and rate of change for every feature in a percept, and (iii) connectives, information relating vector informational features at adjacent temporal locations in the ICOH. The ICOH integrates products of perceptual processing with recent historical information in sensory memory on the subsecond time scale. Perceptual information about happening in informational sensory memory is encoded in semantic form that preserves connected semantic trails of vector and timing information. The basic properties of the ICOH must be supported by a general and widespread timing mechanism that generates ordinal and interval timing information and it is suggested that state-dependent networks may suffice for that purpose. Happening, therefore, is perceived at a moment and is constituted by an information structure of connected recent historical information.

The extended present: an informational context for perception

Several previous authors have proposed a kind of specious or subjective present moment that covers a few seconds of recent information. This article proposes a new hypothesis about the subjective present, renamed the extended present, defined not in terms of time covered but as a thematically connected information structure held in working memory and in transiently accessible form in long-term memory. The three key features of the extended present are that information in it is thematically connected, both internally and to current attended perceptual input, it is organised in a hierarchical structure, and all information in it is marked with temporal information, specifically ordinal and duration information. Temporal boundaries to the information structure are determined by hierarchical structure processing and by limits on processing and storage capacity. Supporting evidence for the importance of hierarchical structure analysis is found in the domains of music perception, speech and language processing, perception and production of goal-directed action, and exact arithmetical calculation. Temporal information marking is also discussed and a possible mechanism for representing ordinal and duration information on the time scale of the extended present is proposed. It is hypothesised that the extended present functions primarily as an informational context for making sense of current perceptual input, and as an enabler for perception and generation of complex structures and operations in language, action, music, exact calculation, and other domains.

The perceived present: What is it, and what is it there for?

It is proposed that the perceived present is not a moment in time, but an information structure comprising an integrated set of products of perceptual processing. All information in the perceived present carries an informational time marker identifying it as "present". This marker is exclusive to information in the perceived present. There are other kinds of time markers, such as ordinality ("this stimulus occurred before that one") and duration ("this stimulus lasted for 50 ms"). These are different from the "present" time marker and may be attached to information regardless of whether it is in the perceived present or not. It is proposed that the perceived present is a very short-term and very high-capacity holding area for perceptual information. The maximum holding time for any given piece of information is ~100 ms: This is affected by the need to balance the value of informational persistence for further processing against the problem of obsolescence of the information. The main function of the perceived present is to facilitate access by other specialized, automatic processes.

Reorganization of spatial configurations in visual working memory: A matter of set size?

Humans process single objects in relation to other simultaneously maintained objects in visual working memory. This interdependence is called spatial configuration. Humans are able to reorganize global spatial configurations into relevant partial configurations. We conducted three experiments investigating the process underlying reorganization by manipulating memory set size and the presence of configurations at retrieval. Participants performed a location change detection task for a single object probed at retrieval. At the beginning of each trial, participants memorized the locations of all objects (set size: 4, 8, 12, or 16). During maintenance, a valid retro cue highlighted the side containing the object probed at retrieval, thus enabling participants to reorganize the memorized global spatial configuration to the partial cued configuration. At retrieval, the object probed was shown together with either all objects (complete configuration; Experiment 1a), the cued objects only (congruent configuration; all Experiments), the non-cued objects only (incongruent configuration, all Experiments) or alone (no configuration; Experiment 1b). We observed reorganization of spatial configurations as indicated by a superior location change detection performance with a congruent partial configuration than an incongruent partial configuration across all three experiments. We also observed an overall decrease in accuracy with increasing set size. Most importantly, however, we did not find evidence for a reliable impairment of reorganization with increasing set size. We discuss these findings with regard to the memory representation underlying spatial configurations.

Effects of Exogenous and Endogenous Attention on Metacontrast Masking

To efficiently use its finite resources, the visual system selects for further processing only a subset of the rich sensory information. Visual masking and spatial attention control the information transfer from visual sensory-memory to visual short-term memory. There is still a debate whether these two processes operate independently or interact, with empirical evidence supporting both arguments. However, recent studies pointed out that earlier studies showing significant interactions between common-onset masking and attention suffered from ceiling and/or floor effects. Our review of previous studies reporting metacontrast-attention interactions revealed similar artifacts. Therefore, we investigated metacontrast-attention interactions by using an experimental paradigm, in which ceiling/floor effects were avoided. We also examined whether metacontrast masking is differently influenced by endogenous and exogenous attention. We analyzed mean absolute-magnitude of response-errors and their statistical distribution. When targets are masked, our results support the hypothesis that manipulations of the levels of metacontrast and of endogenous/exogenous attention have largely independent effects. Moreover, statistical modeling of the distribution of response-errors suggests weak interactions modulating the probability of "guessing" behavior for some observers in both types of attention. Nevertheless, our data suggest that any joint effect of attention and metacontrast can be adequately explained by their independent and additive contributions.

Sensory Memory Is Allocated Exclusively to the Current Event-Segment

The Atkinson-Shiffrin modal model forms the foundation of our understanding of human memory. It consists of three stores (Sensory Memory (SM), also called iconic memory, Short-Term Memory (STM), and Long-Term Memory (LTM)), each tuned to a different time-scale. Since its inception, the STM and LTM components of the modal model have undergone significant modifications, while SM has remained largely unchanged, representing a large capacity system funneling information into STM. In the laboratory, visual memory is usually tested by presenting a brief static stimulus and, after a delay, asking observers to report some aspect of the stimulus. However, under ecological viewing conditions, our visual system receives a continuous stream of inputs, which is segmented into distinct spatio-temporal segments, called events. Events are further segmented into event-segments. Here we show that SM is not an unspecific general funnel to STM but is allocated exclusively to the current event-segment. We used a Multiple-Object Tracking (MOT) paradigm in which observers were presented with disks moving in different directions, along bi-linear trajectories, i.e., linear trajectories, with a single deviation in direction at the mid-point of each trajectory. The synchronized deviation of all of the trajectories produced an event stimulus consisting of two event-segments. Observers reported the pre-deviation or the post-deviation directions of the trajectories. By analyzing observers' responses in partial- and full-report conditions, we investigated the involvement of SM for the two event-segments. The hallmarks of SM hold only for the current event segment. As the large capacity SM stores only items involved in the current event-segment, the need for event-tagging in SM is eliminated, speeding up processing in active vision. By characterizing how memory systems are interfaced with ecological events, this new model extends the Atkinson-Shiffrin model by specifying how events are stored in the first stage of multi-store memory systems.

Parallel consolidation into visual working memory results in reduced precision representations

Information can be consolidated into visual working memory in parallel, i.e. two items can be consolidated in the same time required to consolidate one. However, while motion direction items consolidated in parallel are encoded at a reduced precision, no such reduction has been reported for colour. Here we examine two possible explanations for the inconsistency between the phenomena associated with consolidating these features in parallel: i) that reduced precision can only be detected when more than two colour items are consolidated in parallel, or ii) that the exposure duration used in previous studies was too long, allowing observers serially consolidate items. Our results show that (like motion direction) colour items consolidated in parallel are encoded at a reduced precision and the critical feature for detecting this phenomenon is the exposure duration. Furthermore, we demonstrate that this process is limited to two items. These findings indicate a general principle of consolidation into visual working memory, that is, a trade-off between the number of items consolidated in parallel and the precision at which they are encoded.

Is conscious perception a series of discrete temporal frames?

This paper reviews proposals that conscious perception consists, in whole or part, of successive discrete temporal frames on the sub-second time scale, each frame containing information registered as simultaneous or static. Although the idea of discrete frames in conscious perception cannot be regarded as falsified, there are many problems. Evidence does not consistently support any proposed duration or range of durations for frames. EEG waveforms provide evidence of periodicity in brain activity, but not necessarily in conscious perception. Temporal properties of perceptual processes are flexible in response to competing processing demands, which is hard to reconcile with the relative inflexibility of regular frames. There are also problems concerning the definition of frames, the need for informational connections between frames, the means by which boundaries between frames are established, and the apparent requirement for a storage buffer for information awaiting entry to the next frame.

Metacontrast masking and attention do not interact

Visual masking and attention have been known to control the transfer of information from sensory memory to visual short-term memory. A natural question is whether these processes operate independently or interact. Recent evidence suggests that studies that reported interactions between masking and attention suffered from ceiling and/or floor effects. The objective of the present study was to investigate whether metacontrast masking and attention interact by using an experimental design in which saturation effects are avoided. We asked observers to report the orientation of a target bar randomly selected from a display containing either two or six bars. The mask was a ring that surrounded the target bar. Attentional load was controlled by set-size and masking strength by the stimulus onset asynchrony between the target bar and the mask ring. We investigated interactions between masking and attention by analyzing two different aspects of performance: (i) the mean absolute response errors and (ii) the distribution of signed response errors. Our results show that attention affects observers' performance without interacting with masking. Statistical modeling of response errors suggests that attention and metacontrast masking exert their effects by independently modulating the probability of "guessing" behavior. Implications of our findings for models of attention are discussed.

The reference frame for encoding and retention of motion depends on stimulus set size

The goal of this study was to investigate the reference frames used in perceptual encoding and storage of visual motion information. In our experiments, observers viewed multiple moving objects and reported the direction of motion of a randomly selected item. Using a vector-decomposition technique, we computed performance during smooth pursuit with respect to a spatiotopic (nonretinotopic) and to a retinotopic component and compared them with performance during fixation, which served as the baseline. For the stimulus encoding stage, which precedes memory, we found that the reference frame depends on the stimulus set size. For a single moving target, the spatiotopic reference frame had the most significant contribution with some additional contribution from the retinotopic reference frame. When the number of items increased (Set Sizes 3 to 7), the spatiotopic reference frame was able to account for the performance. Finally, when the number of items became larger than 7, the distinction between reference frames vanished. We interpret this finding as a switch to a more abstract nonmetric encoding of motion direction. We found that the retinotopic reference frame was not used in memory. Taken together with other studies, our results suggest that, whereas a retinotopic reference frame may be employed for controlling eye movements, perception and memory use primarily nonretinotopic reference frames. Furthermore, the use of nonretinotopic reference frames appears to be capacity limited. In the case of complex stimuli, the visual system may use perceptual grouping in order to simplify the complexity of stimuli or resort to a nonmetric abstract coding of motion information.

Hydrocortisone Counteracts Adverse Stress Effects on Dual-Task Performance by Improving Visual Sensory Processes

The impact of acute stress on executive processes is commonly attributed to glucocorticoid-induced disruptions of the pFC. However, the occipital cortex seems to express a higher density of glucocorticoid receptors. Consequently, acute stress effects on executive processes could as well be mediated by glucocorticoid (e.g., cortisol)-induced alterations of visual sensory processes. To investigate this alternative route of stress action by demarcating the effects of acute stress and cortisol on executive from those on visual sensory processes, 40 healthy young men completed a standardized stress induction (i.e., the Trier Social Stress Test) and control protocol in two consecutive sessions. In addition, they received either a placebo or hydrocortisone (0.12-mg/kg bodyweight) pill and processed a dual and a partial report task to assess their executive and visual sensory processing abilities, respectively. Hydrocortisone administration improved both partial report and dual-task performance as indicated by increased response accuracies and/or decreased RTs. Intriguingly, the hydrocortisone-induced increase in dual-task performance was completely mediated by its impact on partial report performance (i.e., visual sensory processes). Moreover, RT measures in both tasks shared approximately 26% of variance, which was only in part attributable to hydrocortisone administration (ΔR² = 8%). By contrast, acute stress selectively impaired dual-task performance (i.e., executive processes), presumably through an alternative route of action. In summary, the present results suggest that cortisol secretion (as mimicked by hydrocortisone administration) may counteract adverse residual stress effects on executive processes by improving visual sensory processes (e.g., the maintenance and amplification of task-relevant sensory information).

A statistical perspective to visual masking

A stimulus (mask) reduces the visibility of another stimulus (target) when they are presented in close spatio-temporal vicinity of each other, a phenomenon called visual masking. Visual masking has been extensively studied to understand dynamics of information processing in the visual system. In this study, we adopted a statistical point of view, rather than a mechanistic one, to investigate how mask-related activities might influence target-related ones within the context of visual masking. We modeled the distribution of response errors of human observers in three different visual masking experiments, namely para-/meta-contrast masking, pattern masking by noise, and pattern masking by structure. We adopted statistical models, which have been used previously in studies of visual short-term memory, to capture response characteristics of observers under masking conditions. We tested the following scenarios: (i) mask activity may reduce a target's signal-to-noise ratio (SNR) without interfering with its encoding precision. (ii) Mask activity may "interfere" with the encoding of a target and cause decreased precision in observer's reports. (iii) Decreased performance due to masking may result from the confusion or "misbinding" of a mask's features with those of the target, when they are similar as in the case of pattern masking by structure. Our results show that in all three types of masking, the reduction of a target's SNR was the primary process whereby masking occurred. A significant decrease, correlated with masking strength, in the precision of the target's encoding was observed in para-/meta-contrast and pattern masking by structure, but not in pattern masking by noise. We interpret these findings as the mask reducing the target's SNR (i) by suppressing or interrupting the signal of the target in para-/meta-contrast, (ii) by increasing noise in pattern masking by noise, and (iii) a combination of the two in pattern masking by structure.

Different attentional abilities among inbred mice strains using virtual object recognition task (VORT): SNAP25⁺/⁻ mice as a model of attentional deficit

Autism spectrum disorder (ASD), attention-deficit hyperactivity disorder (ADHD), schizophrenia, Alzheimer's and Parkinson's disease are characterized by attentional deficits. In the present study we first applied the virtual object recognition test (VORT), where 3D objects were replaced with highly discriminated geometrical shapes and presented on two 3.5-inch widescreen displays, in different inbred mice strains (C57BL/6N, DBA/2J, BALB/cJ), in comparison with the standard object recognition test (NOR). In both NOR and VORT, there was a progressive decay of performance in terms of reduced discrimination index from 5 min to 72 h of inter-trial delay in all strains. However, BALB/cJ inbred mice showed a better long lasting performance than C57BL/6N and DBA/2J, when tested in NOR. In VORT, BALB/cJ showed the best performance. Total exploration time was always higher in BALB/cJ than C57BL/6N and DBA/2J mice. C57BL/6N were less explorative strain than DBA/2J and BALB/cJ mice. When VORT was applied to SNAP-25(+/-) mice, an impairment in both NOR and VORT was shown. However, when moving shapes were applied, these heterozygous mice improved their performance, suggesting that the introduction of motion is a strong cue that makes the task more valuable to study attention deficits. Taken together, these data indicate that VORT provides a useful and rapid tool to identify the attentional deficit in different inbred strains and genetically modified mice, enhancing the value of psychiatric mouse models.

Hydrocortisone accelerates the decay of iconic memory traces: on the modulation of executive and stimulus-driven constituents of sensory information maintenance

A substantial amount of research documents the impact of glucocorticoids on higher-order cognitive functioning. By contrast, surprisingly little is known about the susceptibility of basic sensory processes to glucocorticoid exposure given that the glucocorticoid receptor density in the human visual cortex exceeds those observed in prefrontal and most hippocampal brain regions. As executive tasks also rely on these sensory processes, the present study investigates the impact of glucocorticoid exposure on different performance parameters characterizing the maintenance and transfer of sensory information from iconic memory (IM; the sensory buffer of the visual system) to working memory (WM). Using a crossover factorial design, we administered one out of three doses of hydrocortisone (0.06, 0.12, or 0.24mg/kg bodyweight) and a placebo to 18 healthy young men. Thereafter participants performed a partial report task, which was used to assess their individual ability to process sensory information. Blood samples were concurrently drawn to determine free and total cortisol concentrations. The compiled pharmacokinetic and psychophysical data demonstrates that free cortisol specifically accelerated the decay of sensory information (r=0.46) without significantly affecting the selective information transfer from IM to WM or the capacity limit of WM. Specifically, nonparametric regression revealed a sigmoid dose-response relationship between free cortisol levels during the testing period and the IM decay rates. Our findings highlight that glucocorticoid exposure may not only impact on the recruitment of top-down control for an active maintenance of sensory information, but alter their passive (stimulus-driven) maintenance thereby changing the availability of information prior to subsequent executive processing.