Feature integration across space, time, and orientation

Interference across time: dissociating short from long temporal interference

Our ability to identify an object is often impaired by the presence of preceding and/or succeeding task-irrelevant items. Understanding this temporal interference is critical for any theoretical account of interference across time and for minimizing its detrimental effects. Therefore, we used the same sequences of 3 orientation items, orientation estimation task, and computational models, to examine temporal interference over both short (<150 ms; visual masking) and long (175-475 ms; temporal crowding) intervals. We further examined how inter-item similarity modifies these different instances of temporal interference. Qualitatively different results emerged for interference of different scales. Interference over long intervals mainly degraded the precision of the target encoding while interference over short intervals mainly affected the signal-to-noise ratio. Although both interference instances modulated substitution errors (reporting a wrong item) and were alleviated with dissimilar items, their characteristics were markedly disparate. These findings suggest that different mechanisms mediate temporal interference of different scales.

Processing load, and not stimulus evidence, determines the duration of unconscious visual feature integration

Integration across space and time is essential for the analysis of motion, low contrast, and many more stimuli. A crucial question is what determines the duration of integration. Based on classical models of decision-making, one might expect that integration terminates as soon as sufficient evidence about a stimulus is accumulated and a threshold is crossed. However, there is very little research on this question as most experimental paradigms cannot monitor processing following stimulus presentation. In particular, it is difficult to determine when processing terminates. Here, using the sequential metacontrast paradigm (SQM), in which information is mandatorily integrated along motion trajectories, we show that the processing load determines the extent of integration but that evidence accumulation does not. Further, the extent of integration is determined by absolute time instead of the number of elements presented. These results have important implications for understanding the time course and mechanisms of temporal integration.

Information Integration and Information Storage in Retinotopic and Non-Retinotopic Sensory Memory

The first stage of the Atkinson–Shiffrin model of human memory is a sensory memory (SM). The visual component of the SM was shown to operate within a retinotopic reference frame. However, a retinotopic SM (rSM) is unable to account for vision under natural viewing conditions because, for example, motion information needs to be analyzed across space and time. For this reason, the SM store of the Atkinson–Shiffrin model has been extended to include a non-retinotopic component (nrSM). In this paper, we analyze findings from two experimental paradigms and show drastically different properties of rSM and nrSM. We show that nrSM involves complex processes such as motion-based reference frames and Gestalt grouping, which establish object identities across space and time. We also describe a quantitative model for nrSM and show drastic differences between the spatio-temporal properties of rSM and nrSM. Since the reference-frame of the latter is non-retinotopic and motion-stream based, we suggest that the spatiotemporal properties of the nrSM are in accordance with the spatiotemporal properties of the motion system. Overall, these findings indicate that, unlike the traditional rSM, which is a relatively passive store, nrSM exhibits sophisticated processing properties to manage the complexities of ecological perception.

Features integrate along a motion trajectory when object integrity is preserved

Information about a moving object is usually poor at each retinotopic location because photoreceptor activation is short, noisy, and affected by shadows, reflections of other objects, and so on. Integration across the motion trajectory may yield a much better estimate about the objects’ features. Using the sequential metacontrast paradigm, we have shown previously that features, indeed, integrate along a motion trajectory in a long-lasting window of unconscious processing. In the sequential metacontrast paradigm, a percept of two diverging streams is elicited by the presentation of a central line followed by a sequence of flanking pairs of lines. When several lines are spatially offset, the offsets integrate mandatorily for several hundreds of milliseconds along the motion trajectory of the streams. We propose that, within these long-lasting windows, stimuli are first grouped based on Gestalt principles of grouping. These processes establish reference frames that are used to attribute features. Features are then integrated following their respective reference frame. Here using occlusion and bouncing effects, we show that indeed such grouping operations are in place. We found that features integrate only when the spatiotemporal integrity of the object is preserved. Moreover, when several moving objects are present, only features belonging to the same object integrate. Overall, our results show that feature integration is a deliberate strategy of the brain and long-lasting windows of processing can be seen as periods of sense making.

Temporal crowding is a unique phenomenon reflecting impaired target encoding over large temporal intervals

Crowding refers to impaired object identification when presented with other objects, and it is well established that spatial crowding-crowding from adjacent objects-affects many aspects of visual perception and cognition. A similar interference also occurs across time-the identification of a target object is impaired when distracting objects precede and succeed it. When such interference is observed with relatively long interitem intervals it is termed temporal crowding. Thus far, little was known about temporal crowding and its underlying processes. Particularly it was unknown which aspects of visual processing are impaired by temporal crowding, and the answer to this question bears critical theoretical implications. To reveal the nature of this impairment we used a continuous-report task and a mixture-model analysis. In three experiments, observers viewed sequences of three oriented items separated by relatively long intervals (170-475ms). The target was the second item in the sequence, and the task was to reproduce its orientation. The findings suggest that temporal crowding impairs target encoding and increases substitution errors, but there was no evidence of a reduced signal-to-noise ratio. This pattern of results was similar regardless of stimuli duration and target-distractor similarity. However, it differed considerably from the pattern found for ordinary masking and spatial crowding, indicating that temporal crowding is a unique phenomenon. Moreover, the finding that temporal crowding affected the precision of target encoding even when the items were separated by almost half a second suggests that visual processing requires a surprisingly long time to complete.

All in Good Time: Long-Lasting Postdictive Effects Reveal Discrete Perception

Is consciousness a continuous stream of percepts or is it discrete, occurring only at certain moments in time? This question has puzzled philosophers, psychologists, and neuroscientists for centuries. Both hypotheses have fallen repeatedly in and out of favor. Here, we review recent studies exploring long-lasting postdictive effects and show that the results favor a two-stage discrete model, in which substantial periods of continuous unconscious processing precede discrete conscious percepts. We propose that such a model marries the advantages of both continuous and discrete models and resolves centuries old debates about perception and consciousness.

Object identity determines trans-saccadic integration

Humans make two to four rapid eye movements (saccades) per second, which, surprisingly, does not lead to abrupt changes in vision. To the contrary, we perceive a stable world. Hence, an important question is how information is integrated across saccades. To investigate this question, we used the sequential metacontrast paradigm (SQM), where two expanding streams of lines are presented. When one line is spatially offset, the other lines are perceived as being offset, too. When more lines are offset, all offsets integrate mandatorily; that is, observers cannot report the individual offsets but perceive one integrated offset. Here, we asked observers to make a saccade during the SQM. Even though the saccades caused a highly disrupted motion trajectory on the retina, offsets presented before and after the saccade integrated mandatorily. When observers made no saccade and the streams were displaced on the screen so that a similarly disrupted retinal image occurred as in the previous condition, no integration occurred. We suggest that trans-saccadic integration and perception are determined by object identity in spatiotopic coordinates and not by the retinal image.

Feature integration within discrete time windows

Sensory information must be integrated over time to perceive, for example, motion and melodies. Here, to study temporal integration, we used the sequential metacontrast paradigm in which two expanding streams of lines are presented. When a line in one stream is offset observers perceive all other lines to be offset too, even though they are straight. When more lines are offset the offsets integrate mandatorily, i.e., observers cannot report the individual offsets. We show that mandatory integration lasts for up to 450 ms, depending on the observer. Importantly, integration occurs only when offsets are presented within a discrete window of time. Even stimuli that are in close spatio-temporal proximity do not integrate if they are in different windows. A window of integration starts with stimulus onset and integration in the next window has similar characteristics. We present a two-stage computational model based on discrete time windows that captures these effects.

In order to perceive moving or changing objects, sensory information must be integrated over time. Here, using a visual sequential metacontrast paradigm, the authors show that integration occurs only when subsequent stimuli are presented within a discrete window of time after the initial stimulus.

Visual percepts modify iconic memory in humans

Our visual system briefly retains a trace of a stimulus after it disappears. This phenomenon is known as iconic memory and its contents are thought to be temporally integrated with subsequent visual inputs to produce a single composite representation. However, there is little consensus on the temporal integration between iconic memory and subsequent visual inputs. Here, we show that iconic memory revises its contents depending upon the configuration of the newly produced single representation with particular temporal characteristics. The Poggendorff illusion, in which two collinear line segments are perceived as non-collinear by an intervening rectangle, was observed when the rectangle was presented during a period spanning from 50 ms before to 200 ms after the presentation of the line segments. The illusion was most prominent when the rectangle was presented approximately 100 to 150 ms after the line segments. Furthermore, the illusion was observed at the center of a moving object, but only when the line segments were presented before the rectangle. These results indicate that the contents of iconic memory are susceptible to the modulatory influence of subsequent visual inputs before being translated into conscious perception in a time-locked manner both in retinotopic and non-retinotopic, object-centered frames of reference.

Perceptual episodes, temporal attention, and the role of cognitive control: Lessons from the attentional blink

The ability to identify a target is usually hindered if it appears shortly after another target. This simple and somewhat intuitive observation is qualified by a multitude of unexpected findings and conflicting theories that originate from the attentional blink paradigm. In this review, the major results, implications, and outstanding questions that stem from the paradigm are presented and discussed. The extant literature suggests that when the temporal domain is densely stacked with numerous stimuli, the entities that underlie attentional selection and cognitive control are brief perceptual episodes. Specifically, attention is deployed over an interval that frequently encompasses several stimuli. Most theories agree that the length and boundaries of this interval are influenced by cognitive control mechanisms. However, there is little agreement as to the extent and nature of this influence. Some theories suggest that control is needed in order to initiate a temporally limited attentional response. Other theories argue that cognitive control is actively suppressing attentional mechanisms in order to terminate the perceptual episode. Another formulation suggests that both ends of the interval are partially controlled and that the exertion of control corresponds to the focusing of attention on a narrow interval. The contents of perceptual episodes, as well as their deficiencies, can shed light on the features that guide attentional deployment, the goals that guide cognitive control, and the interactions between these mechanisms. Electrophysiological recordings are extremely useful when one tries to pinpoint the timing of attentional selection. Other neural indicators can elucidate the factors that define perceptual episodes.

A New Conceptualization of Human Visual Sensory-Memory

Memory is an essential component of cognition and disorders of memory have significant individual and societal costs. The Atkinson–Shiffrin “modal model” forms the foundation of our understanding of human memory. It consists of three stores: Sensory Memory (SM), whose visual component is called iconic memory, Short-Term Memory (STM; also called working memory, WM), and Long-Term Memory (LTM). Since its inception, shortcomings of all three components of the modal model have been identified. While the theories of STM and LTM underwent significant modifications to address these shortcomings, models of the iconic memory remained largely unchanged: A high capacity but rapidly decaying store whose contents are encoded in retinotopic coordinates, i.e., according to how the stimulus is projected on the retina. The fundamental shortcoming of iconic memory models is that, because contents are encoded in retinotopic coordinates, the iconic memory cannot hold any useful information under normal viewing conditions when objects or the subject are in motion. Hence, half-century after its formulation, it remains an unresolved problem whether and how the first stage of the modal model serves any useful function and how subsequent stages of the modal model receive inputs from the environment. Here, we propose a new conceptualization of human visual sensory memory by introducing an additional component whose reference-frame consists of motion-grouping based coordinates rather than retinotopic coordinates. We review data supporting this new model and discuss how it offers solutions to the paradoxes of the traditional model of sensory memory.

Spatiotemporal Form Integration: sequentially presented inducers can lead to representations of stationary and rigidly rotating objects

Objects in the world often are occluded and in motion. The visible fragments of such objects are revealed at different times and locations in space. To form coherent representations of the surfaces of these objects, the visual system must integrate local form information over space and time. We introduce a new illusion in which a rigidly rotating square is perceived on the basis of sequentially presented Pacman inducers. The illusion highlights two fundamental processes that allow us to perceive objects whose form features are revealed over time: Spatiotemporal Form Integration (STFI) and Position Updating. STFI refers to the spatial integration of persistent representations of local form features across time. Position updating of these persistent form representations allows them to be integrated into a rigid global motion percept. We describe three psychophysical experiments designed to identify spatial and temporal constraints that underlie these two processes and a fourth experiment that extends these findings to more ecologically valid stimuli. Our results indicate that although STFI can occur across relatively long delays between successive inducers (i.e., greater than 500 ms), position updating is limited to a more restricted temporal window (i.e., ~300 ms or less), and to a confined range of spatial (mis)alignment. These findings lend insight into the limits of mechanisms underlying the visual system's capacity to integrate transient, piecemeal form information, and support coherent object representations in the ever-changing environment.

Extrastriate Visual Areas Integrate Form Features over Space and Time to Construct Representations of Stationary and Rigidly Rotating Objects

When an object moves behind a bush, for example, its visible fragments are revealed at different times and locations across the visual field. Nonetheless, a whole moving object is perceived. Unlike traditional modal and amodal completion mechanisms known to support spatial form integration when all parts of a stimulus are simultaneously visible, relatively little is known about the neural substrates of the spatiotemporal form integration (STFI) processes involved in generating coherent object representations from a succession visible fragments. We used fMRI to identify brain regions involved in two mechanisms supporting the representation of stationary and rigidly rotating objects whose form features are shown in succession: STFI and position updating. STFI allows past and present form cues to be integrated over space and time into a coherent object even when the object is not visible in any given frame. STFI can occur whether or not the object is moving. Position updating allows us to perceive a moving object, whether rigidly rotating or translating, even when its form features are revealed at different times and locations in space. Our results suggest that STFI is mediated by visual regions beyond V1 and V2. Moreover, although widespread cortical activation has been observed for other motion percepts derived solely from form-based analyses [Tse, P. U. Neural correlates of transformational apparent motion. Neuroimage, 31, 766-773, 2006; Krekelberg, B., Vatakis, A., & Kourtzi, Z. Implied motion from form in the human visual cortex. Journal of Neurophysiology, 94, 4373-4386, 2005], increased responses for the position updating that lead to rigidly rotating object representations were only observed in visual areas KO and possibly hMT+, indicating that this is a distinct and highly specialized type of processing.

The fate of visible features of invisible elements

To investigate the integration of features, we have developed a paradigm in which an element is rendered invisible by visual masking. Still, the features of the element are visible as part of other display elements presented at different locations and times (sequential metacontrast). In this sense, we can "transport" features non-retinotopically across space and time. The features of the invisible element integrate with features of other elements if and only if the elements belong to the same spatio-temporal group. The mechanisms of this kind of feature integration seem to be quite different from classical mechanisms proposed for feature binding. We propose that feature processing, binding, and integration occur concurrently during processes that group elements into wholes.

Perceptual learning in a nonretinotopic frame of reference

Perceptual learning is the ability to improve perception through practice. Perceptual learning is usually specific for the task and features learned. For example, improvements in performance for a certain stimulus do not transfer if the stimulus is rotated by 90 degrees or is presented at a different location. These findings are usually taken as evidence that orientation-specific, retinotopic encoding processes are changed during training. In this study, we used a novel masking paradigm in which the offset in an invisible, oblique vernier stimulus was perceived in an aligned vertical or horizontal flanking stimulus presented at a different location. Our results show that learning is specific for the perceived orientation of the vernier offset but not for its actual orientation and location. Specific encoding processes cannot be invoked to explain this improvement. We propose that perceptual learning involves changes in nonretinotopic, attentional readout processes.