Structure-from-motion: dissociating perception, neural persistence, and sensory memory of illusory depth and illusory rotation

Spontaneous Necker-cube reversals may not be that spontaneous

Introduction

During observation of the ambiguous Necker cube, our perception suddenly reverses between two about equally possible 3D interpretations. During passive observation, perceptual reversals seem to be sudden and spontaneous. A number of theoretical approaches postulate destabilization of neural representations as a pre-condition for reversals of ambiguous figures. In the current study, we focused on possible Electroencephalogram (EEG) correlates of perceptual destabilization, that may allow prediction of an upcoming perceptual reversal.

Methods

We presented ambiguous Necker cube stimuli in an onset-paradigm and investigated the neural processes underlying endogenous reversals as compared to perceptual stability across two consecutive stimulus presentations. In a separate experimental condition, disambiguated cube variants were alternated randomly, to exogenously induce perceptual reversals. We compared the EEG immediately before and during endogenous Necker cube reversals with corresponding time windows during exogenously induced perceptual reversals of disambiguated cube variants.

Results

For the ambiguous Necker cube stimuli, we found the earliest differences in the EEG between reversal trials and stability trials already 1 s before a reversal occurred, at bilateral parietal electrodes. The traces remained similar until approximately 1100 ms before a perceived reversal, became maximally different at around 890 ms (p = 7.59 × 10^-6, Cohen's d = 1.35) and remained different until shortly before offset of the stimulus preceding the reversal. No such patterns were found in the case of disambiguated cube variants.

Discussion

The identified EEG effects may reflect destabilized states of neural representations, related to destabilized perceptual states preceding a perceptual reversal. They further indicate that spontaneous Necker cube reversals are most probably not as spontaneous as generally thought. Rather, the destabilization may occur over a longer time scale, at least 1 s before a reversal event, despite the reversal event as such being perceived as spontaneous by the viewer.

Reconstructing a disambiguation sequence that forms perceptual memory of multistable displays via reverse correlation method: Bias onset perception but gently

When multistable displays are presented intermittently with long blank intervals, their onset perception is determined by perceptual memory of multistable displays. We investigated when and how it is formed using a reverse correlation method and bistable kinetic depth effect displays. Each experimental block consisted of interleaved fully ambiguous probe and exogenously disambiguated prime displays. The purpose of the former was to "read out" the perceptual memory, whereas the latter contained purely random disambiguation sequences that were presented at the beginning of the prime display, throughout the entire presentation, or at the beginning and the end of the presentation. For each experiment and condition, we selected a subset of trials with disambiguation sequences that led to a change in perception of either the prime itself (sequences that modified perception) or the following fully ambiguous probe (sequences that modified perceptual memory). We estimated average disambiguation sequences for each participant using additive linear models. We found that an optimal sequence started at the onset with a moderate disambiguation against the previously dominant state (dominant perception for the previous probe) that gradually reduced until the display is fully ambiguous. We also show that the same sequence leads to an altered perception of the prime, indicating that perception and perceptual memory form at the same time. We suggest that perceptual memory is a consequence of an earlier evidence accumulation process and is informative about how the visual system treated ambiguity in the past rather than how it anticipates an uncertain future.

Binocular rivalry reveals an out-of-equilibrium neural dynamics suited for decision-making

In ambiguous or conflicting sensory situations, perception is often ‘multistable’ in that it perpetually changes at irregular intervals, shifting abruptly between distinct alternatives. The interval statistics of these alternations exhibits quasi-universal characteristics, suggesting a general mechanism. Using binocular rivalry, we show that many aspects of this perceptual dynamics are reproduced by a hierarchical model operating out of equilibrium. The constitutive elements of this model idealize the metastability of cortical networks. Independent elements accumulate visual evidence at one level, while groups of coupled elements compete for dominance at another level. As soon as one group dominates perception, feedback inhibition suppresses supporting evidence. Previously unreported features in the serial dependencies of perceptual alternations compellingly corroborate this mechanism. Moreover, the proposed out-of-equilibrium dynamics satisfies normative constraints of continuous decision-making. Thus, multistable perception may reflect decision-making in a volatile world: integrating evidence over space and time, choosing categorically between hypotheses, while concurrently evaluating alternatives.

What happens in the brain of meditators when perception changes but not the stimulus?

During the observation of an ambiguous figure our perception alternates between mutually exclusive interpretations, although the stimulus itself remains unchanged. The rate of these endogenous reversals has been discussed as reflecting basic aspects of endogenous brain dynamics. Recent evidence indicates that extensive meditation practice evokes long-term functional and anatomic changes in the brain, also affecting the endogenous brain dynamics. As one of several consequences the rate of perceptual reversals during ambiguous figure perception decreases. In the present study we compared EEG-correlates of endogenous reversals of ambiguous figures between meditators and non-meditating controls in order to better understand timing and brain locations of this altered endogenous brain dynamics. A well-established EEG paradigm was used to measure the neural processes underlying endogenous perceptual reversals of ambiguous figures with high temporal precision. We compared reversal-related ERPs between experienced meditators and non-meditating controls. For both groups we found highly similar chains of reversal-related ERPs, starting early in visual areas, therewith replicating previous findings from the literature. Meditators, however, showed an additional frontal ERP signature already 160 ms after stimulus onset (Frontal Negativity). We interpret the additional, meditation-specific ERP results as evidence that extensive meditation practice provides control of frontal brain areas over early sensory processing steps. This may allow meditators to overcome phylogenetically evolved perceptual and attentional processing automatisms.

Positive and negative hysteresis effects for the perception of geometric and emotional ambiguities

AIM

The present study utilizes perceptual hysteresis effects to compare the ambiguity of Mona Lisa's emotional face expression (high-level ambiguity) and of geometric cube stimuli (low-level ambiguity).

METHODS

In two experiments we presented series of nine Mona Lisa variants and nine cube variants. Stimulus ambiguity was manipulated by changing Mona Lisa's mouth curvature (Exp. 1) and the cubes' back-layer luminance (Exp. 2). Each experiment consisted of three conditions, two with opposite stimulus presentation sequences with increasing and decreasing degrees of ambiguity, respectively, and a third condition with a random presentation sequence. Participants indicated happy or sad face percepts (Exp. 1) and alternative 3D cube percepts (Exp. 2) by key presses. We studied the influences of a priori perceptual biases (long-term memory) and presentation order (short-term memory) on perception.

RESULTS

Perception followed sigmoidal functions of the stimulus ambiguity morphing parameters. The morphing parameter for the functions' inflection points depended strongly on stimulus presentation order with similar effect sizes but different signs for the two stimulus types (positive hysteresis / priming for the cubes; negative hysteresis / adaptation for Mona Lisa). In the random conditions, the inflection points were located in the middle between those from the two directional conditions for the Mona Lisa stimuli. For the cube stimuli, they were superimposed on one sigmoidal function for the ordered condition.

DISCUSSION

The hysteresis effects reflect the influence of short-term memory during the perceptual disambiguation of ambiguous sensory information. The effects for the two stimulus types are of similar size, explaining up to 34% of the perceptual variance introduced by the paradigm. We explain the qualitative difference between positive and negative hysteresis with adaptation for Mona Lisa and with priming for the cubes. In addition, the hysteresis paradigm allows a quantitative determination of the impact of adaptation and priming during the resolution of perceptual ambiguities. The asymmetric shifts of inflection points in the case of the cube stimuli is likely due to an a priori perceptual bias, reflecting an influence of long-term memory. Whether corresponding influences also exist for the Mona Lisa variants is so far unclear.

Out of sight, out of mind: Occlusion and eye closure destabilize moving bistable structure-from-motion displays

Our brain constantly tries to anticipate the future by using a variety of memory mechanisms. Interestingly, studies using the intermittent presentation of multistable displays have shown little perceptual persistence for interruptions longer than a few hundred milliseconds. Here we examined whether we can facilitate the perceptual stability of bistable displays following a period of invisibility by employing a physically plausible and ecologically valid occlusion event sequence, as opposed to the typical intermittent presentation, with sudden onsets and offsets. To this end, we presented a bistable rotating structure-from-motion display that was moving along a linear horizontal trajectory on the screen and either was temporarily occluded by another object (a cardboard strip in Exp. 1, a computer-generated image in Exp. 2) or became invisible due to eye closure (Exp. 3). We report that a bistable rotation direction reliably persisted following occlusion or interruption only (1) if the pre- and postinterruption locations overlapped spatially (an occluder with apertures in Exp. 2 or brief, spontaneous blinks in Exp. 3) or (2) if an object's size allowed for the efficient grouping of dots on both sides of the occluding object (large objects in Exp. 1). In contrast, we observed no persistence whenever the pre- and postinterruption locations were nonoverlapping (large solid occluding objects in Exps. 1 and 2 and long, prompted blinks in Exp. 3). We report that the bistable rotation direction of a moving object persisted only for spatially overlapping neural representations, and that persistence was not facilitated by a physically plausible and ecologically valid occlusion event.

A hierarchical stochastic model for bistable perception

Viewing of ambiguous stimuli can lead to bistable perception alternating between the possible percepts. During continuous presentation of ambiguous stimuli, percept changes occur as single events, whereas during intermittent presentation of ambiguous stimuli, percept changes occur at more or less regular intervals either as single events or bursts. Response patterns can be highly variable and have been reported to show systematic differences between patients with schizophrenia and healthy controls. Existing models of bistable perception often use detailed assumptions and large parameter sets which make parameter estimation challenging. Here we propose a parsimonious stochastic model that provides a link between empirical data analysis of the observed response patterns and detailed models of underlying neuronal processes. Firstly, we use a Hidden Markov Model (HMM) for the times between percept changes, which assumes one single state in continuous presentation and a stable and an unstable state in intermittent presentation. The HMM captures the observed differences between patients with schizophrenia and healthy controls, but remains descriptive. Therefore, we secondly propose a hierarchical Brownian model (HBM), which produces similar response patterns but also provides a relation to potential underlying mechanisms. The main idea is that neuronal activity is described as an activity difference between two competing neuronal populations reflected in Brownian motions with drift. This differential activity generates switching between the two conflicting percepts and between stable and unstable states with similar mechanisms on different neuronal levels. With only a small number of parameters, the HBM can be fitted closely to a high variety of response patterns and captures group differences between healthy controls and patients with schizophrenia. At the same time, it provides a link to mechanistic models of bistable perception, linking the group differences to potential underlying mechanisms.

Patients suffering from schizophrenia show specific cognitive deficits. One way to study these cognitive phenomena works with the presentation of ambiguous stimuli. During viewing of an ambiguous stimulus, perception alters spontaneously between different percepts. Percept changes occur as single events during continuous presentation, whereas during intermittent presentation, percept changes occur at regular intervals either as single events or bursts. Here we investigate perceptual responses to continuous and intermittent stimulation in healthy control subjects and patients with schizophrenia. Interestingly, the response patterns can be highly variable but show systematic group differences. We propose a model that connects these perceptual responses to underlying neuronal processes. The model mainly describes the activity difference between competing neuronal populations on different neuronal levels. In a hierarchical manner, the differential activity generates switching between the conflicting percepts as well as between states of higher and lower perceptual stability. By fitting the model directly to empirical responses, a high variety of patterns can be reproduced, and group differences between healthy controls and patients with schizophrenia can be captured. This helps to link the observed group differences to potential neuronal mechanisms, suggesting that patients with schizophrenia tend to spend more time in neuronal states of lower perceptual stability.

Perception and the strongest sensory memory trace of multi-stable displays both form shortly after the stimulus onset

We investigated the relation between perception and sensory memory of multi-stable structure-from-motion displays. The latter is an implicit visual memory that reflects a recent history of perceptual dominance and influences only the initial perception of multi-stable displays. First, we established the earliest time point when the direction of an illusory rotation can be reversed after the display onset (29-114 ms). Because our display manipulation did not bias perception towards a specific direction of illusory rotation but only signaled the change in motion, this means that the perceptual dominance was established no later than 29-114 ms after the stimulus onset. Second, we used orientation-selectivity of sensory memory to establish which display orientation produced the strongest memory trace and when this orientation was presented during the preceding prime interval (80-140 ms). Surprisingly, both estimates point towards the time interval immediately after the display onset, indicating that both perception and sensory memory form at approximately the same time. This suggests a tighter integration between perception and sensory memory than previously thought, warrants a reconsideration of its role in visual perception, and indicates that sensory memory could be a unique behavioral correlate of the earlier perceptual inference that can be studied post hoc.

Transformation priming helps to disambiguate sudden changes of sensory inputs

Retinal input is riddled with abrupt transients due to self-motion, changes in illumination, object-motion, etc. Our visual system must correctly interpret each of these changes to keep visual perception consistent and sensitive. This poses an enormous challenge, as many transients are highly ambiguous in that they are consistent with many alternative physical transformations. Here we investigated inter-trial effects in three situations with sudden and ambiguous transients, each presenting two alternative appearances (rotation-reversing structure-from-motion, polarity-reversing shape-from-shading, and streaming-bouncing object collisions). In every situation, we observed priming of transformations as the outcome perceived in earlier trials tended to repeat in subsequent trials and this repetition was contingent on perceptual experience. The observed priming was specific to transformations and did not originate in priming of perceptual states preceding a transient. Moreover, transformation priming was independent of attention and specific to low level stimulus attributes. In summary, we show how "transformation priors" and experience-driven updating of such priors helps to disambiguate sudden changes of sensory inputs. We discuss how dynamic transformation priors can be instantiated as "transition energies" in an "energy landscape" model of the visual perception.

EEG correlates of perceptual reversals in Boring's ambiguous old/young woman stimulus

Ambiguous figures attract observers because perception alternates between different interpretations while the sensory information stays unchanged. Understanding the underlying processes is difficult because the precise time instant of this endogenous reversal event needs to be known but is difficult to measure. Presenting ambiguous figures discontinuously and using stimulus onset as estimation of the reversal event increased temporal resolution and provided a series of well-confirmed EEG signatures. In the current EEG study we used this 'onset paradigm' for the first time with Boring's old/young woman stimulus. We found an early occipital event-related potential (ERP) correlate of reversals between the perception of the old woman and the perception of the young woman that fits well with previous ERP findings. This component was not followed by the often-reported occipito-parietal Reversal Negativity at 260 ms, but instead by an occipito-temporal N170, that is typically reported with face stimuli. We interpret our results as follows: ambiguity conflicts take place during processing of stimulus elements in early visual areas roughly 130 ms after stimulus onset. The disambiguation of these elements and their assembly to object 'gestalts' result from an interplay between early visual and object-specific brain areas in a temporal window between 130 and 260 ms after stimulus onset. In the particular case of Boring's old/young woman the processes of element disambiguation and gestalt construction are already finished at 170 ms and, thus, 90 ms earlier than in the case of ambiguous geometric figures (eg Necker cube or Schroeder staircase) or of binocular rivalrous gratings.

Temporal dynamics of different cases of bi-stable figure-ground perception

Segmentation of a visual scene in "figure" and "ground" is essential for perception of the three-dimensional layout of a scene. In cases of bi-stable perception, two distinct figure-ground interpretations alternate over time. We were interested in the temporal dynamics of these alternations, in particular when the same image is presented repeatedly, with short blank periods in-between. Surprisingly, we found that the intermittent presentation of Rubin's classical "face-or-vase" figure, which is frequently taken as a standard case of bi-stable figure-ground perception, often evoked perceptual switches during the short presentations and stabilization was not prominent. Interestingly, bi-stable perception of Kanizsa's anomalous transparency figure did strongly stabilize across blanks. We also found stabilization for the Necker cube, which we used for comparison. The degree of stabilization (and the lack of it) varied across stimuli and across individuals. Our results indicate, against common expectation, that the stabilization phenomenon cannot be generally evoked by intermittent presentation. We argue that top-down feedback factors such as familiarity, semantics, expectation, and perceptual bias contribute to the complex processes underlying the temporal dynamics of bi-stable figure-ground perception.

What you see depends on what you saw, and what else you saw: the interactions between motion priming and object priming

Both visual object priming and motion priming have been reported independently, but the interactions between the two are still largely unexplored. Here we investigated this question using a novel type of SFM stimuli, 3-D helixes, and found that the motion direction perception of an ambiguous helix can be biased by the motion direction of a preceding SFM stimulus - a classic motion priming effect. However, the effectiveness of motion priming depends on object priming: a neutral object priming produced a weak motion priming, a congruent object priming led to a strong motion priming, and critically, an incongruent object priming abolished and overpowered the motion priming. In contrast, object priming alone (in the absence of motion overlap) had little effects biasing motion perception. Taken together, these results suggest that there exists an integrated neural representation of motion and structure of 3-D SFM stimuli, and motion priming of 3-D SFM stimuli might happen at an intermediate stage between MT/V5 (which is not shape selective) and LO (lateral occipital, which is not motion selective). This novel type of stimuli, 3-D helixes, along with the prime-target paradigm, thus might offer a unique tool to examine neural bases underlying the perception of 3-D SFM stimuli and perceptual priming.

Perceptual adaptation to structure-from-motion depends on the size of adaptor and probe objects, but not on the similarity of their shapes

Perceptual adaptation destabilizes the phenomenal appearance of multistable visual displays. Prolonged dominance of a perceptual state fatigues the associated neural population, lowering the likelihood of renewed perception of the same appearance (Nawrot & Blake in Perception & Psychophysics, 49, 230-44, 1991). Here, we used a selective adaptation paradigm to investigate perceptual adaptation for the illusory rotation of ambiguous structure-from-motion (SFM) displays. Specifically, we generated SFM objects with different three-dimensional shapes and presented them in random order, separating successive objects by brief blank periods, which included a mask. To assess the specificity of perceptual adaptation to the shape of SFM objects, we established the probability that a perceived direction of rotation persisted between successive objects of similar or dissimilar shape. We found that the strength of negative aftereffects depended on the volume, but not the shape, of adaptor and probe objects. More voluminous objects were both more effective as adaptor objects and more sensitive as probe objects. Surprisingly, we found these volume effects to be completely independent, since any relationship between two shapes (such as overlap between volumes, similarity of shape, or similarity of velocity profiles) failed to modulate the negative aftereffect. This pattern of results was the opposite of that observed for sensory memory of SFM objects, which reflects similarity between objects, but not volume of individual objects (Pastukhov et al. in Attention, Perception & Psychophysics, 75, 1215-1229, 2013). The disparate specificities of perceptual adaptation and sensory memory for identical SFM objects suggest that the two aftereffects engage distinct neural representations, consistent with recent brain imaging results (Schwiedrzik et al. in Cerebral Cortex, 2012).

Sensory memory of structure-from-motion is shape-specific

Perceptual priming can stabilize the phenomenal appearance of multistable visual displays (Leopold, Wilke, Maier, & Logothetis, Nature Neuroscience, 5, 605-609, 2002). Prior exposure to such displays induces a sensory memory of their appearance, which persists over long intervals and intervening stimulation, and which facilitates renewed perception of the same appearance. Here, we investigated perceptual priming for the apparent rotation in depth of ambiguous structure-from-motion (SFM) displays. Specifically, we generated SFM objects with different three-dimensional shapes and presented them in random order and with intervening blank periods. To assess perceptual priming, we established the probability that a perceived direction of rotation would persist between successive objects. In general, persistence was greatest between identical objects, intermediate between similar objects, and negligible between dissimilar objects. These results demonstrate unequivocally that sensory memory for apparent rotation is specific to three-dimensional shape, contrary to previous reports (e.g., Maier, Wilke, Logothetis, & Leopold, Current Biology, 13, 1076-1085, 2003). Because persistence did not depend on presentation order for any pair of objects, it provides a commutative measure for the similarity of object shapes. However, it is not clear exactly which features or aspects of object shape determine similarity. At least, we did not find simple, low-level features (such as volume overlap, heterogeneity, or rotational symmetry) that could have accounted for all observations. Accordingly, it seems that sensory memory of SFM (which underlies priming of ambiguous rotation) engages higher-level representations of object surface and shape.

Sensory memory of illusory depth in structure-from-motion

When multistable displays (stimuli consistent with two or more equally plausible perceptual interpretations) are presented intermittently, their perceptions are stabilized by sensory memory. Independent memory traces are generated not only for different types of multistable displays (Maier, Wilke, Logothetis, & Leopold, Current Biology 13:1076-1085, 2003), but also for different ambiguous features of binocular rivalry (Pearson & Clifford, Journal of Vision 4:196-202, 2004). In the present study, we examined whether a similar independence of sensory memories is observed in structure-from-motion (SFM), a multistable display with two ambiguous properties. In SFM, a 2-D planar motion creates a vivid impression of a rotating 3-D volume. Both the illusory rotation and illusory depth (i.e., how close parts of an object appear to the observer) of an SFM object are ambiguous. We dissociated the sensory memories of these two ambiguous properties by using an intermittent presentation in combination with a forced-ambiguous-switch paradigm (Pastukhov, Vonau, & Braun, PLoS ONE 7:e37734, 2012). We demonstrated that the illusory depth of SFM generates a sensory memory trace that is independent from that of illusory rotation. Despite this independence, the specificities levels of the sensory memories were identical for illusory depth and illusory rotation. The history effect was weakened by a change in the volumetric property of a shape (whether it was a hollow band or a filled drum volume), but not by changes in color or size. We discuss how these new results constrain models of sensory memory and SFM processing.

Event-related potential responses to perceptual reversals are modulated by working memory load

While viewing ambiguous figures, such as the Necker cube, the available perceptual interpretations alternate with one another. The role of higher level mechanisms in such reversals remains unclear. We tested whether perceptual reversals of discontinuously presented Necker cube pairs depend on working memory resources by manipulating cognitive load while recording event-related potentials (ERPs). The ERPs showed early enhancements of negativity, which were obtained in response to the first cube approximately 500 ms before perceived reversals. We found that working memory load influenced reversal-related brain responses in response to the second cube over occipital areas at the 150-300 ms post-stimulus and over central areas at P3 time window (300-500 ms), suggesting that it modulates intermediate visual processes. Interestingly, reversal rates remained unchanged by the working memory load. We propose that perceptual reversals in discontinuous presentation of ambiguous stimuli are governed by an early (well preceding pending reversals) mechanism, while the effects of load on the reversal related ERPs may reflect general top-down influences on visual processing, possibly mediated by the prefrontal cortex.