Learning What to See in a Changing World

Effect of motion state variability on error-related potentials during continuous feedback paradigms and their consequences for classification

BACKGROUND

An erroneous motion would elicit the error-related potential (ErrP) when humans monitor the behavior of the external devices. This EEG modality has been largely applied to brain-computer interface in an active or passive manner with discrete visual feedback. However, the effect of variable motion state on ErrP morphology and classification performance raises concerns when the interaction is conducted with continuous visual feedback.

NEW METHOD

In the present study, we designed a cursor control experiment. Participants monitored a continuously moving cursor to reach the target on one side of the screen. Motion state varied multiple times with two factors: (1) motion direction and (2) motion speed. The effects of these two factors on the morphological characteristics and classification performance of ErrP were analyzed. Furthermore, an offline simulation was performed to evaluate the effectiveness of the proposed extended ErrP-decoder in resolving the interference by motion direction changes.

RESULTS

The statistical analyses revealed that motion direction and motion speed significantly influenced the amplitude of feedback-ERN and frontal-Pe components, while only motion direction significantly affected the classification performance.

COMPARISON WITH EXISTING METHODS

Significant deviation was found in ErrP detection utilizing classical correct-versus-erroneous event training. However, this bias can be alleviated by 16% by the extended ErrP-decoder.

CONCLUSION

The morphology and classification performance of ErrP signal can be affected by motion state variability during continuous feedback paradigms. The results enhance the comprehension of ErrP morphological components and shed light on the detection of BCI's error behavior in practical continuous control.

A novel task and methods to evaluate inter-individual variation in audio-visual associative learning

Learning audio-visual associations is foundational to a number of real-world skills, such as reading acquisition or social communication. Characterizing individual differences in such learning has therefore been of interest to researchers in the field. Here, we present a novel audio-visual associative learning task designed to efficiently capture inter-individual differences in learning, with the added feature of using non-linguistic stimuli, so as to unconfound language and reading proficiency of the learner from their more domain-general learning capability. By fitting trial-by-trial performance in our novel learning task using simple-to-use statistical tools, we demonstrate the expected inter-individual variability in learning rate as well as high precision in its estimation. We further demonstrate that such measured learning rate is linked to working memory performance in Italian-speaking (N = 58) and French-speaking (N = 51) adults. Finally, we investigate the extent to which learning rate in our task, which measures cross-modal audio-visual associations while mitigating familiarity confounds, predicts reading ability across participants with different linguistic backgrounds. The present work thus introduces a novel non-linguistic audio-visual associative learning task that can be used across languages. In doing so, it brings a new tool to researchers in the various domains that rely on multi-sensory integration from reading to social cognition or socio-emotional learning.

Priming and reversals of the perceived ambiguous orientation of a structure-from-motion shape and relation to personality traits

We demonstrate contributions of top-down and bottom-up influences in perception as explored by priming and counts of perceived reversals and mixed percepts, as probed by an ambiguously slanted structure-from-motion (SFM) test-cylinder. We included three different disambiguated primes: a SFM cylinder, a still image of a cylinder, and an imagined cylinder. In Experiment 1 where the prime and test sequentially occupied the same location, we also administered questionnaires with the Big-5 trait openness and vividness of visual imagery to probe possible relations to top-down driven priming. Since influences of gaze or position in the prime conditions in Experiment 1 could not be ruled out completely, Experiment 2 was conducted where the test cylinder appeared at a randomly chosen position after the prime. In Experiment 2 we also measured the number of perceptual reversals and mixed percepts during prolonged viewing of our ambiguous SFM-cylinder, and administered questionnaires to measure all Big-5 traits, autism, spatial and object imagery, and rational or experiential cognitive styles, associated with bottom-up and top-down processes. The results revealed contributions of position-invariant and cue-invariant priming. In addition, residual contributions of low-level priming was found when the prime and test were both defined by SFM, and were presented at the same location, and the correlation between the SFM priming and the other two priming conditions were weaker than between the pictorial and imagery priming. As previously found with ambiguous binocular rivalry stimuli, we found positive correlations between mixed percepts and the Big-5 dimension openness to experience, and between reversals, mixed percepts and neuroticism. Surprisingly, no correlations between the scores from the vividness of imagery questionnaires and influence from any of the primes were obtained. An intriguing finding was the significant differences between the positive correlation from the experiential cognitive style scores, and the negative correlation between rational style and the cue invariant priming. Among other results, negative correlations between agreeableness and all priming conditions were obtained. These results not only support the notion of multiple processes involved in the perception of ambiguous SFM, but also link these processes in perception to specific personality traits.

Psychotic Experiences in Schizophrenia and Sensitivity to Sensory Evidence

Perceptual inference depends on an optimal integration of current sensory evidence with prior beliefs about the environment. Alterations of this process have been related to the emergence of positive symptoms in schizophrenia. However, it has remained unclear whether delusions and hallucinations arise from an increased or decreased weighting of prior beliefs relative to sensory evidence. To investigate the relation of this prior-to-likelihood ratio to positive symptoms in schizophrenia, we devised a novel experimental paradigm which gradually manipulates perceptually ambiguous visual stimuli by disambiguating stimulus information. As a proxy for likelihood precision, we assessed the sensitivity of individual participants to sensory evidence. As a surrogate for the precision of prior beliefs in perceptual stability, we measured phase duration in ambiguity. Relative to healthy controls, patients with schizophrenia showed a stronger increment in congruent perceptual states for increasing levels of disambiguating stimulus evidence. Sensitivity to sensory evidence correlated positively with the individual patients' severity of perceptual anomalies and hallucinations. Moreover, the severity of such experiences correlated negatively with phase duration. Our results indicate that perceptual anomalies and hallucinations are associated with a shift of perceptual inference toward sensory evidence and away from prior beliefs. This reduced prior-to-likelihood ratio in sensory processing may contribute to the phenomenon of aberrant salience, which has been suggested to give rise to the false inferences underlying psychotic experiences.

Sustained effects of corrupted feedback on perceptual inference

Feedback is central to most forms of learning, and its reliability is therefore critical. Here, we investigated the effects of corrupted, and hence unreliable, feedback on perceptual inference. Within the framework of Bayesian inference, we hypothesised that corrupting feedback in a demanding perceptual task would compromise sensory information processing and bias inference towards prior information if available. These hypotheses were examined by a simulation and in two behavioural experiments with visual detection (experiment 1) and discrimination (experiment 2) tasks. Both experiments consisted of two sessions comprising intervention runs with either corrupted or uncorrupted (correct) feedback, and pre- and post-intervention tests to assess the effects of feedback. In the tests alone, additional prior beliefs were induced through predictive auditory cues to assess sustained effects of feedback on the balance between sensory evidence and prior beliefs. Both experiments and the simulation showed the hypothesised decrease in performance and increased reliance on prior beliefs after corrupted but not uncorrupted feedback. Exploratory analyses indicated reduced confidence regarding perceptual decisions during delivery of corrupted feedback. Our results suggest that corrupted feedback on perceptual decisions leads to sustained changes in perceptual inference, characterised by a shift from sensory likelihood to prior beliefs when those are accessible.

Does task relevance shape the 'shift to global' in ambiguous motion perception?

Perception can differ even when the stimulus information is the same. Previous studies have demonstrated the importance of experience and relevance on visual perception. We examined the influence of perceptual relevance in an auxiliary task on subsequent perception of an ambiguous stimulus. Observers were presented with an ambiguous motion stimulus that could either be perceived as rotating dot pairs ("local") or pulsating geometrical figures ("global"). Prolonged perception of this stimulus is characterized by a "shift to global", but it remained unclear whether this process is due to relevance of the global percept or mere exposure to the stimulus. During a relevance learning phase over 5 successive days, participants were divided into conditions determining the relevant percept in an auxiliary task: local, global, or none (active exposure). In a pre- and posttest, individual points of subjective equality between local and global percepts were measured. Results indicate that there is indeed a shift to global. Interestingly, auxiliary task relevance does not seem to modify this process.

Delusion Proneness is Linked to a Reduced Usage of Prior Beliefs in Perceptual Decisions

Predictive coding theories state an aberrant weighting of prior beliefs and present sensory information as a core computational pathology in psychosis. Specifically, it has been proposed that the influence of prior beliefs which attenuate improbable sensory information is weakened, resulting in an overweighing of this potentially misleading information. However, it is currently unclear whether this alteration is specific to perceptual processes or whether it represents a more pervasive deficit that extends to cognitive processes. Here, we carried out 2 behavioral experiments that probed the usage of priors during perceptual and cognitive processes, respectively, in 123 healthy individuals with varying degrees of delusion proneness. In an audio-visual perceptual discrimination task, participants had to judge the global motion direction of random dot kinematograms. Prior beliefs were induced by auditory cues that probabilistically predicted the global motion direction of the dot kinematograms, allowing us to measure the impact of prior beliefs on perceptual decision making. A control experiment paralleled the design of the perceptual decision making task in the domain of cognitive decision making. By fitting the participants' responses with a probabilistic decision model, we quantified the impact of prior beliefs on participants' decisions in both tasks. With growing delusion proneness, we found a decreased impact of prior beliefs on perceptual but not on cognitive decision making. Our results show that delusion proneness is linked to a specifically reduced usage of prior beliefs in perceptual decisions, thereby empirically substantiating predictive coding theories of psychosis.

The Neural Correlates of Hierarchical Predictions for Perceptual Decisions

Sensory information is inherently noisy, sparse, and ambiguous. In contrast, visual experience is usually clear, detailed, and stable. Bayesian theories of perception resolve this discrepancy by assuming that prior knowledge about the causes underlying sensory stimulation actively shapes perceptual decisions. The CNS is believed to entertain a generative model aligned to dynamic changes in the hierarchical states of our volatile sensory environment. Here, we used model-based fMRI to study the neural correlates of the dynamic updating of hierarchically structured predictions in male and female human observers. We devised a crossmodal associative learning task with covertly interspersed ambiguous trials in which participants engaged in hierarchical learning based on changing contingencies between auditory cues and visual targets. By inverting a Bayesian model of perceptual inference, we estimated individual hierarchical predictions, which significantly biased perceptual decisions under ambiguity. Although "high-level" predictions about the cue-target contingency correlated with activity in supramodal regions such as orbitofrontal cortex and hippocampus, dynamic "low-level" predictions about the conditional target probabilities were associated with activity in retinotopic visual cortex. Our results suggest that our CNS updates distinct representations of hierarchical predictions that continuously affect perceptual decisions in a dynamically changing environment.SIGNIFICANCE STATEMENT Bayesian theories posit that our brain entertains a generative model to provide hierarchical predictions regarding the causes of sensory information. Here, we use behavioral modeling and fMRI to study the neural underpinnings of such hierarchical predictions. We show that "high-level" predictions about the strength of dynamic cue-target contingencies during crossmodal associative learning correlate with activity in orbitofrontal cortex and the hippocampus, whereas "low-level" conditional target probabilities were reflected in retinotopic visual cortex. Our findings empirically corroborate theorizations on the role of hierarchical predictions in visual perception and contribute substantially to a longstanding debate on the link between sensory predictions and orbitofrontal or hippocampal activity. Our work fundamentally advances the mechanistic understanding of perceptual inference in the human brain.

The influence of action-effect anticipation on bistable perception: differences between onset rivalry and ambiguous motion

Perception is strongly shaped by the actions we perform. According to the theory of event coding, and forward models of motor control, goal-directed action preparation activates representations of desired effects. These expectations about the precise stimulus identity of one's action-outcomes (i.e. identity predictions) are thought to selectively influence perceptual processing of action-contingent effects. However, the existing evidence for such identity-prediction effects is scarce and mixed. Here, we developed a new paradigm to capture such effects and examined whether action-outcome predictions can bias the perception of binocular onset rivalry (Experiments 1a and 1b) and bistable motion (Experiment 2). Participants performed learning tasks in which they were exposed to action-outcome associations. On test trials, actions were followed by bistable stimuli that could be perceived as being either congruent or incongruent with the aforementioned associations (i.e. rivalrous oriented gratings in Experiments 1a and 1b and spheres with ambiguous rotation directions in Experiment 2). Across three experiments, we show that, whilst exposure to action-effect associations can bias the apparent motion direction of ambiguous spheres, it fails to influence perceptual selection of grating orientations in binocular onset rivalry. This pattern of results extends previous work on ambiguous motion by demonstrating that action-induced modulations do not generalize to all types of bistable percepts.

Combined fMRI- and eye movement-based decoding of bistable plaid motion perception

The phenomenon of bistable perception, in which perception alternates spontaneously despite constant sensory stimulation, has been particularly useful in probing the neural bases of conscious perception. The study of such bistability requires access to the observer's perceptual dynamics, which is usually achieved via active report. This report, however, constitutes a confounding factor in the study of conscious perception and can also be biased in the context of certain experimental manipulations. One approach to circumvent these problems is to track perceptual alternations using signals from the eyes or the brain instead of observers' reports. Here we aimed to optimize such decoding of perceptual alternations by combining eye and brain signals. Eye-tracking and functional magnetic resonance imaging (fMRI) was performed in twenty participants while they viewed a bistable visual plaid motion stimulus and reported perceptual alternations. Multivoxel pattern analysis (MVPA) for fMRI was combined with eye-tracking in a Support vector machine to decode participants' perceptual time courses from fMRI and eye-movement signals. While both measures individually already yielded high decoding accuracies (on average 86% and 88% correct, respectively) classification based on the two measures together further improved the accuracy (91% correct). These findings show that leveraging on both fMRI and eye movement data may pave the way for optimized no-report paradigms through improved decodability of bistable motion perception and hence for a better understanding of the neural correlates of consciousness.

fMRI-based decoding of reward effects in binocular rivalry

Binocular rivalry is a phenomenon where the simultaneous presentation of two different stimuli to the two eyes leads to alternating perception of the two stimuli. The temporary dominance of one stimulus over the other is influenced by several factors. Here, we studied the influence of reward on binocular rivalry dynamics and its neural representation in visual cortex. Orthogonal rotating grating stimuli were shown continuously, while monetary reward was given during the conscious perception of one stimulus but not the other. Periods of perceptual dominance were assessed both through participants’ subjective report and objectively using functional magnetic resonance imaging and multi-voxel pattern analysis. Results did not confirm previous evidence for an effect of reward on perceptual dominance durations. Exploratory post-hoc analyses indicated that knowledge regarding both the reward contingency and the subjective nature of perceptual alternations may have interfered with potential reward effects on perceptual phase durations, suggesting a moderating role of meta-cognitive awareness in reward-based perceptual inference. Future studies of top-down influences on bistable perception should carefully consider the methodological challenges related to meta-cognitive awareness.

Thought Insertion as a Self-Disturbance: An Integration of Predictive Coding and Phenomenological Approaches

Current theories in the framework of hierarchical predictive coding propose that positive symptoms of schizophrenia, such as delusions and hallucinations, arise from an alteration in Bayesian inference, the term inference referring to a process by which learned predictions are used to infer probable causes of sensory data. However, for one particularly striking and frequent symptom of schizophrenia, thought insertion, no plausible account has been proposed in terms of the predictive-coding framework. Here we propose that thought insertion is due to an altered experience of thoughts as coming from “nowhere”, as is already indicated by the early 20th century phenomenological accounts by the early Heidelberg School of psychiatry. These accounts identified thought insertion as one of the self-disturbances (from German: “Ichstörungen”) of schizophrenia and used mescaline as a model-psychosis in healthy individuals to explore the possible mechanisms. The early Heidelberg School (Gruhle, Mayer-Gross, Beringer) first named and defined the self-disturbances, and proposed that thought insertion involves a disruption of the inner connectedness of thoughts and experiences, and a “becoming sensory” of those thoughts experienced as inserted. This account offers a novel way to integrate the phenomenology of thought insertion with the predictive coding framework. We argue that the altered experience of thoughts may be caused by a reduced precision of context-dependent predictions, relative to sensory precision. According to the principles of Bayesian inference, this reduced precision leads to increased prediction-error signals evoked by the neural activity that encodes thoughts. Thus, in analogy with the prediction-error related aberrant salience of external events that has been proposed previously, “internal” events such as thoughts (including volitions, emotions and memories) can also be associated with increased prediction-error signaling and are thus imbued with aberrant salience. We suggest that the individual’s attempt to explain the aberrant salience of thoughts results in their interpretation as being inserted by an alien agent, similarly to the emergence of delusions in response to the aberrant salience of sensory stimuli.