Dissecting (un)crowding

In crowding, perception of a target deteriorates in the presence of nearby flankers. Surprisingly, perception can be rescued from crowding if additional flankers are added (uncrowding). Uncrowding is a major challenge for all classic models of crowding and vision in general, because the global configuration of the entire stimulus is crucial. However, it is unclear which characteristics of the configuration impact (un)crowding. Here, we systematically dissected flanker configurations and showed that (un)crowding cannot be easily explained by the effects of the sub-parts or low-level features of the stimulus configuration. Our modeling results suggest that (un)crowding requires global processing. These results are well in line with previous studies showing the importance of global aspects in crowding.

A comparative biology approach to DNN modeling of vision: A focus on differences, not similarities

Deep neural networks (DNNs) have revolutionized computer science and are now widely used for neuroscientific research. A hot debate has ensued about the usefulness of DNNs as neuroscientific models of the human visual system; the debate centers on to what extent certain shortcomings of DNNs are real failures and to what extent they are redeemable. Here, we argue that the main problem is that we often do not understand which human functions need to be modeled and, thus, what counts as a falsification. Hence, not only is there a problem on the DNN side, but there is also one on the brain side (i.e., with the explanandum-the thing to be explained). For example, should DNNs reproduce illusions? We posit that we can make better use of DNNs by adopting an approach of comparative biology by focusing on the differences, rather than the similarities, between DNNs and humans to improve our understanding of visual information processing in general.

Adaptive mechanisms of visual motion discrimination, integration, and segregation

Under ecological conditions, the luminance impinging on the retina varies within a dynamic range of 220 dB. Stimulus contrast can also vary drastically within a scene and eye movements leave little time for sampling luminance. Given these fundamental problems, the human brain allocates a significant amount of resources and deploys both structural and functional solutions that work in tandem to compress this range. Here we propose a new dynamic neural model built upon well-established canonical neural mechanisms. The model consists of two feed-forward stages. The first stage encodes the stimulus spatially and normalizes its activity by extracting contrast and discounting the background luminance. These normalized activities allow a second stage to implement a contrast-dependent spatial-integration strategy. We show how the properties of this model can account for adaptive properties of motion discrimination, integration, and segregation.

How do visual skills relate to action video game performance?

It has been claimed that video gamers possess increased perceptual and cognitive skills compared to non-video gamers. Here, we examined to which extent gaming performance in CS:GO (Counter-Strike: Global Offensive) correlates with visual performance. We tested 94 players ranging from beginners to experts with a battery of visual paradigms, such as visual acuity and contrast detection. In addition, we assessed performance in specific gaming skills, such as shooting and tracking, and administered personality traits. All measures together explained about 70% of the variance of the players’ rank. In particular, regression models showed that a few visual abilities, such as visual acuity in the periphery and the susceptibility to the Honeycomb illusion, were strongly associated with the players’ rank. Although the causality of the effect remains unknown, our results show that high-rank players perform better in certain visual skills compared to low-rank players.

Shrinking Bouma's window: How to model crowding in dense displays

In crowding, perception of a target deteriorates in the presence of nearby flankers. Traditionally, it is thought that visual crowding obeys Bouma's law, i.e., all elements within a certain distance interfere with the target, and that adding more elements always leads to stronger crowding. Crowding is predominantly studied using sparse displays (a target surrounded by a few flankers). However, many studies have shown that this approach leads to wrong conclusions about human vision. Van der Burg and colleagues proposed a paradigm to measure crowding in dense displays using genetic algorithms. Displays were selected and combined over several generations to maximize human performance. In contrast to Bouma's law, only the target's nearest neighbours affected performance. Here, we tested various models to explain these results. We used the same genetic algorithm, but instead of selecting displays based on human performance we selected displays based on the model's outputs. We found that all models based on the traditional feedforward pooling framework of vision were unable to reproduce human behaviour. In contrast, all models involving a dedicated grouping stage explained the results successfully. We show how traditional models can be improved by adding a grouping stage.

A guideline for linking brain wave findings to the various aspects of discrete perception

Brain waves, determined by electrical and magnetic brain recordings (e.g., EEG and MEG), and fluctuating behavioral responses, determined by response time or accuracy measures, are frequently taken to support discrete perception. For example, it has been proposed that humans experience only one conscious percept per brain wave (e.g., during one alpha cycle). However, the proposed link between brain waves and discrete perception is typically rather vague. More importantly, there are many models and aspects of discrete perception and it is often not apparent in what theoretical framework brain wave findings are interpreted and to what specific aspects of discrete perception they relate. Here, we review different approaches to discrete perception and highlight issues with particular interpretations. We then discuss how certain findings on brain waves may relate to certain aspects of discrete perception. The main purpose of this meta‐contribution is to give a short overview of discrete models of perception and to illustrate the need to make explicit what aspects of discrete theories are addressed by what aspects of brain wave findings.

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

Individual differences in the perception of visual illusions are stable across eyes, time, and measurement methods

Vision scientists have tried to classify illusions for more than a century. For example, some studies suggested that there is a unique common factor for all visual illusions. Other studies proposed that there are several subclasses of illusions, such as illusions of linear extent or distortions. We previously observed strong within-illusion correlations but only weak between-illusion correlations, arguing in favor of an even higher multifactorial space with-more or less-each illusion making up its own factor. These mixed results are surprising. Here, we examined to what extent individual differences in the perception of visual illusions are stable across eyes, time, and measurement methods. First, we did not find any significant differences in the magnitudes of the seven illusions tested with monocular or binocular viewing conditions. In addition, illusion magnitudes were not significantly predicted by visual acuity. Second, we observed stable individual differences over time. Last, we compared two illusion measurements, namely an adjustment procedure and a method of constant stimuli, which both led to similar individual differences. Hence, it is unlikely that the individual differences in the perception of visual illusions arise from instability across eyes, time, and measurement methods.

Electrophysiological correlates of visual backward masking in patients with bipolar disorder

In visual backward masking (VBM), a target is followed by a mask that decreases target discriminability. Schizophrenia patients (SZ) show strong and reproducible masking impairments, which are associated with reduced EEG amplitudes. Patients with bipolar disorder (BP) show masking deficits, too. Here, we investigated the neural EEG correlates of VBM in BP. 122 SZ, 94 unaffected controls, and 38 BP joined a standard VBM experiment. 123 SZ, 94 unaffected controls and 16 BP joined a corresponding EEG experiment, analyzed in terms of global field power. As in previous studies, SZ and BP show strong masking deficits. Importantly and similarly to SZ, BP show decreased global field power amplitudes at approximately 200 ms after the target onset, compared to controls. These results suggest that VBM deficits are not specific for schizophrenia but for a broader range of functional psychoses. Potentially, both SZ and BP show deficient target enhancement.

Response to commentaries on 'hard criteria for empirical theories of consciousness'

In consciousness research, we have a very large number of theories, which exceeds by far the number of theories in other fields. We recently presented a set of criteria for evaluating and comparing theories of consciousness, and then applied the criteria to a number of different theories. Our publication sparked strong responses as evident by the many comments published in Cognitive Neuroscience (this issue). Overall, there seems to be consensus that a theory of consciousness (ToC) needs to have an unconscious alternative, but other criteria sparked controversy. The hottest debate is to what extent consciousness needs to work with purely 1^st person data, containing information not available in 3^rd person reports. We would like to thank all the commentators for their lively input and we look forward to continued dialog as theories evolve and compete.

All in Good Time: Long-Lasting Postdictive Effects Reveal Discrete Perception

Is consciousness a continuous stream of percepts or is it discrete, occurring only at certain moments in time? This question has puzzled philosophers, psychologists, and neuroscientists for centuries. Both hypotheses have fallen repeatedly in and out of favor. Here, we review recent studies exploring long-lasting postdictive effects and show that the results favor a two-stage discrete model, in which substantial periods of continuous unconscious processing precede discrete conscious percepts. We propose that such a model marries the advantages of both continuous and discrete models and resolves centuries old debates about perception and consciousness.

When illusions merge

We recently found only weak correlations between the susceptibility to various visual illusions. However, we observed strong correlations among different variants of an illusion, suggesting that the visual space of illusions includes several illusion-specific factors. Here, we specifically examined how factors for the vertical-horizontal, Müller-Lyer, and Ponzo illusions relate to each other. We measured the susceptibility to each illusion separately and to combinations of two illusions, which we refer to as a merged illusion; for example, we tested the Müller-Lyer illusion and the vertical-horizontal illusion, as well as a merged version of both illusions. We used an adjustment procedure in two experiments with 306 and 98 participants, respectively. Using path analyses, correlations, and exploratory factor analyses, we found that the susceptibility to a merged illusion is well predicted from the susceptibilities to the individual illusions. We suggest that there are illusion-specific factors that, by independent combinations, represent the whole visual structure underlying illusions.

Hard criteria for empirical theories of consciousness

Consciousness is now a well-established field of empirical research. A large body of experimental results has been accumulated and is steadily growing. In parallel, many Theories of Consciousness (ToCs) have been proposed. These theories are diverse in nature, ranging from computational to neurophysiological and quantum theoretical approaches. This contrasts with other fields of natural science, which host a smaller number of competing theories. We suggest that one reason for this abundance of extremely different theories may be the lack of stringent criteria specifying how empirical data constrains ToCs. First, we argue that consciousness is a well-defined topic from an empirical point of view and motivate a purely empirical stance on the quest for consciousness. Second, we present a checklist of criteria that, we propose, empirical ToCs need to cope with. Third, we review 13 of the most influential ToCs and subject them to the criteria. Our analysis helps to situate these different ToCs in the theoretical landscapeand sheds light on their strengths and weaknesses from a strictly empirical point of view.

Neural Compensation Mechanisms of Siblings of Schizophrenia Patients as Revealed by High-Density EEG

Visual backward masking (VBM) deficits are candidate endophenotypes of schizophrenia indexing genetic liability of the disorder. In VBM, a target is followed by a mask that deteriorates target perception. Schizophrenia patients and, to a lesser extent, their unaffected relatives show strong and reproducible VBM deficits. In patients, VBM deficits are associated with strongly decreased amplitudes in the evoked-related potentials (ERPs). Here, to unveil the neural mechanisms of VBM in schizophrenia, circumventing illness-specific confounds, we investigated the electroencephalogram correlates of VBM in unaffected siblings of schizophrenia patients. We tested 110 schizophrenia patients, 60 siblings, and 83 healthy controls. As in previous studies, patients showed strong behavioral deficits and decreased ERP amplitudes compared to controls. Surprisingly, the ERP amplitudes of siblings were even higher than the ones of controls, while their performances were similar. ERP amplitudes in siblings were found to correlate with performance. These results suggest that VBM is deteriorated in patients and siblings. However, siblings, unlike patients, can partially compensate for the deficits by over-activating a network of brain regions.

Object identity determines trans-saccadic integration

Humans make two to four rapid eye movements (saccades) per second, which, surprisingly, does not lead to abrupt changes in vision. To the contrary, we perceive a stable world. Hence, an important question is how information is integrated across saccades. To investigate this question, we used the sequential metacontrast paradigm (SQM), where two expanding streams of lines are presented. When one line is spatially offset, the other lines are perceived as being offset, too. When more lines are offset, all offsets integrate mandatorily; that is, observers cannot report the individual offsets but perceive one integrated offset. Here, we asked observers to make a saccade during the SQM. Even though the saccades caused a highly disrupted motion trajectory on the retina, offsets presented before and after the saccade integrated mandatorily. When observers made no saccade and the streams were displaced on the screen so that a similarly disrupted retinal image occurred as in the previous condition, no integration occurred. We suggest that trans-saccadic integration and perception are determined by object identity in spatiotopic coordinates and not by the retinal image.

Capsule networks as recurrent models of grouping and segmentation

Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

EEG microstates are a candidate endophenotype for schizophrenia

Electroencephalogram microstates are recurrent scalp potential configurations that remain stable for around 90 ms. The dynamics of two of the four canonical classes of microstates, commonly labeled as C and D, have been suggested as a potential endophenotype for schizophrenia. For endophenotypes, unaffected relatives of patients must show abnormalities compared to controls. Here, we examined microstate dynamics in resting-state recordings of unaffected siblings of patients with schizophrenia, patients with schizophrenia, healthy controls, and patients with first episodes of psychosis (FEP). Patients with schizophrenia and their siblings showed increased presence of microstate class C and decreased presence of microstate class D compared to controls. No difference was found between FEP and chronic patients. Our findings suggest that the dynamics of microstate classes C and D are a candidate endophenotype for schizophrenia.

Individual differences in the Müller-Lyer and Ponzo illusions are stable across different contexts

Vision scientists have attempted to classify visual illusions according to certain aspects, such as brightness or spatial features. For example, Piaget proposed that visual illusion magnitudes either decrease or increase with age. Subsequently, it was suggested that illusions are segregated according to their context: real-world contexts enhance and abstract contexts inhibit illusion magnitudes with age. We tested the effects of context on the Müller-Lyer and Ponzo illusions with a standard condition (no additional context), a line-drawing perspective condition, and a real-world perspective condition. A mixed-effects model analysis, based on data from 76 observers with ages ranging from 6 to 66 years, did not reveal any significant interaction between context and age. Although we found strong intra-illusion correlations for both illusions, we found only weak inter-illusion correlations, suggesting that the structure underlying these two spatial illusions includes several specific factors.

How stable is perception in #TheDress and #TheShoe?

#TheDress is perceived by some people as black and blue while others perceive it as white and gold. We have previously shown that the first encounter with #TheDress strongly biases its perception. This percept remained stable during the experiment, suggesting a role of one-shot learning. #TheShoe is another image that elicits similar bimodal color percepts. Here, we investigated how percepts change over time in both #TheShoe and #TheDress. First, we show that the important role of one-shot learning, which we found for #TheDress extends to #TheShoe. Similarly to our previous results with the dress, hiding large parts of the image with occluders biased the percept of the shoe. The percept did not change for the majority of observers when the occluders were removed. Second, we investigated if and how percepts switch over a time course of 14 days. We found that although some observers experienced percept switches, the percept was largely stable for most observers.

Crowding reveals fundamental differences in local vs. global processing in humans and machines

Feedforward Convolutional Neural Networks (ffCNNs) have become state-of-the-art models both in computer vision and neuroscience. However, human-like performance of ffCNNs does not necessarily imply human-like computations. Previous studies have suggested that current ffCNNs do not make use of global shape information. However, it is currently unclear whether this reflects fundamental differences between ffCNN and human processing or is merely an artefact of how ffCNNs are trained. Here, we use visual crowding as a well-controlled, specific probe to test global shape computations. Our results provide evidence that ffCNNs cannot produce human-like global shape computations for principled architectural reasons. We lay out approaches that may address shortcomings of ffCNNs to provide better models of the human visual system.

Electrophysiological correlates of visual backward masking in patients with major depressive disorder

Depression and schizophrenia are two psychiatric diseases with high co-morbidity. For this reason, it is important to find sensitive endophenotypes, which may disentangle the two disorders. The Shine-Through paradigm, a visual backward masking task, is a potential endophenotype for schizophrenia. Masking is strongly deteriorated in schizophrenia patients, which is reflected in reduced EEG amplitudes. Here, we tested whether masking deficits and associated EEG changes are also found in patients with major depressive disorder. First, we replicated previous findings showing that depressive patients exhibit, at most, only weak masking deficits. Second, we found that the EEG amplitudes of depressive patients were reduced compared to controls and slightly increased compared to schizophrenia patients. As a secondary analysis, we compared the performance in the masking paradigm with three cognitive tasks, namely: the Wisconsin card sorting test, a verbal fluency test and a degraded continuous performance test. Performance in all but the verbal fluency test could discriminate schizophrenia from depression.

One-shot learning and behavioral eligibility traces in sequential decision making

In many daily tasks, we make multiple decisions before reaching a goal. In order to learn such sequences of decisions, a mechanism to link earlier actions to later reward is necessary. Reinforcement learning (RL) theory suggests two classes of algorithms solving this credit assignment problem: In classic temporal-difference learning, earlier actions receive reward information only after multiple repetitions of the task, whereas models with eligibility traces reinforce entire sequences of actions from a single experience (one-shot). Here, we show one-shot learning of sequences. We developed a novel paradigm to directly observe which actions and states along a multi-step sequence are reinforced after a single reward. By focusing our analysis on those states for which RL with and without eligibility trace make qualitatively distinct predictions, we find direct behavioral (choice probability) and physiological (pupil dilation) signatures of reinforcement learning with eligibility trace across multiple sensory modalities.

Dominant men are faster in decision-making situations and exhibit a distinct neural signal for promptness

Social dominance, the main organizing principle of social hierarchies, facilitates priority access to resources by dominant individuals. Throughout taxa, individuals are more likely to become dominant if they act first in social situations and acting fast may provide evolutionary advantage; yet whether fast decision-making is a behavioral predisposition of dominant persons outside of social contexts is not known. Following characterization of participants for social dominance motivation, we found that, indeed, men high in social dominance respond faster–without loss of accuracy–than those low in dominance across a variety of decision-making tasks. Both groups did not differ in a simple reaction task. Then, we selected a decision-making task and applied high-density electroencephalography (EEG) to assess temporal dynamics of brain activation through event related potentials. We found that promptness to respond in the choice task in dominant individuals is related to a strikingly amplified brain signal at approximately 240 ms post-stimulus presentation. Source imaging analyses identified higher activity in the left insula and in the cingulate, right inferior temporal and right angular gyri in high than in low dominance participants. Our findings suggest that promptness to respond in choice situations, regardless of social context, is a biomarker for social disposition.