Deep Neural Networks for Modeling Visual Perceptual Learning

A hierarchical reinforcement learning model explains individual differences in attentional set shifting

Attentional set shifting refers to the ease with which the focus of attention is directed and switched. Cognitive tasks, such as the widely used CANTAB IED, reveal great variation in set shifting ability in the general population, with notable impairments in those with psychiatric diagnoses. The attentional and learning processes underlying this cognitive ability and how they lead to the observed variation remain unknown. To directly test this, we used a modelling approach on two independent large-scale online general-population samples performing CANTAB IED, with one including additional psychiatric symptom assessment. We found a hierarchical model that learnt both feature values and dimension attention best explained the data and that compulsive symptoms were associated with slower learning and higher attentional bias to the first relevant stimulus dimension. These data showcase a new methodology to analyse data from the CANTAB IED task, as well as suggest a possible mechanistic explanation for the variation in set shifting performance, and its relationship to compulsive symptoms.

Time courses of brain plasticity underpinning visual motion perceptual learning

Visual perceptual learning (VPL) refers to a long-term improvement of visual task performance through training or experience, reflecting brain plasticity even in adults. In human subjects, VPL has been mostly studied using functional magnetic resonance imaging (fMRI). However, due to the low temporal resolution of fMRI, how VPL affects the time course of visual information processing is largely unknown. To address this issue, we trained human subjects to perform a visual motion direction discrimination task. Their behavioral performance and magnetoencephalography (MEG) signals responding to the motion stimuli were measured before, immediately after, and two weeks after training. Training induced a long-lasting behavioral improvement for the trained direction. Based on the MEG signals from occipital sensors, we found that, for the trained motion direction, VPL increased the motion direction decoding accuracy, reduced the motion direction decoding latency, enhanced the direction-selective channel response, and narrowed the tuning profile. Following the MEG source reconstruction, we showed that VPL enhanced the cortical response in early visual cortex (EVC) and strengthened the feedforward connection from EVC to V3A. These VPL-induced neural changes co-occurred in 160-230 ms after stimulus onset. Complementary to previous fMRI findings on VPL, this study provides a comprehensive description on the neural mechanisms of visual motion perceptual learning from a temporal perspective and reveals how VPL shapes the time course of visual motion processing in the adult human brain.

Hearing temperatures: employing machine learning for elucidating the cross-modal perception of thermal properties through audition

People can use their sense of hearing for discerning thermal properties, though they are for the most part unaware that they can do so. While people unequivocally claim that they cannot perceive the temperature of pouring water through the auditory properties of hearing it being poured, our research further strengthens the understanding that they can. This multimodal ability is implicitly acquired in humans, likely through perceptual learning over the lifetime of exposure to the differences in the physical attributes of pouring water. In this study, we explore people's perception of this intriguing cross modal correspondence, and investigate the psychophysical foundations of this complex ecological mapping by employing machine learning. Our results show that not only can the auditory properties of pouring water be classified by humans in practice, the physical characteristics underlying this phenomenon can also be classified by a pre-trained deep neural network.

Feature variability determines specificity and transfer in multiorientation feature detection learning

Historically, in many perceptual learning experiments, only a single stimulus is practiced, and learning is often specific to the trained feature. Our prior work has demonstrated that multi-stimulus learning (e.g., training-plus-exposure procedure) has the potential to achieve generalization. Here, we investigated two important characteristics of multi-stimulus learning, namely, roving and feature variability, and their impacts on multi-stimulus learning and generalization. We adopted a feature detection task in which an oddly oriented target bar differed by 16° from the background bars. The stimulus onset asynchrony threshold between the target and the mask was measured with a staircase procedure. Observers were trained with four target orientation search stimuli, either with a 5° deviation (30°-35°-40°-45°) or with a 45° deviation (30°-75°-120°-165°), and the four reference stimuli were presented in a roving manner. The transfer of learning to the swapped target-background orientations was evaluated after training. We found that multi-stimulus training with a 5° deviation resulted in significant learning improvement, but learning failed to transfer to the swapped target-background orientations. In contrast, training with a 45° deviation slowed learning but produced a significant generalization to swapped orientations. Furthermore, a modified training-plus-exposure procedure, in which observers were trained with four orientation search stimuli with a 5° deviation and simultaneously passively exposed to orientations with high feature variability (45° deviation), led to significant orientation learning generalization. Learning transfer also occurred when the four orientation search stimuli with a 5° deviation were presented in separate blocks. These results help us to specify the condition under which multistimuli learning produces generalization, which holds potential for real-world applications of perceptual learning, such as vision rehabilitation and expert training.

Profiles of visual perceptual learning in feature space

Visual perceptual learning (VPL), experience-induced gains in discriminating visual features, has been studied extensively and intensively for many years, its profile in feature space, however, remains unclear. Here, human subjects were trained to perform either a simple low-level feature (grating orientation) or a complex high-level object (face view) discrimination task over a long-time course. During, immediately after, and one month after training, all results showed that in feature space VPL in grating orientation discrimination was a center-surround profile; VPL in face view discrimination, however, was a monotonic gradient profile. Importantly, these two profiles can be emerged by a deep convolutional neural network with a modified AlexNet consisted of 7 and 12 layers, respectively. Altogether, our study reveals for the first time a feature hierarchy-dependent profile of VPL in feature space, placing a necessary constraint on our understanding of the neural computation of VPL.

Asymmetric stimulus representations bias visual perceptual learning

The primate visual cortex contains various regions that exhibit specialization for different stimulus properties, such as motion, shape, and color. Within each region, there is often further specialization, such that particular stimulus features, such as horizontal and vertical orientations, are over-represented. These asymmetries are associated with well-known perceptual biases, but little is known about how they influence visual learning. Most theories would predict that learning is optimal, in the sense that it is unaffected by these asymmetries. However, other approaches to learning would result in specific patterns of perceptual biases. To distinguish between these possibilities, we trained human observers to discriminate between expanding and contracting motion patterns, which have a highly asymmetrical representation in the visual cortex. Observers exhibited biased percepts of these stimuli, and these biases were affected by training in ways that were often suboptimal. We simulated different neural network models and found that a learning rule that involved only adjustments to decision criteria, rather than connection weights, could account for our data. These results suggest that cortical asymmetries influence visual perception and that human observers often rely on suboptimal strategies for learning.

Subtle adversarial image manipulations influence both human and machine perception

Although artificial neural networks (ANNs) were inspired by the brain, ANNs exhibit a brittleness not generally observed in human perception. One shortcoming of ANNs is their susceptibility to adversarial perturbations-subtle modulations of natural images that result in changes to classification decisions, such as confidently mislabelling an image of an elephant, initially classified correctly, as a clock. In contrast, a human observer might well dismiss the perturbations as an innocuous imaging artifact. This phenomenon may point to a fundamental difference between human and machine perception, but it drives one to ask whether human sensitivity to adversarial perturbations might be revealed with appropriate behavioral measures. Here, we find that adversarial perturbations that fool ANNs similarly bias human choice. We further show that the effect is more likely driven by higher-order statistics of natural images to which both humans and ANNs are sensitive, rather than by the detailed architecture of the ANN.

Variability in training unlocks generalization in visual perceptual learning through invariant representations

Stimulus and location specificity are long considered hallmarks of visual perceptual learning. This renders visual perceptual learning distinct from other forms of learning, where generalization can be more easily attained, and therefore unsuitable for practical applications, where generalization is key. Based on the hypotheses derived from the structure of the visual system, we test here whether stimulus variability can unlock generalization in perceptual learning. We train subjects in orientation discrimination, while we vary the amount of variability in a task-irrelevant feature, spatial frequency. We find that, independently of task difficulty, this manipulation enables generalization of learning to new stimuli and locations, while not negatively affecting the overall amount of learning on the task. We then use deep neural networks to investigate how variability unlocks generalization. We find that networks develop invariance to the task-irrelevant feature when trained with variable inputs. The degree of learned invariance strongly predicts generalization. A reliance on invariant representations can explain variability-induced generalization in visual perceptual learning. This suggests new targets for understanding the neural basis of perceptual learning in the higher-order visual cortex and presents an easy-to-implement modification of common training paradigms that may benefit practical applications.

Perceptual learning: Breaking specificity by variability

A new psychophysical study in humans suggests that increasing the variability of task-irrelevant features during training enhances the generalization of visual perceptual learning to untrained stimuli and locations.

Efficient neural codes naturally emerge through gradient descent learning

Human sensory systems are more sensitive to common features in the environment than uncommon features. For example, small deviations from the more frequently encountered horizontal orientations can be more easily detected than small deviations from the less frequent diagonal ones. Here we find that artificial neural networks trained to recognize objects also have patterns of sensitivity that match the statistics of features in images. To interpret these findings, we show mathematically that learning with gradient descent in neural networks preferentially creates representations that are more sensitive to common features, a hallmark of efficient coding. This effect occurs in systems with otherwise unconstrained coding resources, and additionally when learning towards both supervised and unsupervised objectives. This result demonstrates that efficient codes can naturally emerge from gradient-like learning.

Minimum perturbation theory of deep perceptual learning

Perceptual learning (PL) involves long-lasting improvement in perceptual tasks following extensive training and is accompanied by modified neuronal responses in sensory cortical areas in the brain. Understanding the dynamics of PL and the resultant synaptic changes is important for causally connecting PL to the observed neural plasticity. This is theoretically challenging because learning-related changes are distributed across many stages of the sensory hierarchy. In this paper, we modeled the sensory hierarchy as a deep nonlinear neural network and studied PL of fine discrimination, a common and well-studied paradigm of PL. Using tools from statistical physics, we developed a mean-field theory of the network in the limit of a large number of neurons and large number of examples. Our theory suggests that, in this thermodynamic limit, the input-output function of the network can be exactly mapped to that of a deep linear network, allowing us to characterize the space of solutions for the task. Surprisingly, we found that modifying synaptic weights in the first layer of the hierarchy is both sufficient and necessary for PL. To address the degeneracy of the space of solutions, we postulate that PL dynamics are constrained by a normative minimum perturbation (MP) principle, which favors weight matrices with minimal changes relative to their prelearning values. Interestingly, MP plasticity induces changes to weights and neural representations in all layers of the network, except for the readout weight vector. While weight changes in higher layers are not necessary for learning, they help reduce overall perturbation to the network. In addition, such plasticity can be learned simply through slow learning. We further elucidate the properties of MP changes and compare them against experimental findings. Overall, our statistical mechanics theory of PL provides mechanistic and normative understanding of several important empirical findings of PL.

Unifying framework for cognitive training interventions in brain aging

Cognitive training is a promising tool for slowing or preventing cognitive decline in older adults at-risk for dementia. Its success, however, has been limited by a lack of evidence showing that it reliably causes broad training effects: improvements in cognition across a range of domains that lead to real-world benefits. Here, we propose a framework for enhancing the effect of cognitive training interventions in brain aging. The focus is on (A) developing cognitive training task paradigms that are informed by population-level cognitive characteristics and pathophysiology, and (B) personalizing how these sets are presented to participants during training via feedback loops that aim to optimize "mismatch" between participant capacity and training demands using both adaptation and random variability. In this way, cognitive training can better alter whole-brain topology in a manner that supports broad training effects in the context of brain aging.

Current directions in visual perceptual learning

The visual expertise of adult humans is jointly determined by evolution, visual development, and visual perceptual learning. Perceptual learning refers to performance improvements in perceptual tasks after practice or training in the task. It occurs in almost all visual tasks, ranging from simple feature detection to complex scene analysis. In this Review, we focus on key behavioral aspects of visual perceptual learning. We begin by describing visual perceptual learning tasks and manipulations that influence the magnitude of learning, and then discuss specificity of learning. Next, we present theories and computational models of learning and specificity. We then review applications of visual perceptual learning in visual rehabilitation. Finally, we summarize the general principles of visual perceptual learning, discuss the tension between plasticity and stability, and conclude with new research directions.

Brain-inspired chaotic backpropagation for MLP

Backpropagation (BP) algorithm is one of the most basic learning algorithms in deep learning. Although BP has been widely used, it still suffers from the problem of easily falling into the local minima due to its gradient dynamics. Inspired by the fact that the learning of real brains may exploit chaotic dynamics, we propose the chaotic backpropagation (CBP) algorithm by integrating the intrinsic chaos of real neurons into BP. By validating on multiple datasets (e.g. cifar10), we show that, for multilayer perception (MLP), CBP has significantly better abilities than those of BP and its variants in terms of optimization and generalization from both computational and theoretical viewpoints. Actually, CBP can be regarded as a general form of BP with global searching ability inspired by the chaotic learning process in the brain. Therefore, CBP not only has the potential of complementing or replacing BP in deep learning practice, but also provides a new way for understanding the learning process of the real brain.

Statistical Learning in Vision

Vision and learning have long been considered to be two areas of research linked only distantly. However, recent developments in vision research have changed the conceptual definition of vision from a signal-evaluating process to a goal-oriented interpreting process, and this shift binds learning, together with the resulting internal representations, intimately to vision. In this review, we consider various types of learning (perceptual, statistical, and rule/abstract) associated with vision in the past decades and argue that they represent differently specialized versions of the fundamental learning process, which must be captured in its entirety when applied to complex visual processes. We show why the generalized version of statistical learning can provide the appropriate setup for such a unified treatment of learning in vision, what computational framework best accommodates this kind of statistical learning, and what plausible neural scheme could feasibly implement this framework. Finally, we list the challenges that the field of statistical learning faces in fulfilling the promise of being the right vehicle for advancing our understanding of vision in its entirety. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Deep networks may capture biological behavior for shallow, but not deep, empirical characterizations

We assess whether deep convolutional networks (DCN) can account for a most fundamental property of human vision: detection/discrimination of elementary image elements (bars) at different contrast levels. The human visual process can be characterized to varying degrees of "depth," ranging from percentage of correct detection to detailed tuning and operating characteristics of the underlying perceptual mechanism. We challenge deep networks with the same stimuli/tasks used with human observers and apply equivalent characterization of the stimulus-response coupling. In general, we find that popular DCN architectures do not account for signature properties of the human process. For shallow depth of characterization, some variants of network-architecture/training-protocol produce human-like trends; however, more articulate empirical descriptors expose glaring discrepancies. Networks can be coaxed into learning those richer descriptors by shadowing a human surrogate in the form of a tailored circuit perturbed by unstructured input, thus ruling out the possibility that human-model misalignment in standard protocols may be attributable to insufficient representational power. These results urge caution in assessing whether neural networks do or do not capture human behavior: ultimately, our ability to assess "success" in this area can only be as good as afforded by the depth of behavioral characterization against which the network is evaluated. We propose a novel set of metrics/protocols that impose stringent constraints on the evaluation of DCN behavior as an adequate approximation to biological processes.

Emerged human-like facial expression representation in a deep convolutional neural network

Recent studies found that the deep convolutional neural networks (DCNNs) trained to recognize facial identities spontaneously learned features that support facial expression recognition, and vice versa. Here, we showed that the self-emerged expression-selective units in a VGG-Face trained for facial identification were tuned to distinct basic expressions and, importantly, exhibited hallmarks of human expression recognition (i.e., facial expression confusion and categorical perception). We then investigated whether the emergence of expression-selective units is attributed to either face-specific experience or domain-general processing by conducting the same analysis on a VGG-16 trained for object classification and an untrained VGG-Face without any visual experience, both having the identical architecture with the pretrained VGG-Face. Although similar expression-selective units were found in both DCNNs, they did not exhibit reliable human-like characteristics of facial expression perception. Together, these findings revealed the necessity of domain-specific visual experience of face identity for the development of facial expression perception, highlighting the contribution of nurture to form human-like facial expression perception.

Novel Cognitive Functions Arise at the Convergence of Macroscale Gradients

Functions in higher-order brain regions are the source of extensive debate. Although past trends have been to describe the brain-especially posterior cortical areas-in terms of a set of functional modules, a new emerging paradigm focuses on the integration of proximal functions. In this review, we synthesize emerging evidence that a variety of novel functions in the higher-order brain regions are due to convergence: convergence of macroscale gradients brings feature-rich representations into close proximity, presenting an opportunity for novel functions to arise. Using the TPJ as an example, we demonstrate that convergence is enabled via three properties of the brain: (1) hierarchical organization, (2) abstraction, and (3) equidistance. As gradients travel from primary sensory cortices to higher-order brain regions, information becomes abstracted and hierarchical, and eventually, gradients meet at a point maximally and equally distant from their sensory origins. This convergence, which produces multifaceted combinations, such as mentalizing another person's thought or projecting into a future space, parallels evolutionary and developmental characteristics in such regions, resulting in new cognitive and affective faculties.

Stimulation of the ventral tegmental area induces visual cortical plasticity at the neuronal level

fMRI studies have shown that pairing a task-irrelevant visual feature with electrical micro-stimulation of the ventral tegmental area (VTA-EM) is sufficient to increase the sensory cortical representation of the paired feature and to improve perceptual performance. However, since fMRI provides an indirect measure of neural activity, the neural response changes underlying the fMRI activations are unknown. Here, we pair a task-irrelevant grating orientation with VTA-EM while attention is directed to a difficult orthogonal task. We examine the changes in neural response properties in macaques by recording spiking activity in the posterior inferior temporal cortex, the locus of fMRI-defined plasticity in previous studies. We observe a relative increase in mean spike rate and preference for the VTA-EM paired orientation compared to an unpaired orientation, which is unrelated to attention. These results demonstrate that VTA-EM-stimulus pairing is sufficient to induce sensory cortical plasticity at the spiking level in nonhuman primates.

Emergence of prefrontal neuron maturation properties by training recurrent neural networks in cognitive tasks

Working memory and response inhibition are functions that mature relatively late in life, after adolescence, paralleling the maturation of the prefrontal cortex. The link between behavioral and neural maturation is not obvious, however, making it challenging to understand how neural activity underlies the maturation of cognitive function. To gain insights into the nature of observed changes in prefrontal activity between adolescence and adulthood, we investigated the progressive changes in unit activity of recurrent neural networks as they were trained to perform working memory and response inhibition tasks. These included increased delay period activity during working memory tasks and increased activation in antisaccade tasks. These findings reveal universal properties underlying the neuronal computations behind cognitive tasks and explicate the nature of changes that occur as the result of developmental maturation.

•

Properties of RNN networks during training offer insights in prefrontal maturation

•

Fully trained networks exhibit higher levels of activity in working memory tasks

•

Trained networks also exhibit higher activation in antisaccade tasks

•

Partially trained RNNs can generate accurate predictions of immature PFC activity

ImageNet-trained deep neural networks exhibit illusion-like response to the Scintillating grid

Deep neural network (DNN) models for computer vision are capable of human-level object recognition. Consequently, similarities between DNN and human vision are of interest. Here, we characterize DNN representations of Scintillating grid visual illusion images in which white disks are perceived to be partially black. Specifically, we use VGG-19 and ResNet-101 DNN models that were trained for image classification and consider the representational dissimilarity (\(L^1\) distance in the penultimate layer) between pairs of images: one with white Scintillating grid disks and the other with disks of decreasing luminance levels. Results showed a nonmonotonic relation, such that decreasing disk luminance led to an increase and subsequently a decrease in representational dissimilarity. That is, the Scintillating grid image with white disks was closer, in terms of the representation, to images with black disks than images with gray disks. In control nonillusion images, such nonmonotonicity was rare. These results suggest that nonmonotonicity in a deep computational representation is a potential test for illusion-like response geometry in DNN models.

Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future

Convolutional neural networks (CNNs) were inspired by early findings in the study of biological vision. They have since become successful tools in computer vision and state-of-the-art models of both neural activity and behavior on visual tasks. This review highlights what, in the context of CNNs, it means to be a good model in computational neuroscience and the various ways models can provide insight. Specifically, it covers the origins of CNNs and the methods by which we validate them as models of biological vision. It then goes on to elaborate on what we can learn about biological vision by understanding and experimenting on CNNs and discusses emerging opportunities for the use of CNNs in vision research beyond basic object recognition.

Effects of daily training amount on visual motion perceptual learning

Perceptual learning has been widely used to study the plasticity of the visual system in adults. Owing to the belief that practice makes perfect, perceptual learning protocols usually require subjects to practice a task thousands of times over days, even weeks. However, we know very little about the relationship between training amount and behavioral improvement. Here, four groups of subjects underwent motion direction discrimination training over 8 days with 40, 120, 360, or 1080 trials per day. Surprisingly, different daily training amounts induced similar improvement across the four groups, and the similarity lasted for at least 2 weeks. Moreover, the group with 40 training trials per day showed more learning transfer from the trained direction to the untrained directions than the group with 1080 training trials per day immediately after training and 2 weeks later. These findings suggest that perceptual learning of motion direction discrimination is not always dependent on the daily training amount and less training leads to more transfer.

If deep learning is the answer, what is the question?

Neuroscience research is undergoing a minor revolution. Recent advances in machine learning and artificial intelligence research have opened up new ways of thinking about neural computation. Many researchers are excited by the possibility that deep neural networks may offer theories of perception, cognition and action for biological brains. This approach has the potential to radically reshape our approach to understanding neural systems, because the computations performed by deep networks are learned from experience, and not endowed by the researcher. If so, how can neuroscientists use deep networks to model and understand biological brains? What is the outlook for neuroscientists who seek to characterize computations or neural codes, or who wish to understand perception, attention, memory and executive functions? In this Perspective, our goal is to offer a road map for systems neuroscience research in the age of deep learning. We discuss the conceptual and methodological challenges of comparing behaviour, learning dynamics and neural representations in artificial and biological systems, and we highlight new research questions that have emerged for neuroscience as a direct consequence of recent advances in machine learning.

Supervised Learning Occurs in Visual Perceptual Learning of Complex Natural Images

There have been long-standing debates regarding whether supervised or unsupervised learning mechanisms are involved in visual perceptual learning (VPL) [1-14]. However, these debates have been based on the effects of simple feedback only about response accuracy in detection or discrimination tasks of low-level visual features such as orientation [15-22]. Here, we examined whether the content of response feedback plays a critical role for the acquisition and long-term retention of VPL of complex natural images. We trained three groups of human subjects (n = 72 in total) to better detect "grouped microcalcifications" or "architectural distortion" lesions (referred to as calcification and distortion in the following) in mammograms either with no trial-by-trial feedback, partial trial-by-trial feedback (response correctness only), or detailed trial-by-trial feedback (response correctness and target location). Distortion lesions consist of more complex visual structures than calcification lesions [23-26]. We found that partial feedback is necessary for VPL of calcifications, whereas detailed feedback is required for VPL of distortions. Furthermore, detailed feedback during training is necessary for VPL of distortion and calcification lesions to be retained for 6 months. These results show that although supervised learning is heavily involved in VPL of complex natural images, the extent of supervision for VPL varies across different types of complex natural images. Such differential requirements for VPL to improve the detectability of lesions in mammograms are potentially informative for the professional training of radiologists.

Application of Machine Learning Models for Tracking Participant Skills in Cognitive Training

A key need in cognitive training interventions is to personalize task difficulty to each user and to adapt this difficulty to continually apply appropriate challenges as users improve their skill to perform the tasks. Here we examine how Bayesian filtering approaches, such as hidden Markov models and Kalman filters, and deep-learning approaches, such as the long short-term memory (LSTM) model, may be useful methods to estimate user skill level and predict appropriate task challenges. A possible advantage of these models over commonly used adaptive methods, such as staircases or blockwise adjustment methods that are based only upon recent performance, is that Bayesian filtering and deep learning approaches can model the trajectory of user performance across multiple sessions and incorporate data from multiple users to optimize local estimates. As a proof of concept, we fit data from two large cohorts of undergraduate students performing WM training using an N-back task. Results show that all three models predict appropriate challenges for different users. However, the hidden Markov models were most accurate in predicting participants' performances as a function of provided challenges, and thus, they placed participants at appropriate future challenges. These data provide good support for the potential of machine learning approaches as appropriate methods to personalize task performance to users in tasks that require adaptively determined challenges.

Dissociating electrophysiological correlates of contextual and perceptual learning in a visual search task

Perceptual learning and contextual learning are two types of implicit visual learning that can co-occur in the same tasks. For example, to find an animal in the woods, you need to know where to look in the environment (contextual learning) and you must be able to discriminate its features (perceptual learning). However, contextual and perceptual learning are typically studied using distinct experimental paradigms, and little is known regarding their comparative neural mechanisms. In this study, we investigated contextual and perceptual learning in 12 healthy adult humans as they performed the same visual search task, and we examined psychophysical and electrophysiological (event-related potentials) measures of learning. Participants were trained to look for a visual stimulus, a small line with a specific orientation, presented among distractors. We found better performance for the trained target orientation as compared to an untrained control orientation, reflecting specificity of perceptual learning for the orientation of trained elements. This orientation specificity effect was associated with changes in the C1 component. We also found better performance for repeated spatial configurations as compared to novel ones, reflecting contextual learning. This context-specific effect was associated with the N2pc component. Taken together, these results suggest that contextual and perceptual learning are distinct visual learning phenomena that have different behavioral and electrophysiological characteristics.

Moderating effects of visual attention and action video game play on perceptual learning with the texture discrimination task

There is currently substantial controversy regarding the reliability of observed patterns of perceptual learning. Contributing to this controversy are a lack of accounting for individual differences and how variations in training can give rise to different patterns of learning. Here we sought to investigate the impact of individual differences in attention, as measured with the Useful Field of View (UFOV) task, and action video game use on perceptual learning in a large sample of subjects trained on a Texture Discrimination Task (TDT). We examined baseline performance on the TDT, learning on the initially trained TDT stimuli and transfer to a subsequently trained background orientation. We find that participants showing better performance on the UFOV task performed better on the TDT, and also showed greater learning and transfer to an untrained background orientation. On the other hand, self-report of action video game play only inconsistently related performance, learning or transfer on the TDT. Further, we failed to replicate previous findings that training with different backgrounds gives rise to interference on the TDT. Together these results suggest that, while differences between individuals and differences in task structure play a role in perceptual learning, previous findings on the impact of action video game use and interference between training stimuli in perceptual learning may be idiosyncratic.

Can machine learning account for human visual object shape similarity judgments?

We describe and analyze the performance of metric learning systems, including deep neural networks (DNNs), on a new dataset of human visual object shape similarity judgments of naturalistic, part-based objects known as "Fribbles". In contrast to previous studies which asked participants to judge similarity when objects or scenes were rendered from a single viewpoint, we rendered Fribbles from multiple viewpoints and asked participants to judge shape similarity in a viewpoint-invariant manner. Metrics trained using pixel-based or DNN-based representations fail to explain our experimental data, but a metric trained with a viewpoint-invariant, part-based representation produces a good fit. We also find that although neural networks can learn to extract the part-based representation-and therefore should be capable of learning to model our data-networks trained with a "triplet loss" function based on similarity judgments do not perform well. We analyze this failure, providing a mathematical description of the relationship between the metric learning objective function and the triplet loss function. The poor performance of neural networks appears to be due to the nonconvexity of the optimization problem in network weight space. We conclude that viewpoint insensitivity is a critical aspect of human visual shape perception, and that neural network and other machine learning methods will need to learn viewpoint-insensitive representations in order to account for people's visual object shape similarity judgments.

Can the Brain Do Backpropagation? -Exact Implementation of Backpropagation in Predictive Coding Networks

Backpropagation (BP) has been the most successful algorithm used to train artificial neural networks. However, there are several gaps between BP and learning in biologically plausible neuronal networks of the brain (learning in the brain, or simply BL, for short), in particular, (1) it has been unclear to date, if BP can be implemented exactly via BL, (2) there is a lack of local plasticity in BP, i.e., weight updates require information that is not locally available, while BL utilizes only locally available information, and (3) there is a lack of autonomy in BP, i.e., some external control over the neural network is required (e.g., switching between prediction and learning stages requires changes to dynamics and synaptic plasticity rules), while BL works fully autonomously. Bridging such gaps, i.e., understanding how BP can be approximated by BL, has been of major interest in both neuroscience and machine learning. Despite tremendous efforts, however, no previous model has bridged the gaps at a degree of demonstrating an equivalence to BP, instead, only approximations to BP have been shown. Here, we present for the first time a framework within BL that bridges the above crucial gaps. We propose a BL model that (1) produces exactly the same updates of the neural weights as BP, while (2) employing local plasticity, i.e., all neurons perform only local computations, done simultaneously. We then modify it to an alternative BL model that (3) also works fully autonomously. Overall, our work provides important evidence for the debate on the long-disputed question whether the brain can perform BP.

The quantization error in a Self-Organizing Map as a contrast and colour specific indicator of single-pixel change in large random patterns

The quantization error in a fixed-size Self-Organizing Map (SOM) with unsupervised winner-take-all learning has previously been used successfully to detect, in minimal computation time, highly meaningful changes across images in medical time series and in time series of satellite images. Here, the functional properties of the quantization error in SOM are explored further to show that the metric is capable of reliably discriminating between the finest differences in local contrast intensities and contrast signs. While this capability of the QE is akin to functional characteristics of a specific class of retinal ganglion cells (the so-called Y-cells) in the visual systems of the primate and the cat, the sensitivity of the QE surpasses the capacity limits of human visual detection. Here, the quantization error in the SOM is found to reliably signal changes in contrast or colour when contrast information is removed from or added to the image, but not when the amount and relative weight of contrast information is constant and only the local spatial position of contrast elements in the pattern changes. While the RGB Mean reflects coarser changes in colour or contrast well enough, the SOM-QE is shown to outperform the RGB Mean in the detection of single-pixel changes in images with up to five million pixels. This could have important implications in the context of unsupervised image learning and computational building block approaches to large sets of image data (big data), including deep learning blocks, and automatic detection of contrast change at the nanoscale in Transmission or Scanning Electron Micrographs (TEM, SEM), or at the subpixel level in multispectral and hyper-spectral imaging data.

Interfering with a memory without erasing its trace

Previous research has shown that performance of a novice skill can be easily interfered with by subsequent training of another skill. We address the open questions whether extensively trained skills show the same vulnerability to interference as novice skills and which memory mechanism regulates interference between expert skills. We developed a recurrent neural network model of V1 able to learn from feedback experienced over the course of a long-term orientation discrimination experiment. After first exposing the model to one discrimination task for 3480 consecutive trials, we assessed how its performance was affected by subsequent training in a second, similar task. Training the second task strongly interfered with the first (highly trained) discrimination skill. The magnitude of interference depended on the relative amounts of training devoted to the different tasks. We used these and other model outcomes as predictions for a perceptual learning experiment in which human participants underwent the same training protocol as our model. Specifically, over the course of three months participants underwent baseline training in one orientation discrimination task for 15 sessions before being trained for 15 sessions on a similar task and finally undergoing another 15 sessions of training on the first task (to assess interference). Across all conditions, the pattern of interference observed empirically closely matched model predictions. According to our model, behavioral interference can be explained by antagonistic changes in neuronal tuning induced by the two tasks. Remarkably, this did not stem from erasing connections due to earlier learning but rather from a reweighting of lateral inhibition.

Systematic Analysis of Pigeons' Discrimination of Pixelated Stimuli: A Hierarchical Pattern Recognition System Is Not Identifiable

Pigeons learned to discriminate two different patterns displayed with miniature light-emitting diode arrays. They were then tested with 84 interspersed, non-reinforced degraded pattern pairs. Choices ranged between 100% and 50% for one or other of the patterns. Stimuli consisting of few pixels yielded low choice scores whereas those consisting of many pixels yielded a broad range of scores. Those patterns with a high number of pixels coinciding with those of the rewarded training stimulus were preferred and those with a high number of pixels coinciding with the non-rewarded training pattern were avoided; a discrimination index based on this correlated 0.74 with the pattern choices. Pixels common to both training patterns had a minimal influence. A pixel-by-pixel analysis revealed that eight pixels of one pattern and six pixels of the other pattern played a prominent role in the pigeons’ choices. These pixels were disposed in four and two clusters of neighbouring locations. A summary index calculated on this basis still only yielded a weak 0.73 correlation. The individual pigeons’ data furthermore showed that these clusters were a mere averaging mirage. The pigeons’ performance depends on deep learning in a midbrain-based multimillion synapse neuronal network. Pixelated visual patterns should be helpful when simulating perception of patterns with artificial networks.

Can multisensory training aid visual learning? A computational investigation

Although real-world environments are often multisensory, visual scientists typically study visual learning in unisensory environments containing visual signals only. Here, we use deep or artificial neural networks to address the question, Can multisensory training aid visual learning? We examine a network's internal representations of objects based on visual signals in two conditions: (a) when the network is initially trained with both visual and haptic signals, and (b) when it is initially trained with visual signals only. Our results demonstrate that a network trained in a visual-haptic environment (in which visual, but not haptic, signals are orientation-dependent) tends to learn visual representations containing useful abstractions, such as the categorical structure of objects, and also learns representations that are less sensitive to imaging parameters, such as viewpoint or orientation, that are irrelevant for object recognition or classification tasks. We conclude that researchers studying perceptual learning in vision-only contexts may be overestimating the difficulties associated with important perceptual learning problems. Although multisensory perception has its own challenges, perceptual learning can become easier when it is considered in a multisensory setting.

The quantization error in a Self-Organizing Map as a contrast and colour specific indicator of single-pixel change in large random patterns

The quantization error in a fixed-size Self-Organizing Map (SOM) with unsupervised winner-take-all learning has previously been used successfully to detect, in minimal computation time, highly meaningful changes across images in medical time series and in time series of satellite images. Here, the functional properties of the quantization error in SOM are explored further to show that the metric is capable of reliably discriminating between the finest differences in local contrast intensities and contrast signs. While this capability of the QE is akin to functional characteristics of a specific class of retinal ganglion cells (the so-called Y-cells) in the visual systems of the primate and the cat, the sensitivity of the QE surpasses the capacity limits of human visual detection. Here, the quantization error in the SOM is found to reliably signal changes in contrast or colour when contrast information is removed from or added to the image, but not when the amount and relative weight of contrast information is constant and only the local spatial position of contrast elements in the pattern changes. While the RGB Mean reflects coarser changes in colour or contrast well enough, the SOM-QE is shown to outperform the RGB Mean in the detection of single-pixel changes in images with up to five million pixels. This could have important implications in the context of unsupervised image learning and computational building block approaches to large sets of image data (big data), including deep learning blocks, and automatic detection of contrast change at the nanoscale in Transmission or Scanning Electron Micrographs (TEM, SEM), or at the subpixel level in multispectral and hyper-spectral imaging data.

Occam’s Razor for Big Data? On Detecting Quality in Large Unstructured Datasets

Detecting quality in large unstructured datasets requires capacities far beyond the limits of human perception and communicability and, as a result, there is an emerging trend towards increasingly complex analytic solutions in data science to cope with this problem. This new trend towards analytic complexity represents a severe challenge for the principle of parsimony (Occam’s razor) in science. This review article combines insight from various domains such as physics, computational science, data engineering, and cognitive science to review the specific properties of big data. Problems for detecting data quality without losing the principle of parsimony are then highlighted on the basis of specific examples. Computational building block approaches for data clustering can help to deal with large unstructured datasets in minimized computation time, and meaning can be extracted rapidly from large sets of unstructured image or video data parsimoniously through relatively simple unsupervised machine learning algorithms. Why we still massively lack in expertise for exploiting big data wisely to extract relevant information for specific tasks, recognize patterns and generate new information, or simply store and further process large amounts of sensor data is then reviewed, and examples illustrating why we need subjective views and pragmatic methods to analyze big data contents are brought forward. The review concludes on how cultural differences between East and West are likely to affect the course of big data analytics, and the development of increasingly autonomous artificial intelligence (AI) aimed at coping with the big data deluge in the near future.