Feedforward and feedback processes in vision

Unveiling the neural dynamics of conscious perception in rapid object recognition

Our brain excels at recognizing objects, even when they flash by in a rapid sequence. However, the neural processes determining whether a target image in a rapid sequence can be recognized or not remains elusive. We used electroencephalography (EEG) to investigate the temporal dynamics of brain processes that shape perceptual outcomes in these challenging viewing conditions. Using naturalistic images and advanced multivariate pattern analysis (MVPA) techniques, we probed the brain dynamics governing conscious object recognition. Our results show that although initially similar, the processes for when an object can or cannot be recognized diverge around 180 ms post-appearance, coinciding with feedback neural processes. Decoding analyses indicate that gist perception (partial conscious perception) can occur at ∼120 ms through feedforward mechanisms. In contrast, object identification (full conscious perception of the image) is resolved at ∼190 ms after target onset, suggesting involvement of recurrent processing. These findings underscore the importance of recurrent neural connections in object recognition and awareness in rapid visual presentations.

Narrative simulation of social experiences in naturalistic context - A neurocinematic approach

Narratives may be regarded as simulations of everyday social situations. They are key to studying the human mind in socio-culturally determined contexts as they allow anchoring to the common ground of embodied and environmentally-engaged cognition. Here we review recent findings from naturalistic neuroscience on neural functions in conditions that mimic lifelike situations. We will focus particularly on neurocinematics, a research field that applies mediated narratives as stimuli for neuroimaging experiments. During the last two decades, this paradigm has contributed to an accumulation of insights about the neural underpinnings of behavior and sense-making in various narratively contextualized situations particularly pertaining to socio-emotional encounters. One of the key questions in neurocinematics is, how do intersubjectively synchronized brain activations relate to subjective experiences? Another question we address is how to bring natural contexts into experimental studies. Seeking to respond to both questions, we suggest neurocinematic studies to examine three manifestations of the same phenomenon side-by-side: subjective experiences of narrative situations, unfolding of narrative stimulus structure, and neural processes that co-constitute the experience. This approach facilitates identifying experientially meaningful activity patterns in the brain and points out what they may mean in relation to shared and communicable contents. Via rich-featured and temporally contextualized narrative stimuli, neurocinematics attempts to contribute to emerging holistic theories of neural dynamics and connectomics explaining typical and atypical interindividual variability.

Effects of saccade delay, side of deficit, and training on detection of catch-up saccades during head-impulse test in virtual-reality-enhanced mannequin

In this study, a training simulator for the examination of dizzy patients based on a virtual-reality-enhanced mannequin (VREM) was developed to evaluate the detection of catch-up saccades during head impulse test (HIT) and the effect of training in VREM. For novices (n = 35), 2 trials were conducted before and after a training session. Experts (n = 7) were submitted to an evaluation session. In each trial, a left or a right horizontal canal deficit with an overt catch-up saccade (delay between 110 and 320 ms) was randomly presented. Participants scored the difficulty in performing the maneuver, in recognizing the saccades, and the self-confidence in the diagnosis using a visual analogue scale (VAS). Saccade delay significantly influenced the performance. Training significantly improved the sensitivity in the residents (69.1% before to 97.9% after the training, p < 0.001, Fisher's exact test, n = 560 tests), surpassing experts' performances (p < 0.001, versus 87% in experts, Fisher's exact test). The specificity also increased to the expert level (78% before to 95% after the training, and 95% in experts, p < 0.001, Fisher's exact test). The VAS showed a decrease difficulty to execute the HIT, with an increase in the confidence after training. VREM improved the HIT execution performance and the confidence in novice practitioners.

The color phi phenomenon: Not so special, after all?

We show how anomalous time reversal of stimuli and their associated responses can exist in very small connectionist models. These networks are built from dynamical toy model neurons which adhere to a minimal set of biologically plausible properties. The appearance of a "ghost" response, temporally and spatially located in between responses caused by actual stimuli, as in the phi phenomenon, is demonstrated in a similar small network, where it is caused by priming and long-distance feedforward paths. We then demonstrate that the color phi phenomenon can be present in an echo state network, a recurrent neural network, without explicitly training for the presence of the effect, such that it emerges as an artifact of the dynamical processing. Our results suggest that the color phi phenomenon might simply be a feature of the inherent dynamical and nonlinear sensory processing in the brain and in and of itself is not related to consciousness.

Neural correlates of metacontrast masking across different contrast polarities

Metacontrast masking is a powerful illusion to investigate the dynamics of perceptual processing and to control conscious visual perception. However, the neural mechanisms underlying this fundamental investigative tool are still debated. In the present study, we examined metacontrast masking across different contrast polarities by employing a contour discrimination task combined with EEG (Electroencephalography). When the target and mask had the same contrast polarity, a typical U-shaped metacontrast function was observed. A change in mask polarity (i.e., opposite mask polarity) shifted this masking function to a monotonic increasing function such that the target visibility was strongly suppressed at stimulus onset asynchronies less than 50 ms. This transition in metacontrast function has been typically interpreted as an increase in intrachannel inhibition of the sustained activities functionally linked to object visibility and identity. Our EEG analyses revealed an early (160-300 ms) and a late (300-550 ms) spatiotemporal cluster associated with this effect of polarity. The early cluster was mainly over occipital and parieto-occipital scalp sites. On the other hand, the later modulations of the evoked activities were centered over parietal and centro-parietal sites. Since both of these clusters were beyond 160 ms, the EEG results point to late recurrent inhibitory mechanisms. Although the findings here do not directly preclude other proposed mechanisms for metacontrast, they highlight the involvement of recurrent intrachannel inhibition in metacontrast masking.

Facilitating the propagation of spiking activity in feedforward networks by including feedback

Transient oscillations in network activity upon sensory stimulation have been reported in different sensory areas of the brain. These evoked oscillations are the generic response of networks of excitatory and inhibitory neurons (EI-networks) to a transient external input. Recently, it has been shown that this resonance property of EI-networks can be exploited for communication in modular neuronal networks by enabling the transmission of sequences of synchronous spike volleys (’pulse packets’), despite the sparse and weak connectivity between the modules. The condition for successful transmission is that the pulse packet (PP) intervals match the period of the modules’ resonance frequency. Hence, the mechanism was termed communication through resonance (CTR). This mechanism has three severe constraints, though. First, it needs periodic trains of PPs, whereas single PPs fail to propagate. Second, the inter-PP interval needs to match the network resonance. Third, transmission is very slow, because in each module, the network resonance needs to build up over multiple oscillation cycles. Here, we show that, by adding appropriate feedback connections to the network, the CTR mechanism can be improved and the aforementioned constraints relaxed. Specifically, we show that adding feedback connections between two upstream modules, called the resonance pair, in an otherwise feedforward modular network can support successful propagation of a single PP throughout the entire network. The key condition for successful transmission is that the sum of the forward and backward delays in the resonance pair matches the resonance frequency of the network modules. The transmission is much faster, by more than a factor of two, than in the original CTR mechanism. Moreover, it distinctly lowers the threshold for successful communication by synchronous spiking in modular networks of weakly coupled networks. Thus, our results suggest a new functional role of bidirectional connectivity for the communication in cortical area networks.

The cortex is organized as a modular system, with the modules (cortical areas) communicating via weak long-range connections. It has been suggested that the intrinsic resonance properties of population activities in these areas might contribute to enabling successful communication. A module’s intrinsic resonance appears in the damped oscillatory response to an incoming spike volley, enabling successful communication during the peaks of the oscillation. Such communication can be exploited in feedforward networks, provided the participating networks have similar resonance frequencies. This, however, is not necessarily true for cortical networks. Moreover, the communication is slow, as it takes several oscillation cycles to build up the response in the downstream network. Also, only periodic trains of spikes volleys (and not single volleys) with matching intervals can propagate. Here, we present a novel mechanism that alleviates these shortcomings and enables propagation of synchronous spiking across weakly connected networks with not necessarily identical resonance frequencies. In this framework, an individual spike volley can propagate by local amplification through reverberation in a loop between two successive networks, connected by feedforward and feedback connections: the resonance pair. This overcomes the need for activity build-up in downstream networks, causing the volley to propagate distinctly faster and more reliably.

Spontaneous Feedforward Connectivity in Electrically Stimulated Retinal Degeneration Mice. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020;2020:3513-3516. [PMID: 33018761 DOI: 10.1109/embc44109.2020.9175231]

Retinal degeneration (Rd) is a neurodegenerative disorder primarily associated with the degeneration of the retina neurons and culminates in the eventual loss of visual perception or blindness. Decrease in fronto-, parietal and occipital brain connectivity have been reported in a number of neurodegeneration diseases involving cognitive decline. However, cortical communication in the brain of retinal degeneration patients remains largely unknown and strategies to remediate observed dysfunctional brain connectivity in such instance have not be thoroughly investigated. We used rd10 mice as a model to study brain connectivity in the human retinal degeneration disease, retinitis pigmentosa. Rd10 mice with sham matched controls were electrically stimulated at varying stimulation frequencies and the consequent perturbations in feedforward brain connectivity were studied in the visual cortex and pre-frontal cortex using electrocorticography (ECoG) and normalized symbolic transfer entropy (NSTE). Contra Vcx - contra PFx feed forward connectivity significantly (p<0.05) increased in theta, alpha and beta oscillatory bands of 2 Hz and 10 Hz stimulated rd10 respectively in comparison with sham group. Also, this increase was significantly maintained even after the end of the stimulation period.

Data-Driven Approaches to Understanding Visual Neuron Activity

With modern neurophysiological methods able to record neural activity throughout the visual pathway in the context of arbitrarily complex visual stimulation, our understanding of visual system function is becoming limited by the available models of visual neurons that can be directly related to such data. Different forms of statistical models are now being used to probe the cellular and circuit mechanisms shaping neural activity, understand how neural selectivity to complex visual features is computed, and derive the ways in which neurons contribute to systems-level visual processing. However, models that are able to more accurately reproduce observed neural activity often defy simple interpretations. As a result, rather than being used solely to connect with existing theories of visual processing, statistical modeling will increasingly drive the evolution of more sophisticated theories.

Beyond core object recognition: Recurrent processes account for object recognition under occlusion

Core object recognition, the ability to rapidly recognize objects despite variations in their appearance, is largely solved through the feedforward processing of visual information. Deep neural networks are shown to achieve human-level performance in these tasks, and explain the primate brain representation. On the other hand, object recognition under more challenging conditions (i.e. beyond the core recognition problem) is less characterized. One such example is object recognition under occlusion. It is unclear to what extent feedforward and recurrent processes contribute in object recognition under occlusion. Furthermore, we do not know whether the conventional deep neural networks, such as AlexNet, which were shown to be successful in solving core object recognition, can perform similarly well in problems that go beyond the core recognition. Here, we characterize neural dynamics of object recognition under occlusion, using magnetoencephalography (MEG), while participants were presented with images of objects with various levels of occlusion. We provide evidence from multivariate analysis of MEG data, behavioral data, and computational modelling, demonstrating an essential role for recurrent processes in object recognition under occlusion. Furthermore, the computational model with local recurrent connections, used here, suggests a mechanistic explanation of how the human brain might be solving this problem.

In recent years, deep-learning-based computer vision algorithms have been able to achieve human-level performance in several object recognition tasks. This has also contributed in our understanding of how our brain may be solving these recognition tasks. However, object recognition under more challenging conditions, such as occlusion, is less characterized. Temporal dynamics of object recognition under occlusion is largely unknown in the human brain. Furthermore, we do not know if the previously successful deep-learning algorithms can similarly achieve human-level performance in these more challenging object recognition tasks. By linking brain data with behavior, and computational modeling, we characterized temporal dynamics of object recognition under occlusion, and proposed a computational mechanism that explains both behavioral and the neural data in humans. This provides a plausible mechanistic explanation for how our brain might be solving object recognition under more challenging conditions.

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Convolutional neural network (CNN) driven by image recognition has been shown to be able to explain cortical responses to static pictures at ventral-stream areas. Here, we further showed that such CNN could reliably predict and decode functional magnetic resonance imaging data from humans watching natural movies, despite its lack of any mechanism to account for temporal dynamics or feedback processing. Using separate data, encoding and decoding models were developed and evaluated for describing the bi-directional relationships between the CNN and the brain. Through the encoding models, the CNN-predicted areas covered not only the ventral stream, but also the dorsal stream, albeit to a lesser degree; single-voxel response was visualized as the specific pixel pattern that drove the response, revealing the distinct representation of individual cortical location; cortical activation was synthesized from natural images with high-throughput to map category representation, contrast, and selectivity. Through the decoding models, fMRI signals were directly decoded to estimate the feature representations in both visual and semantic spaces, for direct visual reconstruction and semantic categorization, respectively. These results corroborate, generalize, and extend previous findings, and highlight the value of using deep learning, as an all-in-one model of the visual cortex, to understand and decode natural vision.

Hierarchical decision-making produces persistent differences in learning performance

Human organizations are commonly characterized by a hierarchical chain of command that facilitates division of labor and integration of effort. Higher-level employees set the strategic frame that constrains lower-level employees who carry out the detailed operations serving to implement the strategy. Typically, strategy and operational decisions are carried out by different individuals that act over different timescales and rely on different kinds of information. We hypothesize that when such decision processes are hierarchically distributed among different individuals, they produce highly heterogeneous and strongly path-dependent joint learning dynamics. To investigate this, we design laboratory experiments of human dyads facing repeated joint tasks, in which one individual is assigned the role of carrying out strategy decisions and the other operational ones. The experimental behavior generates a puzzling bimodal performance distribution–some pairs learn, some fail to learn after a few periods. We also develop a computational model that mirrors the experimental settings and predicts the heterogeneity of performance by human dyads. Comparison of experimental and simulation data suggests that self-reinforcing dynamics arising from initial choices are sufficient to explain the performance heterogeneity observed experimentally.

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision

The human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas, especially those along the dorsal stream. As a fully observable model of visual processing, the RNN also revealed a cortical hierarchy of temporal receptive window, dynamics of process memory, and spatiotemporal representations. These results support the hypothesis of process memory, and demonstrate the potential of using the RNN for in-depth computational understanding of dynamic natural vision.

The consequence of spatial visual processing dysfunction caused by traumatic brain injury (TBI)

OBJECTIVE

A bi-modal visual processing model is supported by research to affect dysfunction following a traumatic brain injury (TBI). TBI causes dysfunction of visual processing affecting binocularity, spatial orientation, posture and balance. Research demonstrates that prescription of prisms influence the plasticity between spatial visual processing and motor-sensory systems improving visual processing and reducing symptoms following a TBI.

RATIONALE

The rationale demonstrates that visual processing underlies the functional aspects of binocularity, balance and posture. The bi-modal visual process maintains plasticity for efficiency. Compromise causes Post Trauma Vision Syndrome (PTVS) and Visual Midline Shift Syndrome (VMSS). Rehabilitation through use of lenses, prisms and sectoral occlusion has inter-professional implications in rehabilitation affecting the plasticity of the bi-modal visual process, thereby improving binocularity, spatial orientation, posture and balance Main outcomes: This review provides an opportunity to create a new perspective of the consequences of TBI on visual processing and the symptoms that are often caused by trauma. It also serves to provide a perspective of visual processing dysfunction that has potential for developing new approaches of rehabilitation.

CONCLUSIONS

Understanding vision as a bi-modal process facilitates a new perspective of visual processing and the potentials for rehabilitation following a concussion, brain injury or other neurological events.

Biologically Inspired Model for Inference of 3D Shape from Texture

A biologically inspired model architecture for inferring 3D shape from texture is proposed. The model is hierarchically organized into modules roughly corresponding to visual cortical areas in the ventral stream. Initial orientation selective filtering decomposes the input into low-level orientation and spatial frequency representations. Grouping of spatially anisotropic orientation responses builds sketch-like representations of surface shape. Gradients in orientation fields and subsequent integration infers local surface geometry and globally consistent 3D depth. From the distributions in orientation responses summed in frequency, an estimate of the tilt and slant of the local surface can be obtained. The model suggests how 3D shape can be inferred from texture patterns and their image appearance in a hierarchically organized processing cascade along the cortical ventral stream. The proposed model integrates oriented texture gradient information that is encoded in distributed maps of orientation-frequency representations. The texture energy gradient information is defined by changes in the grouped summed normalized orientation-frequency response activity extracted from the textured object image. This activity is integrated by directed fields to generate a 3D shape representation of a complex object with depth ordering proportional to the fields output, with higher activity denoting larger distance in relative depth away from the viewer.

Time for Awareness: The Influence of Temporal Properties of the Mask on Continuous Flash Suppression Effectiveness

Visual processing is not instantaneous, but instead our conscious perception depends on the integration of sensory input over time. In the case of Continuous Flash Suppression (CFS), masks are flashed to one eye, suppressing awareness of stimuli presented to the other eye. One potential explanation of CFS is that it depends, at least in part, on the flashing mask continually interrupting visual processing before the stimulus reaches awareness. We investigated the temporal features of masks in two ways. First, we measured the suppression effectiveness of a wide range of masking frequencies (0-32Hz), using both complex (faces/houses) and simple (closed/open geometric shapes) stimuli. Second, we varied whether the different frequencies were interleaved within blocks or separated in homogenous blocks, in order to see if suppression was stronger or weaker when the frequency remained constant across trials. We found that break-through contrast differed dramatically between masking frequencies, with mask effectiveness following a skewed-normal curve peaking around 6Hz and little or no masking for low and high temporal frequencies. Peak frequency was similar for trial-randomized and block randomized conditions. In terms of type of stimulus, we found no significant difference in peak frequency between the stimulus groups (complex/simple, face/house, closed/open). These findings suggest that temporal factors play a critical role in perceptual awareness, perhaps due to interactions between mask frequency and the time frame of visual processing.

A closer look at four-dot masking of a foveated target

Four-dot masking with a common onset mask was recently demonstrated in a fully attended and foveated target (Filmer, Mattingley & Dux, 2015). Here, we replicate and extend this finding by directly comparing a four-dot mask with an annulus mask while probing masking as a function of mask duration, and target-mask separation. Our results suggest that while an annulus mask operates via spatially local contour interactions, a four-dot mask operates through spatially global mechanisms. We also measure how the visual system’s representation of an oriented bar is impacted by a four-dot mask, and find that masking here does not degrade the precision of perceived targets, but instead appears to be driven exclusively by rendering the target completely invisible.