Visual parsing after recovery from blindness

Contrastive learning explains the emergence and function of visual category-selective regions

Modular and distributed coding theories of category selectivity along the human ventral visual stream have long existed in tension. Here, we present a reconciling framework-contrastive coding-based on a series of analyses relating category selectivity within biological and artificial neural networks. We discover that, in models trained with contrastive self-supervised objectives over a rich natural image diet, category-selective tuning naturally emerges for faces, bodies, scenes, and words. Further, lesions of these model units lead to selective, dissociable recognition deficits, highlighting their distinct functional roles in information processing. Finally, these pre-identified units can predict neural responses in all corresponding face-, scene-, body-, and word-selective regions of human visual cortex, under a highly constrained sparse positive encoding procedure. The success of this single model indicates that brain-like functional specialization can emerge without category-specific learning pressures, as the system learns to untangle rich image content. Contrastive coding, therefore, provides a unifying account of object category emergence and representation in the human brain.

Motion's privilege in recognizing facial expressions following treatment for blindness

In his 1872 monograph, Charles Darwin posited that "… the habit of expressing our feelings by certain movements, though now rendered innate, had been in some manner gradually acquired."1 Nearly 150 years later, researchers are still teasing apart innate versus experience-dependent contributions to expression recognition. Indeed, studies have shown that face detection is surprisingly resilient to early visual deprivation,2,3,4,5 pointing to plasticity that extends beyond dogmatic critical periods.6,7,8 However, it remains unclear whether such resilience extends to downstream processing, such as the ability to recognize facial expressions. The extent to which innate versus experience-dependent mechanisms contribute to this ability has yet to be fully explored.9,10,11,12,13 To investigate the impact of early visual experience on facial-expression recognition, we studied children with congenital cataracts who have undergone sight-correcting treatment14,15 and tracked their longitudinal skill acquisition as they gain sight late in life. We introduce and explore two potential facilitators of late-life plasticity: the availability of newborn-like coarse visual acuity prior to treatment16 and the privileged role of motion following treatment.4,17,18 We find that early visual deprivation does not preclude partial acquisition of facial-expression recognition. While rudimentary pretreatment vision is sufficient to allow a low level of expression recognition, it does not facilitate post-treatment improvements. Additionally, only children commencing vision with high visual acuity privilege the use of dynamic cues. We conclude that skipping typical visual experience early in development and introducing high-resolution imagery late in development restricts, but does not preclude, facial-expression skill acquisition and that the representational mechanisms driving this learning differ from those that emerge during typical visual development.

Do you hear what I see? How do early blind individuals experience object motion?

One of the most important tasks for 3D vision is tracking the movement of objects in space. The ability of early blind individuals to understand motion in the environment from noisy and unreliable auditory information is an impressive example of cortical adaptation that is only just beginning to be understood. Here, we compare visual and auditory motion processing, and discuss the effect of early blindness on the perception of auditory motion. Blindness leads to cross-modal recruitment of the visual motion area hMT+ for auditory motion processing. Meanwhile, the planum temporale, associated with auditory motion in sighted individuals, shows reduced selectivity for auditory motion. We discuss how this dramatic shift in the cortical basis of motion processing might influence the perceptual experience of motion in early blind individuals. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Stereopsis following surgery in children with congenital and developmental cataracts: A systematic review and meta-analysis

We estimated the proportion of children with stereopsis following surgery in congenital and developmental cataracts by systematic review and meta-analysis and also considered the factors influencing stereopsis, such as intervention age and presence of strabismus. Stereopsis is directly related to quality of life, and investigating its levels following cataract surgery in children may help decide the right time to intervene, particularly in the context of brain plasticity. We conducted a systematic literature search using Scopus, PubMed, and Web of Science and found 25 case series, 3 cohorts, and 3 clinical trial studies from 1/1/1995 to 31/12/2020. Study-specific proportions of stereopsis from 923 children were pooled using a random-effects model, and stratified analyses were conducted based on intervention age and pre-existing strabismus as a confounder. We appraised the risk of bias using tools published by National Institutes of Health and evaluated publication bias with funnel plots and the Egger test. The pooled proportions of stereopsis based on 8 unilateral and 6 bilateral congenital cataract studies were 0.37 (95% CIs: [0.24, 0.53]) and 0.45 (95% CIs: [0.24,0.68]) when patients with preexisting strabismus were excluded as a confounder. When the intervention age was ≤6 months, proportions in unilateral congenital cataract group significantly increased to 0.52 (95% CIs: [0.37, 0.66]; P = 0.49) compared to 0.26 (95% CIs: [0.14, 0.44]; P = 0.16) otherwise. A similar increase in proportions was found when intervention age ≤4 months. In both unilateral and bilateral congenital cataract groups, proportions increased significantly when the confounder was excluded. Overall, proportions in bilateral congenital cataracts were significantly greater than unilateral cases (irrespective of confounder). Eight unilateral and 5 bilateral developmental cataract studies resulted in pooled proportions of 0.62 (95% CIs: [0.27, 0.88] and 0.82 (95% CIs: [0.4, 0.97]), respectively. Although proportions for bilateral developmental cataracts were greater than unilateral cataracts (irrespective of confounder), results were not statistically significant. Finally, proportions in unilateral developmental cataracts were significantly greater than unilateral congenital cataracts (Z = 7.413, P = 6.173694e-14). We conclude that surgical intervention within first 4-6 months can significantly affect postoperative outcomes in unilateral congenital cataracts. Analysis of existing data does not show a significant effect of intervention age on stereopsis outcomes for developmental cataracts.

Calibrating Vision: Concepts and Questions

The idea that visual coding and perception are shaped by experience and adjust to changes in the environment or the observer is universally recognized as a cornerstone of visual processing, yet the functions and processes mediating these calibrations remain in many ways poorly understood. In this article we review a number of facets and issues surrounding the general notion of calibration, with a focus on plasticity within the encoding and representational stages of visual processing. These include how many types of calibrations there are - and how we decide; how plasticity for encoding is intertwined with other principles of sensory coding; how it is instantiated at the level of the dynamic networks mediating vision; how it varies with development or between individuals; and the factors that may limit the form or degree of the adjustments. Our goal is to give a small glimpse of an enormous and fundamental dimension of vision, and to point to some of the unresolved questions in our understanding of how and why ongoing calibrations are a pervasive and essential element of vision.

Learning to see after early and extended blindness: A scoping review

Purpose

If an individual has been blind since birth due to a treatable eye condition, ocular treatment is urgent. Even a brief period of visual deprivation can alter the development of the visual system. The goal of our structured scoping review was to understand how we might better support children with delayed access to ocular treatment for blinding conditions.

Method

We searched MEDLINE, Embase and Global Health for peer-reviewed publications that described the impact of early (within the first year) and extended (lasting at least 2 years) bilateral visual deprivation.

Results

Of 551 reports independently screened by two authors, 42 studies met our inclusion criteria. Synthesizing extracted data revealed several trends. The data suggests persistent deficits in visual acuity, contrast sensitivity, global motion, and visual-motor integration, and suspected concerns for understanding complex objects and faces. There is evidence for resilience in color perception, understanding of simple shapes, discriminating between a face and non-face, and the perception of biological motion. There is currently insufficient data about specific (re)habilitation strategies to update low vision services, but there are several insights to guide future research in this domain.

Conclusion

This summary will help guide the research and services provision to help children learn to see after early and extended blindness.

Active vision in sight recovery individuals with a history of long-lasting congenital blindness

What we see is intimately linked to how we actively and systematically explore the world through eye movements. However, it is unknown to what degree visual experience during early development is necessary for such systematic visual exploration to emerge. The present study investigated visual exploration behavior in ten human participants whose sight had been restored only in childhood or adulthood, after a period of congenital blindness due to dense bilateral congenital cataracts. Participants freely explored real-world images while their eye movements were recorded. Despite severe residual visual impairments and gaze instability (nystagmus), visual exploration patterns were preserved in individuals with reversed congenital cataract. Modelling analyses indicated that similar to healthy controls, visual exploration in individuals with reversed congenital cataract was based on the low-level (luminance contrast) and high-level (object components) visual content of the images. Moreover, participants used visual short-term memory representations for narrowing down the exploration space. More systematic visual exploration in individuals with reversed congenital cataract was associated with better object recognition, suggesting that active vision might be a driving force for visual system development and recovery. The present results argue against a sensitive period for the development of neural mechanisms associated with visual exploration.SIGNIFICANCE STATEMENTHumans explore the visual world with systematic patterns of eye movements, but it is unknown whether early visual experience is necessary for the acquisition of visual exploration. Here, we show that sight recovery individuals who had been born blind demonstrate highly systematic eye movements while exploring real-world images, despite visual impairments and pervasive gaze instability. In fact, their eye movement patterns were predicted by those of normally sighted controls and models calculating eye movements based on low- and high-level visual features, and they moreover took memory information into account. Since object recognition performance was associated with systematic visual exploration it was concluded that eye movements might be a driving factor for the development of the visual system.

Head turning is an effective cue for gaze following: Evidence from newly sighted individuals, school children and adults

In referential communication, gaze is often interpreted as a social cue that facilitates comprehension and enables word learning. Here we investigated the degree to which head turning facilitates gaze following. We presented participants with static pictures of a man looking at a target object in a first and third block of trials (pre- and post-intervention), while they saw short videos of the same man turning towards the target in the second block of trials (intervention). In Experiment 1, newly sighted individuals (treated for congenital cataracts; N = 8) benefited from the motion cues, both when comparing their initial performance with static gaze cues to their performance with dynamic head turning, and their performance with static cues before and after the videos. In Experiment 2, neurotypical school children (ages 5-10 years; N = 90) and adults (N = 30) also revealed improved performance with motion cues, although most participants had started to follow the static gaze cues before they saw the videos. Our results confirm that head turning is an effective social cue when interpreting new words, offering new insights for a pathways approach to development.

Human (but not animal) motion can be recognized at first sight - After treatment for congenital blindness

The long-standing nativist vs. empiricist debate asks a foundational question in epistemology - does our knowledge arise through experience or is it available innately? Studies that probe the sensitivity of newborns and patients recovering from congenital blindness are central in informing this dialogue. One of the most robust sensitivities our visual system possesses is to 'biological motion' - the movement patterns of humans and other vertebrates. Various biological motion perception skills (such as distinguishing between movement of human and non-human animals, or between upright and inverted human movement) become evident within the first months of life. The mechanisms of acquiring these capabilities, and specifically the contribution of visual experience to their development, are still under debate. We had the opportunity to directly examine the role of visual experience in biological motion perception, by testing what level of sensitivity is present immediately upon onset of sight following years of congenital visual deprivation. Two congenitally blind patients who underwent sight-restorative cataract-removal surgery late in life (at the ages of 7 and 20 years) were tested before and after sight restoration. The patients were shown displays of walking humans, pigeons, and cats, and asked to describe what they saw. Visual recognition of movement patterns emerged immediately upon eye-opening following surgery, when the patients spontaneously began to identify human, but not animal, biological motion. This recognition ability was evident contemporaneously for upright and inverted human displays. These findings suggest that visual recognition of human motion patterns may not critically depend on visual experience, as it was evident upon first exposure to un-obstructed sight in patients with very limited prior visual exposure, and furthermore, was not limited to the typical (upright) orientation of humans in real-life settings.

A case study in phenomenology of visual experience with retinal prosthesis versus visual-to-auditory sensory substitution

The phenomenology of the blind has provided an age-old, unparalleled means of exploring the enigmatic link between the brain and mind. This paper delves into the unique phenomenological experience of a man who became blind in adulthood. He subsequently underwent both an Argus II retinal prosthesis implant and training, and extensive training on the EyeMusic visual to auditory sensory substitution device (SSD), thereby becoming the first reported case to date of dual proficiency with both devices. He offers a firsthand account into what he considers the great potential of combining sensory substitution devices with visual prostheses as part of a complete visual restoration protocol. While the Argus II retinal prosthesis alone provided him with immediate visual percepts by way of electrically stimulated phosphenes elicited by the device, the EyeMusic SSD requires extensive training from the onset. Yet following the extensive training program with the EyeMusic sensory substitution device, our subject reports that the sensory substitution device allowed him to experience a richer, more complex perceptual experience, that felt more "second nature" to him, while the Argus II prosthesis (which also requires training) did not allow him to achieve the same levels of automaticity and transparency. Following long-term use of the EyeMusic SSD, our subject reported that visual percepts representing mainly, but not limited to, colors portrayed by the EyeMusic SSD are elicited in association with auditory stimuli, indicating the acquisition of a high level of automaticity. Finally, the case study indicates an additive benefit to the combination of both devices on the user's subjective phenomenological visual experience.

Perception of an object's global shape is best described by a model of skeletal structure in human infants

Categorization of everyday objects requires that humans form representations of shape that are tolerant to variations among exemplars. Yet, how such invariant shape representations develop remains poorly understood. By comparing human infants (6-12 months; N=82) to computational models of vision using comparable procedures, we shed light on the origins and mechanisms underlying object perception. Following habituation to a never-before-seen object, infants classified other novel objects across variations in their component parts. Comparisons to several computational models of vision, including models of high-level and low-level vision, revealed that infants' performance was best described by a model of shape based on the skeletal structure. Interestingly, infants outperformed a range of artificial neural network models, selected for their massive object experience and biological plausibility, under the same conditions. Altogether, these findings suggest that robust representations of shape can be formed with little language or object experience by relying on the perceptually invariant skeletal structure.

Development of Visual Memory Capacity Following Early-Onset and Extended Blindness

It is unknown whether visual memory capacity can develop if onset of pattern vision is delayed for several years following birth. We had an opportunity to address this question through our work with an unusual population of 12 congenitally blind individuals ranging in age from 8 to 22 years. After providing them with sight surgery, we longitudinally evaluated their visual memory capacity using an image-memorization task. Our findings revealed poor visual memory capacity soon after surgery but significant improvement in subsequent months. Although there may be limits to this improvement, performance 1 year after surgery was found to be comparable with that of control participants with matched visual acuity. These findings provide evidence for plasticity of visual memory mechanisms into late childhood but do not rule out vulnerability to early deprivation. Our computational simulations suggest that a potential mechanism to account for changes in memory performance may be progressive representational elaboration in image encoding.

Critical Periods in Vision Revisited

For four decades, investigations of the biological basis of critical periods in the developing mammalian visual cortex were dominated by study of the consequences of altered early visual experience in cats and nonhuman primates. The neural deficits thus revealed also provided insight into the origin and neural basis of human amblyopia that in turn motivated additional studies of humans with abnormal early visual input. Recent human studies point to deficits arising from alterations in all visual cortical areas and even in nonvisual cortical regions. As the new human data accumulated in parallel with a near-complete shift toward the use of rodent animal models for the study of neural mechanisms, it is now essential to review the human data and the earlier animal data obtained from cats and monkeys to infer general conclusions and to optimize future choice of the most appropriate animal model. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Binocular vision supports the development of scene segmentation capabilities: Evidence from a deep learning model

The application of deep learning techniques has led to substantial progress in solving a number of critical problems in machine vision, including fundamental problems of scene segmentation and depth estimation. Here, we report a novel deep neural network model, capable of simultaneous scene segmentation and depth estimation from a pair of binocular images. By manipulating the arrangement of binocular image pairs, presenting the model with standard left-right image pairs, identical image pairs or swapped left-right images, we show that performance levels depend on the presence of appropriate binocular image arrangements. Segmentation and depth estimation performance are both impaired when images are swapped. Segmentation performance levels are maintained, however, for identical image pairs, despite the absence of binocular disparity information. Critically, these performance levels exceed those found for an equivalent, monocularly trained, segmentation model. These results provide evidence that binocular image differences support both the direct recovery of depth and segmentation information, and the enhanced learning of monocular segmentation signals. This finding suggests that binocular vision may play an important role in visual development. Better understanding of this role may hold implications for the study and treatment of developmentally acquired perceptual impairments.

Resilience of temporal processing to early and extended visual deprivation

Early visual deprivation is known to have profound consequences on the subsequent development of spatial visual processing. However, its impact on temporal processing is not well characterized. We have examined spatial and temporal contrast sensitivity functions following treatment for early and extended bilateral visual deprivation in fifteen children born with congenital cataracts in rural India. The results reveal a marked difference in post-treatment spatial and temporal sensitivities. Whereas spatial processing in newly sighted children is significantly impaired relative to age-matched controls, temporal processing exhibits remarkable resilience and is comparable to that in the control group. This difference in spatial and temporal outcomes is especially surprising given our computational analyses of video sequences which indicate a strong linkage between the spatial and temporal spectral content of natural visual inputs. We consider possible explanations for this discrepancy.

Learning to perceive shape from temporal integration following late emergence from blindness

Visual perception requires massive use of inference because the 3D structure of the world is not directly provided by the sensory input.¹ Particularly challenging is anorthoscopic vision-when an object moves behind a narrow slit such that only a tiny fraction of it is visible at any instant. Impressively, human observers correctly recognize objects in slit-viewing conditions by early childhood,^2,3 via temporal integration of the contours available in each sliver.^4,5 But can this capability be acquired if one has been effectively blind throughout childhood? We studied 23 Ethiopian children which had bilateral early-onset cataracts-resulting in extremely poor vision in infancy-and surgically treated only years later. We tested their anorthoscopic vision, precisely because it requires a cascade of demanding visual inference processes to perceive veridical shape. Failure to perform the task may allow mapping specific bottlenecks for late visual recovery. The patients' visual acuity typically improved substantially within 6 months post-surgery. Still, at this stage many were unable to recover shape under slit-viewing conditions, although they could infer the direction of global motion. However, when retested later, almost all patients could judge shape in slit-conditions necessitating temporal integration. This acquired capability often transferred to novel stimuli, in similar slit-viewing conditions. Thus, learning was not limited to the specific visual features of the original shapes. These results indicate that plasticity of sophisticated visual inference routines is preserved well into adolescence, and vision restoration after prolonged early-onset blindness is feasible to a greater extent than previously thought.

Numerosity representation in a deep convolutional neural network

Enumerating objects in the environment (i.e., “number sense”) is crucial for survival in many animal species, and foundational for the construction of more abstract and complex mathematical knowledge in humans. Perhaps surprisingly, deep convolutional neural networks (DCNNs) spontaneously emerge a similar number sense even without any explicit training for numerosity estimation. However, little is known about how the number sense emerges, and the extent to which it is comparable with human number sense. Here, we examined whether the numerosity underestimation effect, a phenomenon indicating that numerosity perception acts upon the perceptual number rather than the physical number, can be observed in DCNNs. In a typical DCNN, AlexNet, we found that number-selective units at late layers operated on the perceptual number, like humans do. More importantly, this perceptual number sense did not emerge abruptly, rather developed progressively along the hierarchy in the DCNN, shifting from the physical number sense at early layers to perceptual number sense at late layers. Our finding hence provides important implications for the neural implementation of number sense in the human brain and advocates future research to determine whether the representation of numerosity also develops gradually along the human visual stream from physical number to perceptual number.

Brain network mechanisms of visual shape completion

Visual shape completion recovers object shape, size, and number from spatially segregated edges. Despite being extensively investigated, the process’s underlying brain regions, networks, and functional connections are still not well understood. To shed light on the topic, we scanned (fMRI) healthy adults during rest and during a task in which they discriminated pac-man configurations that formed or failed to form completed shapes (illusory and fragmented condition, respectively). Task activation differences (illusory-fragmented), resting-state functional connectivity, and multivariate patterns were identified on the cortical surface using 360 predefined parcels and 12 functional networks composed of such parcels. Brain activity flow mapping (ActFlow) was used to evaluate the likely involvement of resting-state connections for shape completion. We identified 36 differentially-active parcels including a posterior temporal region, PH, whose activity was consistent across 95% of observers. Significant task regions primarily occupied the secondary visual network but also incorporated the frontoparietal dorsal attention, default mode, and cingulo-opercular networks. Each parcel’s task activation difference could be modeled via its resting-state connections with the remaining parcels (r=.62, p<10⁻⁹), suggesting that such connections undergird shape completion. Functional connections from the dorsal attention network were key in modelling task activation differences in the secondary visual network. Dorsal attention and frontoparietal connections could also model activations in the remaining networks. Taken together, these results suggest that shape completion relies upon a sparsely distributed but densely interconnected network coalition that is centered in the secondary visual network, coordinated by the dorsal attention network, and inclusive of at least three other networks.

MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation

In this paper, we present a novel end-to-end learning neural network, i.e., MATNet, for zero-shot video object segmentation (ZVOS). Motivated by the human visual attention behavior, MATNet leverages motion cues as a bottom-up signal to guide the perception of object appearance. To achieve this, an asymmetric attention block, named Motion-Attentive Transition (MAT), is proposed within a two-stream encoder network to firstly identify moving regions and then attend appearance learning to capture the full extent of objects. Putting MATs in different convolutional layers, our encoder becomes deeply interleaved, allowing for close hierarchical interactions between object apperance and motion. Such a biologically-inspired design is proven to be superb to conventional two-stream structures, which treat motion and appearance independently in separate streams and often suffer severe overfitting to object appearance. Moreover, we introduce a bridge network to modulate multi-scale spatiotemporal features into more compact, discriminative and scale-sensitive representations, which are subsequently fed into a boundary-aware decoder network to produce accurate segmentation with crisp boundaries. We perform extensive quantitative and qualitative experiments on four challenging public benchmarks, i.e., DAVIS16, DAVIS17, FBMS and YouTube-Objects. Results show that our method achieves compelling performance against current state-of-the-art ZVOS methods. To further demonstrate the generalization ability of our spatiotemporal learning framework, we extend MATNet to another relevant task: dynamic visual attention prediction (DVAP). The experiments on two popular datasets (i.e., Hollywood-2 and UCF-Sports) further verify the superiority of our model. Our implementations have been made publicly available at https://github.com/tfzhou/MATNet.

Mechanisms underlying simultaneous brightness contrast: Early and innate

In the phenomenon of simultaneous brightness contrast, two patches, one on a dark background and the other on a light one, appear to have different brightness despite being physically equi-luminant. Elucidating the phenomenon's underlying mechanisms is relevant for the larger question of how the visual system makes photometric judgments in images. Accounts over the past century have spanned low-, mid- and high-level visual processes, but a definitive resolution has not emerged. We present three studies that collectively demonstrate that the computations underlying this phenomenon are low-level, instantiated prior to binocular fusion, and available innately, without need for inferential learning via an individual's visual experience. In our first two studies, we find that strong brightness induction is obtained even when observers are unaware of any luminance differences in the neighborhoods of the probe patches. Results with dichoptic displays reveal that eye of origin, although not evident consciously, has a marked influence on the eventual brightness percept of the probe patches, thereby localizing brightness estimation to a site preceding binocular fusion. The third study uses conventional simultaneous brightness contrast displays, but an unusual group of participants: Congenitally blind children whom we were able to treat surgically. The results demonstrate an immediate susceptibility to the simultaneous brightness illusion after sight onset. Together, these data strongly constrain the search for mechanisms underlying a fundamental brightness phenomenon.

Deep Learning: The Good, the Bad, and the Ugly

Artificial vision has often been described as one of the key remaining challenges to be solved before machines can act intelligently. Recent developments in a branch of machine learning known as deep learning have catalyzed impressive gains in machine vision—giving a sense that the problem of vision is getting closer to being solved. The goal of this review is to provide a comprehensive overview of recent deep learning developments and to critically assess actual progress toward achieving human-level visual intelligence. I discuss the implications of the successes and limitations of modern machine vision algorithms for biological vision and the prospect for neuroscience to inform the design of future artificial vision systems.

Alterations in cortical and thalamic connections of somatosensory cortex following early loss of vision

Early loss of vision produces dramatic changes in the functional organization and connectivity of the neocortex in cortical areas that normally process visual inputs, such as the primary and second visual area. This loss also results in alterations in the size, functional organization, and neural response properties of the primary somatosensory area, S1. However, the anatomical substrate for these functional changes in S1 has never been described. In the present investigation, we quantified the cortical and subcortical connections of S1 in animals that were bilaterally enucleated very early in development, prior to the formation of retino-geniculate and thalamocortical pathways. We found that S1 receives dense inputs from novel cortical fields, and that the density of existing cortical and thalamocortical connections was altered. Our results demonstrate that sensory systems develop in tandem and that alterations in sensory input in one system can affect the connections and organization of other sensory systems. Thus, therapeutic intervention following early loss of vision should focus not only on restoring vision, but also on augmenting the natural plasticity of the spared systems.

Potential downside of high initial visual acuity

Children who are treated for congenital cataracts later exhibit impairments in configural face analysis. This has been explained in terms of a critical period for the acquisition of normal face processing. Here, we consider a more parsimonious account according to which deficits in configural analysis result from the abnormally high initial retinal acuity that children treated for cataracts experience, relative to typical newborns. According to this proposal, the initial period of low retinal acuity characteristic of normal visual development induces extended spatial processing in the cortex that is important for configural face judgments. As a computational test of this hypothesis, we examined the effects of training with high-resolution or blurred images, and staged combinations, on the receptive fields and performance of a convolutional neural network. The results show that commencing training with blurred images creates receptive fields that integrate information across larger image areas and leads to improved performance and better generalization across a range of resolutions. These findings offer an explanation for the observed face recognition impairments after late treatment of congenital blindness, suggest an adaptive function for the acuity trajectory in normal development, and provide a scheme for improving the performance of computational face recognition systems.

Does Central Vision Loss Impair Visual Search Performance of Adults More than Children?

SIGNIFICANCE

In general, young adults with normal vision show the best visual search performance when compared with children and older adults. Through our study, we show that this trend is not observed in individuals with vision impairment. An interaction effect of vision impairment with visual development and aging is observed.

PURPOSE

Performance in many visual tasks typically shows improvement with age until young adulthood and then declines with aging. Using a visual search task, this study investigated whether a similar age effect on performance is present in people with central vision loss.

METHODS

A total of 98 participants, 37 with normal sight (NS) and 61 with visual impairment (VI) searched for targets in 150 real-world digital images. Search performance was quantified by an integrated measure combining speed and accuracy. Participant ages ranged from 5 to 74 years, visual acuity from -0.14 (20/14.5) to 1.16 logMAR (20/290), and log contrast sensitivity (CS) from 0.48 to 2.0. Data analysis was performed with participants divided into three age groups: children (aged <14 years, n = 25), young adults (aged 14 to 45 years, n = 47), and older adults (aged >45 years, n = 26).

RESULTS

Regression (r = 0.7) revealed CS (P < .001) and age (P = .003) were significant predictors of search performance. Performance of VI participants was normalized to the age-matched average performance of the NS group. In the VI group, it was found that children's normalized performance (52%) was better than both young (39%, P = .05) and older (40%, P = .048) adults.

CONCLUSIONS

Unlike NS participants, young adults in the VI group may not have search ability superior to children with VI, despite having the same level of visual functions (quantified by visual acuity and CS). This could be because of vision impairment limiting the developmental acquisition of the age dividend for peak performance. Older adults in the VI group had the worst performance, indicating an interaction of aging.

Contour interpolation: A case study in Modularity of Mind

In his monograph Modularity of Mind (1983), philosopher Jerry Fodor argued that mental architecture can be partly decomposed into computational organs termed modules, which were characterized as having nine co-occurring features such as automaticity, domain specificity, and informational encapsulation. Do modules exist? Debates thus far have been framed very generally with few, if any, detailed case studies. The topic is important because it has direct implications on current debates in cognitive science and because it potentially provides a viable framework from which to further understand and make hypotheses about the mind's structure and function. Here, the case is made for the modularity of contour interpolation, which is a perceptual process that represents non-visible edges on the basis of how surrounding visible edges are spatiotemporally configured. There is substantial evidence that interpolation is domain specific, mandatory, fast, and developmentally well-sequenced; that it produces representationally impoverished outputs; that it relies upon a relatively fixed neural architecture that can be selectively impaired; that it is encapsulated from belief and expectation; and that its inner workings cannot be fathomed through conscious introspection. Upon differentiating contour interpolation from a higher-order contour representational ability ("contour abstraction") and upon accommodating seemingly inconsistent experimental results, it is argued that interpolation is modular to the extent that the initiating conditions for interpolation are strong. As interpolated contours become more salient, the modularity features emerge. The empirical data, taken as a whole, show that at least certain parts of the mind are modularly organized.

Lack of Automatic Imitation in Newly Sighted Individuals

Viewing a hand action performed by another person facilitates a response-compatible action and slows a response-incompatible one, even when the viewed action is irrelevant to the task. This automatic imitation effect is taken as the clearest evidence for a direct mapping between action viewing and motor performance. But there is an ongoing debate whether this effect is innate or experience dependent. We tackled this issue by studying a unique group of newly sighted children who suffered from dense bilateral cataracts from early infancy and were surgically treated only years later. The newly sighted children were less affected by viewing task-irrelevant actions than were control children, even 2 years after the cataract-removal surgery. This strongly suggests that visually guided motor experience is necessary for the development of automatic imitation. At the very least, our results indicate that if imitation is based on innate mechanisms, these are clearly susceptible to long periods of visual deprivation.

Functional outcomes following lesions in visual cortex: Implications for plasticity of high-level vision

Understanding the nature and extent of neural plasticity in humans remains a key challenge for neuroscience. Importantly, however, a precise characterization of plasticity and its underlying mechanism has the potential to enable new approaches for enhancing reorganization of cortical function. Investigations of the impairment and subsequent recovery of cognitive and perceptual functions following early-onset cortical lesions in humans provide a unique opportunity to elucidate how the brain changes, adapts, and reorganizes. Specifically, here, we focus on restitution of visual function, and we review the findings on plasticity and re-organization of the ventral occipital temporal cortex (VOTC) in published reports of 46 patients with a lesion to or resection of the visual cortex early in life. Findings reveal that a lesion to the VOTC results in a deficit that affects the visual recognition of more than one category of stimuli (faces, objects and words). In addition, the majority of pediatric patients show limited recovery over time, especially those in whom deficits in low-level vision also persist. Last, given that neither the equipotentiality nor the modularity view on plasticity was clearly supported, we suggest some intermediate possibilities in which some plasticity may be evident but that this might depend on the area that was affected, its maturational trajectory as well as its structural and functional connectivity constraints. Finally, we offer suggestions for future research that can elucidate plasticity further.

Learning to see again: biological constraints on cortical plasticity and the implications for sight restoration technologies

The 'bionic eye'-so long a dream of the future-is finally becoming a reality with retinal prostheses available to patients in both the US and Europe. However, clinical experience with these implants has made it apparent that the visual information provided by these devices differs substantially from normal sight. Consequently, the ability of patients to learn to make use of this abnormal retinal input plays a critical role in whether or not some functional vision is successfully regained. The goal of the present review is to summarize the vast basic science literature on developmental and adult cortical plasticity with an emphasis on how this literature might relate to the field of prosthetic vision. We begin with describing the distortion and information loss likely to be experienced by visual prosthesis users. We then define cortical plasticity and perceptual learning, and describe what is known, and what is unknown, about visual plasticity across the hierarchy of brain regions involved in visual processing, and across different stages of life. We close by discussing what is known about brain plasticity in sight restoration patients and discuss biological mechanisms that might eventually be harnessed to improve visual learning in these patients.

Visual motion serves but is not under the purview of the dorsal pathway

Visual motion processing is often attributed to the dorsal visual pathway despite visual motion's involvement in almost all visual functions. Furthermore, some visual motion tasks critically depend on the structural integrity of regions outside the dorsal pathway. Here, based on numerous studies, I propose that visual motion signals are swiftly transmitted via multiple non-hierarchical routes to primary motion-dedicated processing regions (MT/V5 and MST) that are not part of the dorsal pathway, and then propagated to a multiplicity of brain areas according to task demands, reaching these regions earlier than the dorsal/ventral hierarchical flow. This not only places MT/V5 at the same or even earlier visual processing stage as that of V1, but can also elucidate many findings with implications to visual awareness. While the integrity of the non-hierarchical motion pathway is necessary for all visual motion perception, it is insufficient on its own, and the transfer of visual motion signals to additional brain areas is crucial to allow the different motion perception tasks (e.g. optic flow, visuo-vestibular balance, movement observation, dynamic form detection and perception, and even reading). I argue that this lateral visual motion pathway can be distinguished from the dorsal pathway not only based on faster response latencies and distinct anatomical connections, but also based on its full field representation. I also distinguish between this primary lateral visual motion pathway sensitive to all motion in the visual field, and a much less investigated optic flow sensitive medial processing pathway (from V1 to V6 and V6A) that appears to be part of the dorsal pathway. Multiple additional predictions are provided that allow testing this proposal and distinguishing between the visual pathways.

Sight restoration after congenital blindness does not reinstate alpha oscillatory activity in humans

Functional brain development is characterized by sensitive periods during which experience must be available to allow for the full development of neural circuits and associated behavior. Yet, only few neural markers of sensitive period plasticity in humans are known. Here we employed electroencephalographic recordings in a unique sample of twelve humans who had been blind from birth and regained sight through cataract surgery between four months and 16 years of age. Two additional control groups were tested: a group of visually impaired individuals without a history of total congenital blindness and a group of typically sighted individuals. The EEG was recorded while participants performed a visual discrimination task involving intact and scrambled biological motion stimuli. Posterior alpha and theta oscillations were evaluated. The three groups showed indistinguishable behavioral performance and in all groups evoked theta activity varied with biological motion processing. By contrast, alpha oscillatory activity was significantly reduced only in individuals with a history of congenital cataracts. These data document on the one hand brain mechanisms of functional recovery (related to theta oscillations) and on the other hand, for the first time, a sensitive period for the development of alpha oscillatory activity in humans.

'Visual' parsing can be taught quickly without visual experience during critical periods

Cases of invasive sight-restoration in congenital blind adults demonstrated that acquiring visual abilities is extremely challenging, presumably because visual-experience during critical-periods is crucial for learning visual-unique concepts (e.g. size constancy). Visual rehabilitation can also be achieved using sensory-substitution-devices (SSDs) which convey visual information non-invasively through sounds. We tested whether one critical concept – visual parsing, which is highly-impaired in sight-restored patients – can be learned using SSD. To this end, congenitally blind adults participated in a unique, relatively short (~70 hours), SSD-‘vision’ training. Following this, participants successfully parsed 2D and 3D visual objects. Control individuals naïve to SSDs demonstrated that while some aspects of parsing with SSD are intuitive, the blind’s success could not be attributed to auditory processing alone. Furthermore, we had a unique opportunity to compare the SSD-users’ abilities to those reported for sight-restored patients who performed similar tasks visually, and who had months of eyesight. Intriguingly, the SSD-users outperformed the patients on most criteria tested. These suggest that with adequate training and technologies, key high-order visual features can be quickly acquired in adulthood, and lack of visual-experience during critical-periods can be somewhat compensated for. Practically, these highlight the potential of SSDs as standalone-aids or combined with invasive restoration approaches.

Motion perception: a review of developmental changes and the role of early visual experience

Significant controversies have arisen over the developmental trajectory for the perception of global motion. Studies diverge on the age at which it becomes adult-like, with estimates ranging from as young as 3 years to as old as 16. In this article, we review these apparently conflicting results and suggest a potentially unifying hypothesis that may also account for the contradictory literature in neurodevelopmental disorders, such as Autism Spectrum Disorder (ASD). We also discuss the extent to which patterned visual input during this period is necessary for the later development of motion perception. We conclude by addressing recent studies directly comparing different types of motion integration, both in typical and atypical development, and suggest areas ripe for future research.

A lack of experience-dependent plasticity after more than a decade of recovered sight

In 2000, monocular vision was restored to M. M., who had been blind between the ages of 3 and 46 years. Tests carried out over 2 years following the surgery revealed impairments of 3-D form, object, and face processing and an absence of object- and face-selective blood-oxygen-level-dependent responses in ventral visual cortex. In the present research, we reexamined M. M. to test for experience-dependent recovery of visual function. Behaviorally, M. M. remains impaired in 3-D form, object, and face processing. Accordingly, we found little to no evidence of the category-selective organization within ventral visual cortex typically associated with face, body, scene, or object processing. We did observe remarkably normal object selectivity within lateral occipital cortex, consistent with M. M.'s previously reported shape-discrimination performance. Together, these findings provide little evidence for recovery of high-level visual function after more than a decade of visual experience in adulthood.

Anatomical and functional plasticity in early blind individuals and the mixture of experts architecture

As described elsewhere in this special issue, recent advances in neuroimaging over the last decade have led to a rapid expansion in our knowledge of anatomical and functional correlations within the normal and abnormal human brain. Here, we review how early blindness has been used as a model system for examining the role of visual experience in the development of anatomical connections and functional responses. We discuss how lack of power in group comparisons may provide a potential explanation for why extensive anatomical changes in cortico-cortical connectivity are not observed. Finally we suggest a framework-cortical specialization via hierarchical mixtures of experts-which offers some promise in reconciling a wide range of functional and anatomical data.

A conceptual framework of computations in mid-level vision

If a picture is worth a thousand words, as an English idiom goes, what should those words-or, rather, descriptors-capture? What format of image representation would be sufficiently rich if we were to reconstruct the essence of images from their descriptors? In this paper, we set out to develop a conceptual framework that would be: (i) biologically plausible in order to provide a better mechanistic understanding of our visual system; (ii) sufficiently robust to apply in practice on realistic images; and (iii) able to tap into underlying structure of our visual world. We bring forward three key ideas. First, we argue that surface-based representations are constructed based on feature inference from the input in the intermediate processing layers of the visual system. Such representations are computed in a largely pre-semantic (prior to categorization) and pre-attentive manner using multiple cues (orientation, color, polarity, variation in orientation, and so on), and explicitly retain configural relations between features. The constructed surfaces may be partially overlapping to compensate for occlusions and are ordered in depth (figure-ground organization). Second, we propose that such intermediate representations could be formed by a hierarchical computation of similarity between features in local image patches and pooling of highly-similar units, and reestimated via recurrent loops according to the task demands. Finally, we suggest to use datasets composed of realistically rendered artificial objects and surfaces in order to better understand a model's behavior and its limitations.

Global processing in amblyopia: a review

Amblyopia is a neurodevelopmental disorder of the visual system that is associated with disrupted binocular vision during early childhood. There is evidence that the effects of amblyopia extend beyond the primary visual cortex to regions of the dorsal and ventral extra-striate visual cortex involved in visual integration. Here, we review the current literature on global processing deficits in observers with either strabismic, anisometropic, or deprivation amblyopia. A range of global processing tasks have been used to investigate the extent of the cortical deficit in amblyopia including: global motion perception, global form perception, face perception, and biological motion. These tasks appear to be differentially affected by amblyopia. In general, observers with unilateral amblyopia appear to show deficits for local spatial processing and global tasks that require the segregation of signal from noise. In bilateral cases, the global processing deficits are exaggerated, and appear to extend to specialized perceptual systems such as those involved in face processing.

Segmentation of Moving Objects by Long Term Video Analysis

Motion is a strong cue for unsupervised object-level grouping. In this paper, we demonstrate that motion will be exploited most effectively, if it is regarded over larger time windows. Opposed to classical two-frame optical flow, point trajectories that span hundreds of frames are less susceptible to short-term variations that hinder separating different objects. As a positive side effect, the resulting groupings are temporally consistent over a whole video shot, a property that requires tedious post-processing in the vast majority of existing approaches. We suggest working with a paradigm that starts with semi-dense motion cues first and that fills up textureless areas afterwards based on color. This paper also contributes the Freiburg-Berkeley motion segmentation (FBMS) dataset, a large, heterogeneous benchmark with 59 sequences and pixel-accurate ground truth annotation of moving objects.

Miscoded Visual Processing in Degenerative Retinal Disorder?

Standard electrophysiological procedures for visual testing were applied to record the retinal and cortical electrophysiological responses to contrast stimulation from 35 subjects with unambiguously diagnosed retinitis pigmentosa and severe impairment of visual acuity and field. Stimuli (central 9° of visual field) were sinusoidal bars with spatial frequencies of 0.6–1.2 cycle/degree and 1.3–5.0 cycle/degree for the retinal (pattern-ERG) and cortical (pattern-VEP) responses, respectively; contrast was 80%; reversal at 2.13 Hz. Structured pattern-ERG above noise level was recorded from 29 subjects at 0.6 cycle/degree and from 24 subjects at 1.2 cycle/degree; latencies were increased and amplitude reduced. Pattern-VEP responses above noise level, with increased latencies and reduced amplitude, were observed in 92% of subjects with unilateral and in all subjects with bilateral retinal response. Both responses were phase-locked to stimulus. No correlation with the residual visual acuity or field was detected. The observation is consistent with evidence of the disease sparing the neuroretina and with unconscious visual processing and suggests miscoding of visual information processing.

Restoring vision through "Project Prakash": the opportunities for merging science and service

By treating curably blind children in India, “Project Prakash ” brings sight to children of different ages, offering insights into how their brains adapt to enable them to see. The project's experience highlights the benefits of merging basic research with societal service.

Sensory substitution: closing the gap between basic research and widespread practical visual rehabilitation

Sensory substitution devices (SSDs) have come a long way since first developed for visual rehabilitation. They have produced exciting experimental results, and have furthered our understanding of the human brain. Unfortunately, they are still not used for practical visual rehabilitation, and are currently considered as reserved primarily for experiments in controlled settings. Over the past decade, our understanding of the neural mechanisms behind visual restoration has changed as a result of converging evidence, much of which was gathered with SSDs. This evidence suggests that the brain is more than a pure sensory-machine but rather is a highly flexible task-machine, i.e., brain regions can maintain or regain their function in vision even with input from other senses. This complements a recent set of more promising behavioral achievements using SSDs and new promising technologies and tools. All these changes strongly suggest that the time has come to revive the focus on practical visual rehabilitation with SSDs and we chart several key steps in this direction such as training protocols and self-train tools.

Multi-class constrained normalized cut with hard, soft, unary and pairwise priors and its applications to object segmentation

Normalized cut is a powerful method for image segmentation as well as data clustering. However, it does not perform well in challenging segmentation problems, such as segmenting objects in a complex background. Researchers have attempted to incorporate priors or constraints to handle such cases. Available priors in image segmentation problems may be hard or soft, unary or pairwise, but only hard must-link constraints and two-class settings are well studied. The main difficulties may lie in the following aspects: 1) the nontransitive nature of cannot-link constraints makes it hard to use such constraints in multi-class settings and 2) in multi-class or pairwise settings, the output labels have inconsistent representations with given priors, making soft priors difficult to use. In this paper, we propose novel algorithms, which can handle both hard and soft, both unary and pairwise priors in multi-class settings and provide closed form and efficient solutions. We also apply the proposed algorithms to the problem of object segmentation, producing good results by further introducing a spatial regularity term. Experiments show that the proposed algorithms outperform the state-of-the-art algorithms significantly in clustering accuracy. Other merits of the proposed algorithms are also demonstrated.

How to test Molyneux's question empirically

In a recent i-Perception article, Schwenkler (2012) criticizes a 2011 experiment by R. Held and colleagues purporting to answer Molyneux's question. Schwenkler proposes two ways to re-run the original experiment, either by allowing subjects to move around the stimuli, or by simplifying the stimuli to planar objects rather than three-dimensional ones. In Schwenkler (2013), he expands on and defends the former. I argue that this way of re-running the experiment is flawed, since it relies on a questionable assumption that newly sighted subjects will be able to appreciate depth cues. I then argue that the second way of re-running the experiment is successful both in avoiding the flaw of original Held experiment, and in avoiding the problem with the first way of re-running the experiment.

Vision after 53 years of blindness

Several studies have shown that visual recovery after blindness that occurs early in life is never complete. The current study investigated whether an extremely long period of blindness might also cause a permanent impairment of visual performance, even in a case of adult-onset blindness. We examined KP, a 71-year-old man who underwent a successful sight-restoring operation after 53 years of blindness. A set of psychophysical tests designed to assess KP's face perception, object recognition, and visual space perception abilities were conducted six months and eight months after the surgery. The results demonstrate that regardless of a lengthy period of normal vision and rich pre-accident perceptual experience, KP did not fully integrate this experience, and his visual performance remained greatly compromised. This was particularly evident when the tasks targeted finer levels of perceptual processing. In addition to the decreased robustness of his memory representations, which was hypothesized as the main factor determining visual impairment, other factors that may have affected KP's performance were considered, including compromised visual functions, problems with perceptual organization, deficits in the simultaneous processing of visual information, and reduced cognitive abilities.